Well, check out CarBuzzard (http://www.carbuzzard.com). The index in the left sidebar was done with a plug-in and then by a techie who allowed it to take on more entries than it had when it hit a certain limit. I think it was about 180 links, but my memory…
Anyway–and no doubt there are those out there much better qualified to answer that question than I, but this is one example, though I suspect you’re looking for something much larger.
Thank you Carbuzzard for the example. Yes I can see the example, but still your website is only 6k pages. I am asking about massive directories that have more than several million pages.
Using the Google “site” search operator
site:wordpress.org
WordPress.org uses WordPress & BuddyPress
About 1,090,000 results
site:wordpress.com
About 1,300,000,000 results
I build some very large directories, some with over 10,000,000 pages such as Wellness.com. With any dynamic CMS such as WordPress, Joomla, Dotnet Nuke and Drupal, the limiting factor will most always be the database, as constructing each page on the fly is very slow and database-intensive. The way that some sites have managed to build 1-3 million pages, not “millions” is through the extensive use of page caching. An efficiently designed, compressed cached page will be served faster than most anything. So in theory, if most of your pages are cached, you could run millions of pages on any CMS. The problem is back to when you want to update a single line of code, color, style or fix on your dynamic pages. To “re-cache” one million pages for example, takes millions of hits to the database. It doesn’t matter how expensive your hosting plan, you can be looking at more than 1-week of time-consuming re-caching per (1) million pages. While you are recaching your pages, your site will often be extremely slow. Bottom line. Its very hard to serve a large directory from a CMS. I try to keep my CMS-served directory sites under 250,000 pages for speed-sake otherwise, we use a custom CMS that we built which uses a document style database called MongoDB. There are also a few innovative folks trying to use WordPress and MongoDB together. You can look it up. I love large directories–fun stuff. Good luck.
I should have added that the best way to cache a large WP site is probably using Varnish.
Still, caching and recaching your pages will take time and be database-intensive. The common practice that webmasters do is making changes to their site, clear the cache and wait for the cache to build over time. In reality it will never build up a large enough cache before the next time you want to make changes and clear the cache again. So, all too often, Google will be doing your recaching for you making the site slow for Google and faster-seeming for you. Showing Google an uncached page is no good. Google will record the page speed too slow and you’ll never get any SEO traction.
If you’re needing to build a cache faster before Google hits your pages, always crawl your site remotely before Google and Bing do. You can do this with a free crawler took such as Integrity Link Crawler for Mac and Link Sleuth for PC.
If your site is still slow, you’ll need to re-engineer to make fewer and fewer hits to the database, smaller pages, more efficient pages and so on. If you really want to make a directory with millions of pages you’re going to need a lot of help way beyond what caching can provide. That’s very expensive to do a large directory. We spend over $100,000 per month on development. There are no short cuts to large directories.
