I had almost 4,000 categories in one site, and cut that down to about 1,000 for content reasons, not performance ones.
The 1,000 limit is the number of links-per-category, which is done to limit the memory usage, and prevent server congestion due to excessive memory usage between the MySQL threads/process and the Links SQL HASH tables.
Links can handle the ODP/DMOZ directory, which has well over 10,000 categories by now, and probably over a million links. That is starting to push the limits of MySQL, since most of the performance data I've seen is for about 1 million records, but there could be new data posted on the
http://mysql.org site by now.
If you intend to have a directory with that many records, you will not be able to do it on an ordinary server. You will need a high-end server, with as much RAM as your hardware will comfortably allow -- 512 - 1gb minimum. Why? The larger the database, the larger the index files, and the more MySQL needs to load (and work with) in memory at any given time. I'm not even talking about the "Index" files we use to speed data access, but the internal files MySQL uses to manage the database itself.
You will also need _really_ good harddrives, probably a performance RAID system, in order to manage that much data in real-time.
Could you do it with less? Probably. Would you want to do it with less, not likely.
My largest directory was about 4,000 categories and 37,000 links. My average directory is 2-300 categories and < 5,000 links. My performance is great at that level.
I didn't have any problem with the larger directory either (it was sharing the same hardware as the other directories). But, my volume on the server is relatively low at this time of year (which is why I'm playing with it). Under 750,000 hits per day (graphics, pages, cgi, etc). During the busy time, it was doing well over 1,000,000 and I'm expecting to double that starting in September.
I tend to over-buy hardware, since it's something that affects your performace, and you are stuck with it for a lease term or longer. Excess server capacity never hurt a site. And, it's a lot less expensive than bottle necks, down time, and dead-air.
http://www.postcards.com FAQ:
http://www.postcards.com/FAQ/LinkSQL/