Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

How many categories are you using?

Quote Reply
How many categories are you using?
I'm looking at using a very large number of categories when I relaunch Qango under links SQL and wondered just how many categories other people are currently using? This is to help me gauge just how far I can take my new category ideas, and to establish if there is a usability cut-off limit, or if the number of categories is limited only by disk-space and machine power.

All the best
Shaun

Quote Reply
Re: How many categories are you using? In reply to
396 Categories. And my understanding is that Links SQL does not support 1,000's of categories. I believe there are some older Threads that discuss this in this forum.

Regards,

Eliot

Quote Reply
Re: How many categories are you using? In reply to
I had almost 4,000 categories in one site, and cut that down to about 1,000 for content reasons, not performance ones.

The 1,000 limit is the number of links-per-category, which is done to limit the memory usage, and prevent server congestion due to excessive memory usage between the MySQL threads/process and the Links SQL HASH tables.

Links can handle the ODP/DMOZ directory, which has well over 10,000 categories by now, and probably over a million links. That is starting to push the limits of MySQL, since most of the performance data I've seen is for about 1 million records, but there could be new data posted on the http://mysql.org site by now.

If you intend to have a directory with that many records, you will not be able to do it on an ordinary server. You will need a high-end server, with as much RAM as your hardware will comfortably allow -- 512 - 1gb minimum. Why? The larger the database, the larger the index files, and the more MySQL needs to load (and work with) in memory at any given time. I'm not even talking about the "Index" files we use to speed data access, but the internal files MySQL uses to manage the database itself.

You will also need _really_ good harddrives, probably a performance RAID system, in order to manage that much data in real-time.

Could you do it with less? Probably. Would you want to do it with less, not likely.

My largest directory was about 4,000 categories and 37,000 links. My average directory is 2-300 categories and < 5,000 links. My performance is great at that level.

I didn't have any problem with the larger directory either (it was sharing the same hardware as the other directories). But, my volume on the server is relatively low at this time of year (which is why I'm playing with it). Under 750,000 hits per day (graphics, pages, cgi, etc). During the busy time, it was doing well over 1,000,000 and I'm expecting to double that starting in September.

I tend to over-buy hardware, since it's something that affects your performace, and you are stuck with it for a lease term or longer. Excess server capacity never hurt a site. And, it's a lot less expensive than bottle necks, down time, and dead-air.



http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/

Quote Reply
Re: How many categories are you using? In reply to
In Reply To:
The 1,000 limit is the number of links-per-category, which is done to limit the memory usage, and prevent server congestion due to excessive memory usage between the MySQL threads/process and the Links SQL HASH tables.
Also, I chose that number because I thought there's no way somebody would want more then 1,000 links in a single category. There is no reason why you can't increase it by going into nph-build and making it higher (I've had to do this to two people now).

I think however you are better off asking yourself why aren't you subcategorizing this category, no one is going to find anything with a 1000 links there. =)

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: How many categories are you using? In reply to
There was just a thread a week or so ago about that.

I fully agree about the uselessness of a 1,000 link category (and I think I posted that as a reply in that thread <G>).

From a technical point of view, my big concern beyond usefullness is that looking at the MySQL docs, and what it says about internals, and stored searches, etc, and how Links stuffs everything into a hash and a large output variable, increasing the 1,000 limit will start consuming more system resources -- perhaps insidiously -- and largely with no purpose.

If you change this for a "search" for example, each search will be yanking 1,000+ (whatever you set it for) "matches" from the database, and no user is going to page through all 1,000+. They'll most likely do another search.

The only time you'd really want to do that is if you allowed sub-searches on found data, "Search in selected links only" where you could take a 16,000 "hit" search, and whittle it down to a few dozen with progressively tighter keywords.


http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/

Quote Reply
Re: How many categories are you using? In reply to
Thanks for the comments about hardware, etc. I'm currently in discussions with my ISP about a semi-dedicated server account and if it comes-off I just wanted an idea of how far I could go with my cats and sub-cats.

The semi-dedicated account would give me 10GB webspace, more memory and CPU time and a max. of 10 other sites on the server in total - so hopefully that'll be enough to cope with everything I plan to add. I have so far prepared 9000+ cats/sub-cats and would obviously want to do lots of testing before filling it up with links and throwing it out at the general public.

My main problem is revenue - to earn more I need to expand - to expand I need more revenue (I'm sure I'm not the only one in this situation?) so I've proposed a kind of venture/sponsorship deal with my ISP to get the extra facilities - we'll see how that turns out Smile

All the best
Shaun

Quote Reply
Re: How many categories are you using? In reply to
Sounds to me like it is DEDICATED (unless you are sharing the server with other accounts)! What the heck do you mean by "semi-dedicated"?

Regards,

Eliot Lee

Quote Reply
Re: How many categories are you using? In reply to
He means he's on a machine with a limited number of other sites (10), rather than the usual 500+.



http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/