Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

Emergency Help!

Quote Reply
Emergency Help!
We are building up a category with over four thousand links.
In Admin we can see all those links. But in search.cgi we can only see some of the links and they are out of order. Can someone help me? Thanks.

Quote Reply
Re: Emergency Help! In reply to
Hi,

Alex has set may hard-coded limits in his routines to prevent errors in memory usage. The variable he used is $limit=1000; you will find it in many subs.

Try to locate them and set them higher - but no guarantee!

The better way is to split your categorys until each cat has < 1000 links.

regards, alexander

Quote Reply
Re: Emergency Help! In reply to
Have you looked at the Links SQL FAQ web site, which is linked in the Resource Center?? There are a few FAQs about sorting links in the search results that may help you.

Regards,

Eliot

Quote Reply
Re: Emergency Help! In reply to
You are hitting possibly two problems.

First, a category with 4,000 links is not very useful. It can't be browsed, only searched.

Second, in the various routines, Alex has limited the selected links to '1000' for memory reasons, and for other more realistic reasons -- such as performance and the fact that if you are selecting 1000 links, you really need to break up that data chunk.

I'm not sure if you are hitting the search bug/problem, but you are certainly hitting the hard-coded, pre-defined, limits for "SELECT" statements.

You _can_ change those, but you'd need a very powerful machine, and I would _NOT_ do it on a virtual server or shared server.

At some point, even with Unix, the more memory you ask for becomes somewhat exponential rather than linear. It has to do with swap files, caches, and all sorts of OS specific things where the designers had to decide on somewhat arbitrary limits for performance reasons.

People seem to forget that the MySQL server _also_ uses memory and system resources. If you can do a ps -ef or ax (?) to see running threads, you can get an idea of what MySQL is doing (only if it's running on that machine). If you have a program like phpMyAdmin, you can get a list of the statistics for the MySQL server by a click on the front screen. It might surprise you.

So, the more huge searches you do, the more load you put on the MySQL server, and the more work it has to do to try to keep track of it's results.

You start a downward spiral.

1000 hits for a program like links is really good. You should try to get it MORE specfic, not less.

After all, how many times have you gone past the first 100 or 150 links of ANY search on any search engine you've ever used??

If you break your links down by alphabet, region, state, etc then you can still do searches on the entire list, but each build/select becomes a smaller unit of data to work with at one time.


http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/