Got a bit of a weird one here. We have 2.1 million links in the database, INTERNAL INDEXING is turned on, gen_dynamic_list is turned off too.... so they have to pass the ID number to add.cgi (i.e so it doesn't load 300,000+ categories).

Anyone got any ideas?

I've set debug to 0 (incase anything was being printed to STDOUT/STDERR).

I'll keep working on this.. but its very weird :/

TIA

Andy (mod)
andy@ultranerds.co.uk

---

Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!

Where is it freezing?


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

After submitting the form. I have timed it to hang for 10 minutes already, and after that it simply gives a 500 IS error (timeout I would imaging).

Cheers

Andy (mod)
andy@ultranerds.co.uk

---

Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!

Andy,

put some "print" statements in the code, to follow the progress.

print "I've reached ___";

print "I'm inside the ___ routine";

print "I've returned from __";



I do this to make sure I'm actually going and returning from where I am.

do a print $IN->header() of course, and those prints will show up on the screen even if the HTML never prints out.

Once you know where it *is* getting to, you can figure out why it's not going further.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

Hey Andy,

I'd suggest using the MYSQL driver now. I have a feeling after so many links, mysql is having problems managing the size of the Links_Word_Score table. There's probably more than hundred million records in that table by now.

If you've got the latest mysql search driver and mysqld version 4.0 or higher, it will allow faster returns on the search queries as well.

Best Regards,
Aki

Ok, I'm confused. Isn't INTERNAL indexing meant to speed the search processes up? Why would turning it back onto MYSQL indexing make a difference? (may just not be thinking straight after last night...lol).

Cheers

Andy (mod)
andy@ultranerds.co.uk

---

Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!

Here goes the nasty technical explanation ;)

Two piggyback tables are required to maintain the internal table. One is for the individual words found, the other is for each instance the word is found. When a new word is found in the text that has been added, it will be added to the word table. Then, for each word in the text of the record that has just been added, an entry is made into the instance table. The problem arises in the fact that indexes are used in each of the tables to optimize the speed... maintaining the indexes slows down with every insert. It's not noticable for the first few million instances IMHO, but when you grow to several million Mysql slows down. Assuming you've got around 20 words in an average link record, that's around 60 or so million word instance records in the score database. At that point, subsequent records to be inserted will become slower and slower until until it becomes unbearable. And I believe that's what you're encountering.

If you have mysql 4.0, there's a native boolean search system available that has been optimized for very large recordsets. The great things about it is that it greatly minimizes the amount of work the libraries have to do to fetch records and it's a heck of a lot faster and doesn't suffer the same limitations as the internal indexer does.