What is the difference between Links and LinksSQL regarding the number of links the programs databases will handle?

Links2, no more than 8-10,000 a the _max_ - GT recommend about a couple of thousand I think I remember Jack saying.

Links SQL - well I've imported 500,000 into a database before but with a really decent server/servers you could import millions.

Thanks!

I've imported the entire DMOZ dump (about 3 million records) before and basic selects took upwords of 5 minutes sometimes and ate up all CPU time in the process.

AMD Duron 900MHz
1.06GB PC133 RAM
Win2k
Apache 1.3 w/ mod_perl
MySQL 4.0

If you were planning to run an engine with that many records I hope you know some stuff about server farming.

--Philip
Links 2.0 moderator

I run Links SQL and the entire DMOZ dump on two machines (web and database). Both machines are running RedHat. The web server is running Apache w/mod_perl. The database server is:

1000Mhz AMD Athlon
1.5 GB RAM
Ultra160 10k hard drives

I get all queries back in seconds. The speed also depends on the indexing method you are using (I use INTERNAL). I do a bit of tweaking (adding MySQL indexes to certain fields, etc.) I have however limited my search results to 2000 max, which helps. Oh, and I just installed this Search Cache plugin would seems to work very well:

http://www.gossamer-threads.com/...;;page=unread#unread

Sean

Did you reindex???


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

I'm not actually using Links SQL.... thats just from testing with MySQLMan, and yes, I did have some fields indexed. I wasn't limiting search results either. "or" searches (which according to MySQL AB are not yet optimized) contributed to longer search times, as does matching 50k records.

I'm going to be trying this again on XP Pro/Apache 2.0 w/ mod_perl as soon as I get the dump downloaded and parsed again.

--Philip
Links 2.0 moderator

I'm currently running a Dedicated server here at work which runs RedHat 7.2 ATA 100 89.6 GB HardDisk, 2 GB of ram with an AMD Duron 900 Mhz CPU. I'd like to be able for testing purposes only to download the DMOZ information into my machine, any help on doing this would be greatly appreciated. I find it hard that a dedicated machine which only serves this information cannot handle this information. Bandwidth is not a concern of mine either. Please let me know so that I may help contribute to the information on these posts and help others also when reaching such a large limit speed things up. Thank You

I am running 1.4 million links from DMOZ

Any search runs under 2 seconds.

Not running under mod_perl.

Deciated server running redhat 7.1

dregs2

What are your server specs?

I've had slow downs with 300,000 links with mod_perl on a 1ghz server with 612MB RAM.

Hi Paul

PIII 1.4

256 mb ram

Red Hat 7.1

I have seen your comments regarding problems running more than 500,000 links, I was surprised to see that you had problems.

The site is serving 12,000 pages per day with search.cgi having 1,000 plus hits per day.

Don't know whether this is average or not.

dregs2

You may also want to tweak your MySQL setup if you have 2 gigs of ram. Look at their my-huge.cnf sample configuration. This will greatly improve the performance of MySQL, especially on large databases.

Cheers,

Alex
--
Gossamer Threads Inc.

Hi,

Just wanted to know : what application did you use to import DMOZ to Links SQL ?

Thanks !

nph-import.cgi that comes with Links SQL will do it.

I have never gotten that to import the DMOZ database file
www.sagnetwork.com - Internet Portal