What is the difference between Links and LinksSQL regarding the number of links the programs databases will handle?
Apr 12, 2002, 1:32 AM
Veteran (1936 posts)
Apr 12, 2002, 1:32 AM
Post #4 of 15
Views: 15252
Quote:
Links SQL - well I've imported 500,000 into a database before but with a really decent server/servers you could import millions.I've imported the entire DMOZ dump (about 3 million records) before and basic selects took upwords of 5 minutes sometimes and ate up all CPU time in the process.
AMD Duron 900MHz
1.06GB PC133 RAM
Win2k
Apache 1.3 w/ mod_perl
MySQL 4.0
If you were planning to run an engine with that many records I hope you know some stuff about server farming.
--Philip
Links 2.0 moderator
Apr 13, 2002, 8:54 PM
User (456 posts)
Apr 13, 2002, 8:54 PM
Post #5 of 15
Views: 15272
I run Links SQL and the entire DMOZ dump on two machines (web and database). Both machines are running RedHat. The web server is running Apache w/mod_perl. The database server is:
1000Mhz AMD Athlon
1.5 GB RAM
Ultra160 10k hard drives
I get all queries back in seconds. The speed also depends on the indexing method you are using (I use INTERNAL). I do a bit of tweaking (adding MySQL indexes to certain fields, etc.) I have however limited my search results to 2000 max, which helps. Oh, and I just installed this Search Cache plugin would seems to work very well:
http://www.gossamer-threads.com/...;;page=unread#unread
Sean
1000Mhz AMD Athlon
1.5 GB RAM
Ultra160 10k hard drives
I get all queries back in seconds. The speed also depends on the indexing method you are using (I use INTERNAL). I do a bit of tweaking (adding MySQL indexes to certain fields, etc.) I have however limited my search results to 2000 max, which helps. Oh, and I just installed this Search Cache plugin would seems to work very well:
http://www.gossamer-threads.com/...;;page=unread#unread
Sean
Apr 15, 2002, 2:48 PM
Veteran (1936 posts)
Apr 15, 2002, 2:48 PM
Post #7 of 15
Views: 15140
I'm not actually using Links SQL.... thats just from testing with MySQLMan, and yes, I did have some fields indexed. I wasn't limiting search results either. "or" searches (which according to MySQL AB are not yet optimized) contributed to longer search times, as does matching 50k records.
I'm going to be trying this again on XP Pro/Apache 2.0 w/ mod_perl as soon as I get the dump downloaded and parsed again.
--Philip
Links 2.0 moderator
I'm going to be trying this again on XP Pro/Apache 2.0 w/ mod_perl as soon as I get the dump downloaded and parsed again.
--Philip
Links 2.0 moderator
May 4, 2002, 7:10 AM
Novice (8 posts)
May 4, 2002, 7:10 AM
Post #8 of 15
Views: 15074
I'm currently running a Dedicated server here at work which runs RedHat 7.2 ATA 100 89.6 GB HardDisk, 2 GB of ram with an AMD Duron 900 Mhz CPU. I'd like to be able for testing purposes only to download the DMOZ information into my machine, any help on doing this would be greatly appreciated. I find it hard that a dedicated machine which only serves this information cannot handle this information. Bandwidth is not a concern of mine either. Please let me know so that I may help contribute to the information on these posts and help others also when reaching such a large limit speed things up. Thank You
May 4, 2002, 12:45 PM
User (82 posts)
May 4, 2002, 12:45 PM
Post #11 of 15
Views: 15101
Hi Paul
PIII 1.4
256 mb ram
Red Hat 7.1
I have seen your comments regarding problems running more than 500,000 links, I was surprised to see that you had problems.
The site is serving 12,000 pages per day with search.cgi having 1,000 plus hits per day.
Don't know whether this is average or not.
dregs2
PIII 1.4
256 mb ram
Red Hat 7.1
I have seen your comments regarding problems running more than 500,000 links, I was surprised to see that you had problems.
The site is serving 12,000 pages per day with search.cgi having 1,000 plus hits per day.
Don't know whether this is average or not.
dregs2
May 5, 2002, 12:00 PM
Administrator (9387 posts)
May 5, 2002, 12:00 PM
Post #12 of 15
Views: 15030
You may also want to tweak your MySQL setup if you have 2 gigs of ram. Look at their my-huge.cnf sample configuration. This will greatly improve the performance of MySQL, especially on large databases.
Cheers,
Alex
--
Gossamer Threads Inc.
Cheers,
Alex
--
Gossamer Threads Inc.