Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

How many links can SQL handle

Quote Reply
How many links can SQL handle
Hi,

Just a simple question really. I know we are talking that the limit of Links 2.0 is about 15,000 - 20,000 links before is gets to slow.

What is SQL's limit. I am hoping that is will handle over 250,000 and search them in seconds.

Also what's next from SQL. I mean look at Altavista. How on earth does that search engine search through millions of web sites in seconds? Can links SQL do this?

Martin
Quote Reply
Re: How many links can SQL handle In reply to
SQL is a Structured Querry Language. It's a means to talk to a database. The confusion comes when people start to blur the distinctions.

MySQL boasts some very hefty claims. If you check out their site, they have bosts about 45 colums and 4 million+ rows.

The database system, NOT the interactive front end ie: LinksSQL is what will determine capacity, speed and performance.

Unlike Links 2.0, the MySQL package is what is doing all the "work" of indexing, maintaining, and accessing the "database". LinksSQL is just asking for something, and the MySQL package is returning it. (Hopefully!!)

Sure there are things in the front ends that can improve speed and performance, and things that can slow it down, but sort of like the CPU vs Modem problem, which will limit the connection first?

You might find that MySQL is not the best for your needs. If so, the modifications to LinksSQL should be minor to get it to work with other databases, since that is the purpose of SQL (in theory).

I'm sure Alex will have a more detailed answer, as it's probably one of the most asked questions; but the real tests will come only when some site actually hits 1,000,000 links, and see if it works Smile




Quote Reply
Re: How many links can SQL handle In reply to
Links SQL "worked" with the full open directory (600,000 links), but I wasn't happy with performance. The build process took to long and the search/index took to long to be usable.

I would say that at present the recommended limit of Links SQL is 100,000 links (although if you have some more powerful hardware you could do better, best I've used it on is a pII 350 so far).

We are actively working on improving that. Things that need to change:

- Better Index/Searching. Links SQL currently indexes all the information, but it needs to do it differently to still return search results in a reasonable amount of time.

- Build Changed: There's no reason to rebuild a couple hundred megabytes of pages because a new link was added, instead only affected pages should be updated. This feature will make Links usable on very large directories (> 200,000).

Quote:
The database system, NOT the interactive front end ie: LinksSQL is what will determine capacity, speed and performance.

The design of the program has a lot to do with it! If you try and search across a table with 1 million rows, and don't use some indexing scheme, it's going to take a long time! Links SQL maintains it's own search index to help speed up searches.

Quote:
Also what's next from SQL. I mean look at Altavista. How on earth does that search engine search through millions of web sites in seconds? Can links SQL do this?

No and I don't think it ever will. Altavista uses some pretty serious hardware (I couldn't find their about page anymore that listed it), but it's a couple million dollars worth at least. They also use proprietary programs to do the indexing/searching, they don't use an SQL database.

Cheers,

Alex
Quote Reply
Re: How many links can SQL handle In reply to
Well, Alex, I'm willing to use it on a SUN Sparc 10, 333 w/128 meg if I can be certain it will do what I have links 2.0 doing ...

Scott
Quote Reply
Re: How many links can SQL handle In reply to
Hi Alex

With all the data above, that means that I cannot use LinksSQL in my WebSite? does the same have 140.000 URL today registered, does that mean that will have problems?

------------------
By Jānio
-=-=-=-=-=-=-=
http://www.brlink.com.br
Quote Reply
Re: How many links can SQL handle In reply to
If you are using Links 2.0 with 140,000 links, LinkSQL will _certainly_ do better!

Quote Reply
Re: How many links can SQL handle In reply to
No, I use another software that I did, that has characteristics similar to Links 2.0.
And it would like to pass the same for SQL, if somebody is interested in doing that service for me, I am enough to say the price.

the same is in http://www.brlink.com.br

email? janio@brlink.com.br

------------------
By Jānio
-=-=-=-=-=-=-=
http://www.brlink.com.br
Quote Reply
Re: How many links can SQL handle In reply to
LinkSQL comes with installation, and I'm sure Alex could write a script that would import your current database without too much trouble.

Since you have a working site, it would be easy for Alex to tell you if LinkSQL would do what you need.

Ask him Smile
Quote Reply
Re: How many links can SQL handle In reply to
I do not know about other people, but I can not get Links 2.0 or Links SQL to run 5000 links. I even turned the problem over to Alex, since he installed SQL, and he has failed to respond to the problems, after a week+.

I set up 2.0 on a different server to debug the database (never ruling out that the problems started there) and the one common reoccuring problem is that when a directory gets above so many links and/or sub-cat's, the program crashes. And it always crashes at random spots so it is impossible to point to a caterogy or block of links and say, "Here is a problem".

After 2.5 weeks and 100s of man hours, I am uncertain if SQL is really a good option for large commercial databases.

I would welcome any Positive Proof that is available.
Quote Reply
Re: How many links can SQL handle In reply to
 
Hi

Interesting comments regarding Links SQL and 5000 links.

Alex has installed Links SQL for me and I have imported 129,000 links and it works perfectly well.

Perhaps you have hardware problems - but I know it's not the design of Links SQL.

I have a few problems - but that's just design related.

http://www.chillimonster.com

Tony
Quote Reply
Re: How many links can SQL handle In reply to
Just a follow up, the 5,000 links was indeed a hardware problem that has since been fixed up (a long time ago actually).

Cheers,

Alex
Quote Reply
Re: How many links can SQL handle In reply to
Hi Alex,

I have been reading the older posts on LinksSQL and was wondering what's the status on how many links it can handle ?

What would handle 500,000 links entries within 1 yr ?

-----------------

[This message has been edited by theprequels (edited February 05, 2000).]
Quote Reply
Re: How many links can SQL handle In reply to
I am thinking from my use so far, and some of the tests I have been doing that with a decent dedicated server you could handle that.

------------------
Michael Bray
Review your webhost or find a new one.
www.webhostarea.com
Links SQL User
------------------




Quote Reply
Re: How many links can SQL handle In reply to
Could you tell me how you imported resources from Links2 into LinksSQL?

Thanks

From Paul Wilson.
http://www.audio-grabber.com
On error resume next..
Quote Reply
Re: How many links can SQL handle In reply to
Uh...Paul,

All LINKS SQL versions have a script called import.pl or nph-import.cgi in the NEXT GENERATION version that allows you to IMPORT data from Links 2.0 to LINKS SQL.

Regards,

Eliot Lee
Quote Reply
Re: How many links can SQL handle In reply to
I know but when I try to import the Links2 database I get an error saying:

Error opening Category.def - Def file temporarily unavailable.

I have the path the the def files correct and they are chmodded correctly and I have tried several times.

...but I was reading a post by Alex the other day and he said that LinksSQL2 could not import from Links2. He said that you'd have to install LinksSQL1 and import the database from Links2 and then install LinksSQl2 and import the database from LinksSQL1.

I dont have LinksSQL1 though!

From Paul Wilson.
http://www.audio-grabber.com
On error resume next..
Quote Reply
Re: How many links can SQL handle In reply to
In Reply To:
LinksSQL2 could not import from Links2
That is correct...and if you are using the NEXT GENERATION version than you should post questions related to that version in the NEXT GENERATION forum.

Regards,

Eliot Lee
Quote Reply
Re: How many links can SQL handle In reply to
Is this not a forum for general Links SQL discussions?

Also if I were to repeat this thread in the Next Generation Forum then I would get abuse telling me not to create duplicate threads!.....I can't win!

From Paul Wilson.
http://www.audio-grabber.com
On error resume next..
Quote Reply
Re: How many links can SQL handle In reply to
Paul,

Really, all the stuff about the NG version should be in the NG forum. Also, as far as duplicate threads, this thread is no longer "on-topic" so changing to a new thread would be fine.

The import for Links should be fixed in the current release.

PUGDOG®
PUGDOG® Enterprises, Inc.
FAQ: http://pugdog.com/FAQ


Quote Reply
Re: How many links can SQL handle In reply to
Thanks for clarifying that!

The version I am using was downloaded 2 days ago...is that the current release?



From Paul Wilson.
http://www.audio-grabber.com
On error resume next..