Gossamer Forum
Home : Products : Gossamer Links : Pre Sales :

Heavy duty directory?

Quote Reply
Heavy duty directory?
Hello,

Practically speaking is is possible to run a heavy duty directory (using Links SQL) like Yahoo, which sustains millions of searches a day? I know the database is scalable but what are the limitations besides the server resources? I am sure yahoo uses several servers and load balancing, etc., How can one use Links SQL to run a niche directory/search engine similar to yahoo with thousands of users and searches each day? So, obviously for a heavy duty portal one directory needs to be spread accross/hosted on several servers and how does one (yahoo for example) accomplish this?

thanks
Quote Reply
Re: [socrates] Heavy duty directory? In reply to
Quote:
How can one use Links SQL to run a niche directory/search engine similar to yahoo with thousands of users and searches each day?


With the right hardware, meaning a well set up high-end server, mod_perl, and perhaps a separate database machine, Links SQL can do that easily per hour. By "high end" I mean a multi processor machine, optimized for web serving, and those are in the $8-$10k range now.

Did you mean "thousands" per day, or closer to a Million per day? :)

I'm sure Alex has specifics, or can extrapolate them, but I'm sure they are pretty intense.

The reason you'd want a separate database machine, and perhaps even a separate "cache" machine, is that by optimizing each machine for what it's doing, the CPU and resources are maintained, rather than shared or rotated between competing processes. By off-loading intensive operations like database searching from the web machine, that machine might not even break a sweat.

We've been running on compartively small machines, and Links SQL has been able to handle pretty intense loads. I don't have any stats on running on Intel/Unix plat form, as we are still not back close to what we were before our crash.

Alex has posted on (google? I keep forgetting) on what they use... basically where most companies have used a RAID archetecture (Redundant array of inexpensive disks) they went the way of RAIC (Redundant array of inexpensive computers <G>). They went closer to the BBS model, where each connection was handled by a separate low-end CPU, that shared network resources, and the "high end" processing was put in a single location, ie: network server.

By spreading your web connections around and script processing across several machines, and having a higher-end database server dealing with the I/O requests (not script processing) you probably can run a system that staggers the minds of those still thinking "last year" <G>


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [socrates] Heavy duty directory? In reply to
I agree with Pugdog. All Links SQL does is query the database using SQL strings, sort the results, and provide a customized frontend to the whole thing. If you had hardware similar to what Yahoo is using, and copy of Links SQL, you could easily build a directory as big, and support the same amount of traffic. Also, Yahoo is using an Oracle database, which is supposedly supported by Links SQL.

Sean
Quote Reply
Re: [SeanP] Heavy duty directory? In reply to
>>Yahoo is using an Oracle database<<

Yahoo is using a Google database :)


Quote:
At Yahoo Inc.'s Yahoo!Finance division, "we use [MySQL] for a lot of things, a lot of our back-end sort of processing," said Technical Yahoo Jeremy Zawodny. MySQL handles chores like data retrieving and sorting, from which the Web pages are produced. "There's a couple reasons we use it. One is that it's lightning fast. The other is that it's easy to configure. You can literally just set it up and walk away. The downside is that it doesn't have all the features Oracle has, but 99 percent of people don't need them."

Yahoo, of Sunnyvale, Calif., uses Oracle elsewhere in the organization.
Quote Reply
Re: [Paul] Heavy duty directory? In reply to
I know they use Google when results aren't found in the Yahoo Directory. They've changed things around, because I didn't know they were using MySQL for anything (cool). I believe they started with only an Oracle database, but I may be wrong.

Sean
Quote Reply
Re: [SeanP] Heavy duty directory? In reply to
You could still be right....they use mysql for their finance division....perhaps they still use oracle for their directory?
Quote Reply
Re: [Paul] Heavy duty directory? In reply to
I say we call them and find out... Laugh
Quote Reply
Re: [socrates] Heavy duty directory? In reply to
Hi,

Yes, it's quite possible with the proper hardware. There are many techniques for scalaing a database such as replication, load balancing, caching, etc.

If you are serious about persuing something of this size, please do contact us and we'll be happy to provide you more information and recommendations.

Cheers,

Alex
--
Gossamer Threads Inc.