Gossamer Forum
Home : Products : Gossamer Links : Pre Sales :

Server Resources

Quote Reply
Server Resources
My host says that if I surpass a 16MB memory
footprint, and 100 CPU seconds of server time, then all my processes get killed.

Can I run Links SQL
a) At all
b) Under Mod Perl
c) Dynamically
d) Under Mod Perl and dynamically
e) DMOZ Imports?
Many thanks.


Quote Reply
Re: Server Resources In reply to
a) Probably with its basic features. But you may have problems with jump.cgi and search.cgi depending on the amount of links and categories you have in your directory.

b) Most likely not.

c) Most likely not.

d) Most likely not.

e) Definitely not.

Regards,

Eliot Lee
Quote Reply
Re: Server Resources In reply to
Eliot is right.

To use "all" the features of Links SQL you listed, you would need a basic dedicated server, with 256 Meg ram min for Unix/Intel, 128 Sun/Sparc, and 512Meg+ for Windows). Figure the cheapest accounts I've seen offering that are in the $300 range per month, and that doesn't mean they are worth anything at that price. $500 is closer to reality.

DMOZ is a real problem -- it's HUGE and to import it takes a lot of effort, to index it takes a lot of CPU the first time, and loads of disk, space, and to use it requires loads of disk and RAM for cache.

A moderately busy site using Dmoz and Dynamic pages, would probably need to have a 512 Meg Unix box, with at least 2, maybe 4 fast harddrives, and that would still be only a moderately busy site. 40-60 requests per minute average.
Beyond that, you _will_ see cpu use, I/0 churning, and even bottle necks.

Alex posted the hardware for HotBot (?).... thousands of machines all working together. Software may count for something, but you can't do it without the proper hardware.



PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Forum:http://LinkSQL.com/forum
Quote Reply
Re: Server Resources In reply to
In Reply To:
Alex posted the hardware for HotBot
Actually it was Google. In the last year they have expanded the size of their server farm from 4,000 to 8,000 linux machines.

That said, you can still use Links SQL on a virtual host, just keep your expectations realistic. If you are trying to run a site with 2,000,000 links, you will need a powerful computer to do it. If you are trying to run a directory with 20,000 links, you could get a way with a virtual server.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Server Resources In reply to
I don't get why they need so many machines. 8000?

How big is their database?....A few hundred gigs?...a few thousand?

If each of those machines has 20GB hard drive, that is 160,000GB's

What are they gonna do with that?.....or are they all redundant "juice"?

Or are they there for their RAM and CPU's?

Paul
Installations:http://wiredon.net/gt/
Support: http://wiredon.net/forum/

Quote Reply
Re: Server Resources In reply to
There are a lot of reasons to use so many machines. You start hitting performance bottlenecks if you only have one machine. A lot of it is I/0 on the disks, since people are asking for different stuff. You can cache the common searches, but you still have an I/0 problem on each new request.

Additionally, if you have 50,000 requests coming in at the same time, you need to break those requests up among different machines, especially if all of them are searches (cgi-type calls).

You also need to break up your database into balanced chunks, to improve performance, and that is a whole technology unto itself.

Also, by using many machines, if any one machine, switch or bank dies, the rest of the network still runs, with calls being routed past the dead zone.

On top of that, if you have a lot of machines fielding calls, the number of calls hitting any one machine are going to be small, meaning that machine isn't going to have to use disk to cache processes, isn't going to have an I/0 bottleneck, and won't have a general resource depletion state occuring at any given time.

One image, is imagine an arena, with only one main entrence and exit point. One set of access stairways or ramps, one path for everything to get in and out of.

Imagine the same arena, if there were 4 sets of ramps, and 8 entrance and exit points.

It's more complicated to set up initially, but in operation, it runs much smoother, since people aren't all trying to use the same doors, ramps, and space.


PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Forum:http://LinkSQL.com/forum
Quote Reply
Post deleted by Fortune In reply to
Quote Reply
Re: Server Resources In reply to
It's working fine for me :)

PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Forum:http://LinkSQL.com/forum
Quote Reply
Re: Server Resources In reply to
Thanks for the explanation.

Paul
Installations:http://wiredon.net/gt/
Support: http://wiredon.net/forum/