Gossamer Forum
Home : Products : Gossamer Links : Discussions :

Server Overloaded - How to scale up?

Quote Reply
Server Overloaded - How to scale up?
Hi,

I'm running lsql on a dedicated server and there is too much server load. I have tried running things with a single apache with mod_perl, but things seem to hang rather than improve.

How should I scale up? Should I get another server and try to do load balancing or perhaps get a bigger server? Anybody have experience of this kind of thing?

Thanks for your help.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Could I ask how many links/category/hits you have/get?

Last edited by:

Paul: Feb 4, 2003, 1:20 PM
Quote Reply
Re: [Paul] Server Overloaded - How to scale up? In reply to
Hi Paul,

Thanks for the reply.

We have over 14,000 links and 470 categories. Only the links are html articles, rather than hyperlinks.

The traffic is only a problem during peak times, which is between 7-9. I'm not really sure how much traffic is between these times, but server load is getting worringly high.

We have to run things dynamically in order to log and protect the information in the database.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
As long as your traffic isn't extremely more than I'm anticipating then I don't see any reason why that number of links/categories would cause you any problems. Especially under mod_perl.

Does your server have 256MB RAM or more?
Quote Reply
Re: [Paul] Server Overloaded - How to scale up? In reply to
Hi Paul,

We currently have a dedicated linux 512mb RAM 1.4 GHz server, and we are having serious overload problems. Have tried running mod_perl, but it didn't seem to offer much speed advantage.

Do you have any idea of how we could use Links SQL across multiple servers? Alternatively we could get double the ram and double the number of processors on the current machine and switch to SCSI. The latter would be more expensive.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
512MB and 1.4ghz should be more than enough for your directory size. I'm not sure what the reasons for the slow down are.

It might be worth contact Gt directly for assistance - they may be able to trace the cause.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Add another 512 of RAM.


PUGDOG´┐Ż Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [Paul] Server Overloaded - How to scale up? In reply to
In Reply To:
512MB and 1.4ghz should be more than enough for your directory size. I'm not sure what the reasons for the slow down are.

It might be worth contact Gt directly for assistance - they may be able to trace the cause.

Hi Paul,

Everything works fine - normally very quickly. It's just that we are getting deluged with visitors - not a problem - just causing headaches initially.
Quote Reply
Re: [pugdog] Server Overloaded - How to scale up? In reply to
In Reply To:
Add another 512 of RAM.

Hi Pugdog,

Thanks for your response. I might try this, and then perhaps if it doesn't work, then I'll try installing another install of apache, and then run a copy serving the mod_perl pages and another serving the html pages. I think that this is what Alex suggests.

Do you think that adding another 512mb will help a great deal? Is this a big choke point?
Quote Reply
Re: [Paul] Server Overloaded - How to scale up? In reply to
In Reply To:
As long as your traffic isn't extremely more than I'm anticipating then I don't see any reason why that number of links/categories would cause you any problems. Especially under mod_perl.

Does your server have 256MB RAM or more?

Hi Paul,

Basically, I can only see the traffic for the sites going up and up. How would one handle the traffic if say traffic tripled? How would one split the load between 2 servers? Conceptually, I will have to reach this point and overcome this bottleneck. How have you personally managed to do this in the past? How do Gossamer Threads Customers manage to scale up to handle very high traffic?
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
I'm not at the point of having to load balance (I hope I get there though :) ) - my site has only been online a few weeks.

I think Gossamer offer load balancing solutions, or if you don't wish to host with them I'm sure they can give you some great advice.

Last edited by:

Paul: Feb 4, 2003, 3:25 PM
Quote Reply
Re: [Paul] Server Overloaded - How to scale up? In reply to
In Reply To:
I'm not at the point of having to load balance (I hope I get there though :) ) - my site has only been online a few weeks.

I think Gossamer offer load balancing solutions, or if you don't wish to host with them I'm sure they can give you some great advice.

Thanks Paul, I really appreciate your advice. Best of luck with your site!

I think I have a good idea of how to proceed now.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Sorry I couldn't give you a more technical answer.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Quote:
The traffic is only a problem during peak times, which is between 7-9. I'm not really sure how much traffic is between these times, but server load is getting worringly high.
Could not give us at least idea how much traffic is at your site at peak times?

Do you keep have server log, right?
Could not analyze server log (using an analyzer script or software) to see how much the traffic really is?

I mean something like: how many page hits/second are requested?
Even an approximate value would help a bit. E.g. less than 10 hits/sec, or more than 50 hits/sec...
(not that I know how much LSQL can handle, but I'm sure these traffic data can get us closer to the problem)

You may also look in the server log, which scripts are accessed the most times (a top 3 script list would be useful).

Best regards,
Webmaster33


Paid Support
from Webmaster33. Expert in Perl programming & Gossamer Threads applications. (click here for prices)
Webmaster33's products (upd.2004.09.26) | Private message | Contact me | Was my post helpful? Donate my help...
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
my $.02: Maybe divide the static pages from the dynamic pages into 2 seperate servers...

You won't know anything if you don't analyze your traffic logs...
Quote Reply
Re: [BM] Server Overloaded - How to scale up? In reply to
In Reply To:
my $.02: Maybe divide the static pages from the dynamic pages into 2 seperate servers...

You won't know anything if you don't analyze your traffic logs...

Thanks for the suggestion, am trying to follow this up now in another thread.
Quote Reply
Re: [webmaster33] Server Overloaded - How to scale up? In reply to
In Reply To:
Could not give us at least idea how much traffic is at your site at peak times?

Do you keep have server log, right?
Could not analyze server log (using an analyzer script or software) to see how much the traffic really is?

I mean something like: how many page hits/second are requested?
Even an approximate value would help a bit. E.g. less than 10 hits/sec, or more than 50 hits/sec...
(not that I know how much LSQL can handle, but I'm sure these traffic data can get us closer to the problem)

You may also look in the server log, which scripts are accessed the most times (a top 3 script list would be useful).

Thanks for the info. Mainly the problem is with page.cgi and search.cgi. Absolute figures won't really help overcome the fact that traffic is overwhelming the server.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Just would like to know (from the traffic numbers), if that's really the case when very high traffic causes the problems or probably there is another kind of problem. Is your traffic secret?

Usually using normal cgi IMO at least 5-10 hits/sec should work fine, while using mod_perl at least 20-50 hits/sec may work fine. Again IMO. I have no experience with LSQL under very high traffic, so I can't estimate the performance LSQL can really handle. But at least we can see, if really reaches these limits or not.

Finally, definitely the more memory is better (if your server is swapping to disk a lot, you can achieve more performance by adding memory).
Would be fine to know how much memory needs each session for eg. 15.000 links. IMO, needs at least 10-20 Mb for each session, maybe more. On WinXP using about 2000 links, LSQL 2.1.1 memory usage is going up to 9.5Mb / session for page.cgi, dynamic mode, cgi execution (no mod perl).
So for 20 sessions per sec you need at least 200-300 Mb free memory, or more.

Best regards,
Webmaster33


Paid Support
from Webmaster33. Expert in Perl programming & Gossamer Threads applications. (click here for prices)
Webmaster33's products (upd.2004.09.26) | Private message | Contact me | Was my post helpful? Donate my help...
Quote Reply
Re: [webmaster33] Server Overloaded - How to scale up? In reply to
In Reply To:
Just would like to know (from the traffic numbers), if that's really the case when very high traffic causes the problems or probably there is another kind of problem. Is your traffic secret?

Usually using normal cgi IMO at least 5-10 hits/sec should work fine, while using mod_perl at least 20-50 hits/sec may work fine. Again IMO. I have no experience with LSQL under very high traffic, so I can't estimate the performance LSQL can really handle. But at least we can see, if really reaches these limits or not.

Finally, definitely the more memory is better (if your server is swapping to disk a lot, you can achieve more performance by adding memory).
Would be fine to know how much memory needs each session for eg. 15.000 links. IMO, needs at least 10-20 Mb for each session, maybe more. On WinXP using about 2000 links, LSQL 2.1.1 memory usage is going up to 9.5Mb / session for page.cgi, dynamic mode, cgi execution (no mod perl).
So for 20 sessions per sec you need at least 200-300 Mb free memory, or more.

Hi,

Traffic really is the issue here. It's fine most of the time, but at peak times the server gets overloaded and we have angry emails coming in. Not good. In Alexa we're currently at about 15,000. So, not doing badly.

Following everyone's advice, we have decided to double the RAM to 1gb, up the processor, and install dual Mod-Perl and modproxy apache servers. Should work nicely.

Thanks for your help.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
I would welcome your further info how the changes affected the LSQL performance, probably others would also welcome. Would much more welcome numeric traffic values, just to know what is the maximum performance I can expect from LSQL using such config.

Thanks,
Webmaster33
Quote Reply
Re: [webmaster33] Server Overloaded - How to scale up? In reply to
Sure, will try to post how I get on. Thanks for your help :)
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Well most of this conversation is beyond me, but I thought I had read somewhere previously that you can run into similar problems if the debug value as found in Setup->Misc is set to 1 or higher, as it's writing the queries and more to the server logs. It's probably not your problem, but thought I'd mention it in case.

peace.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
Hi,

Just some thoughts on this subject.

First off, don't do anything until you understand the problem. It's worth getting a consultant (or us) to spend half an hour to look at your system and identify where the bottleneck is. There could be a number of possible reasons such as: a bad global that is doing an unoptimized sql call, a misconfigured mod_perl/apache server, a single cgi that could be optimized to be faster, insufficent hardware, missing perl modules, etc. It's important to get someone to determine exactly what the bottleneck is, otherwise you can be improving things needlessly (splittling a site onto two servers when really it could have been handled fine on one for example).

Some common ways to scale your Links SQL install as you get more and more traffic:

1. Move to mod_perl. Set this up properply as a dual apache solution, don't use a single apache with mod_perl enabled or you will quickly run your machine out of memory. This will be your largest performance improvement, and allow you to get more traffic out of a single machine.

2. Move mysql to a separate machine. This would only be recommended on a large directory (> 100,000 links), as small directories do not use much in the way of mysql resources.

3. Move mod_perl to a separate machine. Leave a single machine to handle all image, html requests, and setup apache to proxy all mod_perl requests to the second machine seamlessly.

Hope this helps, and if you have any questions, don't hesitate to ask.

Cheers,

Alex
--
Gossamer Threads Inc.
Quote Reply
Re: [cwi] Server Overloaded - How to scale up? In reply to
In Reply To:
Sure, will try to post how I get on. Thanks for your help :)

For anyone interested, I emailed Gossamer Threads about this, had an email back, paid, and within an hour had the dual_apache set up installed, with absolutely huge performance improvements. Everything really flies now. Amazing speed improvement. Times which used to send server load up to 30 and above now don't even tax the server. Dynamic pages load as fast as HTML pages (no noticeable difference).

Also, while GT were on my server, they also managed to reduce the footprint of an apache process down from 8mb to about 2mb.

I had thought about getting the hosting techs to do it, but GT also put in custom scripting specific to a Links SQL set up, which they would have never been able to do.

So, very recommended, and very much cheaper than getting a hardware upgrade.