Mailing List Archive: ModPerl: ModPerl

Server hangs after 502 request

Index | Next | Previous | View Threaded

rs at plusw

Oct 24, 2009, 12:39 AM

Post #1 of 8 (197 views)
Permalink

Server hangs after 502 request

Hi,

I have a server running, configured with a lightweiht frontend apache
and a backend mod_perl apache.
Starting some weeks ago the server sometimes hangs with an out of
memory problem.
(Kernel panic, no killable process found )
I currently have no idea where that problems comes from, the numer of
backend apache is set to 15, so this should not be a problem.
Yet, what I'have found, that the last request logged in the frontend
aache log looks like :

"GET / HTTP/1.1" 502 465 "http://googleads.g.doubleclick.net/pagead/ads?client=ca-pub-8011132902902466&format=fp_al_lp&output=html&h=15&w=728&lmt=1256317728&channel=0471980043%2B7740394182&ea=0&flash=10.0.32.18&url=http%3A%2F%2Fwww.schulferien.org%2FBaden-Wurttemb_%2Fbaden-wurttemb_.html&dt=1256317728847&correlator=1256317728850&frm=0&ga_vid=795893907.1228664705&ga_sid=1256317729&ga_hid=250028746&ga_fc=1&u_tz=120&u_his=13&u_java=1&u_h=800&u_w=1280&u_ah=770&u_aw=1280&u_cd=32&u_nplug=0&u_nmime=0&biw=1276&bih=569&ref=http%3A%2F%2Fwww.google.de%2Fsearch%3Fhl%3Dde%26rlz%3D1T4SNYK_deDE295DE295%26q%3Dferien%2Bbaden%2Bw%25C3%25BCrttemberg%2B2010%26meta%3D%26aq%3D2%26oq%3Dferien%2Bbaden%2B&fu=0&ifi=1&dtd=271&kw_type=radlink&prev_fmts=728x15_0ads_al_s&rt=ChBK4eMkAAKqhwrMjcLbYhQ7Eg5KYWhyZXNrYWxlbmRlchoIyMwiMKeTu80oAVITCNvXmaHU050CFVGIzAodgFzQag&hl=de&kw0=Ferienkalender+NRW&kw1=Schulferien+2010&kw2=Jahreskalender&kw3=Kalender+Hessen+2008&kw4=Kalender+Schulferien&okw=Jahreskalender
" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0;
GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR
1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"

It always has googleads as referer with a very long refer string and
what das the 502 status mean?

Does anybody have some idea what is happening ?
Can I simulate this request with LWP ?

Rolf Schaufelberger

aw at ice-sa

Oct 24, 2009, 3:13 AM

Post #2 of 8 (185 views)
Permalink

Re: Server hangs after 502 request [In reply to]

Rolf Schaufelberger wrote:
>
> It always has googleads as referer with a very long refer string and
> what das the 502 status mean?
>
http://lmgtfy.com?q=http+status+502

rs at plusw

Oct 25, 2009, 10:44 AM

Post #3 of 8 (178 views)
Permalink

Re: Server hangs after 502 request [In reply to]

Am 24.10.2009 um 12:13 schrieb André Warnier:

> Rolf Schaufelberger wrote:
>> It always has googleads as referer with a very long refer string
>> and what das the 502 status mean?
> http://lmgtfy.com?q=http+status+502

Ok, I can read what's written there, but what does it mean ?
I still have no idea, what leads to that error. As far as I
understood this, my backend server send something else but no HTTP
message to the frontend server.
But why?

Rolf Schaufelberger

aw at ice-sa

Oct 25, 2009, 11:36 AM

Post #4 of 8 (178 views)
Permalink

Re: Server hangs after 502 request [In reply to]

Rolf Schaufelberger wrote:
>
> Am 24.10.2009 um 12:13 schrieb André Warnier:
>
>> Rolf Schaufelberger wrote:
>>> It always has googleads as referer with a very long refer string and
>>> what das the 502 status mean?
>> http://lmgtfy.com?q=http+status+502
>
>
> Ok, I can read what's written there, but what does it mean ?
> I still have no idea, what leads to that error. As far as I understood
> this, my backend server send something else but no HTTP message to the
> frontend server.
> But why?
>
Rolf,

502 Bad Gateway
The server was acting as a gateway or proxy and received an invalid
response from the downstream server.

It means that your front-end server received this request ("GET /"),
from some client, which apparently was at the time displaying a page
which itself had been obtained from the long link that you see in the
message. (It does not mean that the request itself came from a Google
server; it just means that on some page of Google somewhere, there is a
link that points to your site, and that someone clicked on it).

Now, we don't know what the configuration of your front-end and back-end
servers are, nor what is exactly proxied to your back-end server when
the front-end server receives a request for "/".
But the error code 502 that is in the log (see description above) seems
to indicate that your front-end server passed this request somehow to a
back-end server, and that it received an invalid response from the
back-end server.
Why that is, is also not known to us, and quite impossible to figure out
with the information you provided.

If I had to make a guess, I would say that the problem has nothing to do
with this request in particular, and that your front-end server is just
configured with too many children or threads, compared to the physical
memory it really has. So when many requests come in and it tries to
fork more children/threads to handle them, it runs out of memory.
Reducing MaxClients may be a first option.

About reproducing this access with LWP, sure :
lwp-request -m GET http://your-server-name/
but that is not going to tell you much.

rs at plusw

Oct 25, 2009, 2:06 PM

Post #5 of 8 (176 views)
Permalink

Re: Server hangs after 502 request [In reply to]

Am 25.10.2009 um 19:36 schrieb André Warnier:

> Rolf Schaufelberger wrote:
>> Am 24.10.2009 um 12:13 schrieb André Warnier:
>>> Rolf Schaufelberger wrote:
>>>> It always has googleads as referer with a very long refer string
>>>> and what das the 502 status mean?
>>> http://lmgtfy.com?q=http+status+502
>> Ok, I can read what's written there, but what does it mean ?
>> I still have no idea, what leads to that error. As far as I
>> understood this, my backend server send something else but no HTTP
>> message to the frontend server.
>> But why?
> Rolf,
>
> 502 Bad Gateway
> The server was acting as a gateway or proxy and received an
> invalid response from the downstream server.
>
> It means that your front-end server received this request ("GET /"),
> from some client, which apparently was at the time displaying a page
> which itself had been obtained from the long link that you see in
> the message. (It does not mean that the request itself came from a
> Google server; it just means that on some page of Google somewhere,
> there is a link that points to your site, and that someone clicked
> on it).
>
> Now, we don't know what the configuration of your front-end and back-
> end servers are, nor what is exactly proxied to your back-end server
> when the front-end server receives a request for "/".
> But the error code 502 that is in the log (see description above)
> seems to indicate that your front-end server passed this request
> somehow to a back-end server, and that it received an invalid
> response from the back-end server.
> Why that is, is also not known to us, and quite impossible to figure
> out with the information you provided.
>
So far I agree, and a request to / is just normal and handed over
to the backend server.

> If I had to make a guess, I would say that the problem has nothing
> to do with this request in particular, and that your front-end
> server is just configured with too many children or threads,
> compared to the physical memory it really has. So when many
> requests come in and it tries to fork more children/threads to
> handle them, it runs out of memory.
> Reducing MaxClients may be a first option.
>

I've already dona this, the MaxClients for the backend ist set to 10
and there is not much traffic on that server. I had another server
running with 200 frontend and 30 backendserver with much less RAM.
Actually I don't even know for sure, if apache is causing these
problems, It's just the only thing running there,( tha database runs
on another server).
I' just desperately searching for anything that couses thtes problems.

> About reproducing this access with LWP, sure :
> lwp-request -m GET http://your-server-name/
> but that is not going to tell you much.

Of course, but what I meant was, can I tell LWP to pass the same
referer string I found in my logfile, to simultate exactly that
request ?

Rolf Schaufelberger

aw at ice-sa

Oct 25, 2009, 2:46 PM

Post #6 of 8 (175 views)
Permalink

Re: Server hangs after 502 request [In reply to]

Rolf Schaufelberger wrote:
...

> Of course, but what I meant was, can I tell LWP to pass the same
> referer string I found in my logfile, to simultate exactly that request ?
>
I'm sure you can. That "referrer" is just a HTTP header ("Referer:"
with only one r I think), and LWP allows you to add headers to the request.
lwp-request -h

I doubt this will help you however. If you examine your access log, you
probably have lots of such kinds of request, which do not necessarily
bring down your server.

Maybe an easier way :
- on one of your servers (suppose it is called
"yourserver.yourcompany.com"), create a really long path under the
document root and put a html page in it.
Like :
/var/www/htdocs/some/very/long/path/to/a/document/page.html
- in that html page, put a link to your front-end server
- call up that page in a browser, and click on the link.
- then look in your access log

The browser, when you click on the link in the page, sends a request to
your front-end server.
In that request, it will add a header :
Referer:
http://yourserver.yourcompany.com/some/very/long/path/to/a/document/page.html

That's what you see in the log.

torsten.foertsch at gmx

Oct 26, 2009, 4:12 AM

Post #7 of 8 (171 views)
Permalink

Re: Server hangs after 502 request [In reply to]

On Sat 24 Oct 2009, Rolf Schaufelberger wrote:
> Starting some weeks ago the server sometimes hangs with an out of
> memory problem.

Assuming you are running on linux the following sysctls may help to find
the culprit.

vm.overcommit_memory=2
vm.overcommit_ratio=90

By default (overcommit_memory=0) when a process needs more memory linux
simply says "okay here you are" no matter if the memory is currently
available or not. This is based on the assumption that most processes
do not use all memory they allocate at all. Later on, when the process
really accesses the memory a page fault is generated and linux only
then allocates the memory for the process. But now it is too late to
signal the process out-of-memory. So, linux somehow has to obtain
memory no matter what. So, in short-of-memory conditions linux starts
the OOM killer. It implements some heuristic that chooses some "best
fitting" processes to be killed. These "best fitting" processes may be
totally unrelated to the original problem.

I had once a case where a perl program processed mailbox files (using
Mail::Box) on a box where a postgres database ran. Unfortunately
Mail::Box reads in the whole mailbox file. Normally our mailbox files
were about 1-10mb and the program had worked for years. But suddenly we
had one of >1Gb. Instead of signaling out-of-memory to the perl process
linux killed the postgres database.

Make sure you have enough swap space (at least the RAM size) before
experimenting with those sysctls.

Torsten

--
Need professional mod_perl support?
Just hire me: torsten.foertsch[at]gmx.net

rs at plusw

Oct 26, 2009, 5:15 AM

Post #8 of 8 (170 views)
Permalink

Re: Server hangs after 502 request [In reply to]

Hi, Tortsen,

Am 26.10.2009 um 12:12 schrieb Torsten Foertsch:

> On Sat 24 Oct 2009, Rolf Schaufelberger wrote:
>> Starting some weeks ago the server sometimes hangs with an out of
>> memory problem.
>
> Assuming you are running on linux the following sysctls may help to
> find
> the culprit.
>
> vm.overcommit_memory=2
> vm.overcommit_ratio=90
>

how about setting PERL_RLIMIT_DATA ?
see (http://perl.apache.org/docs/2.0/api/Apache2/Resource.html)

Would that work ?

> By default (overcommit_memory=0) when a process needs more memory
> linux
> simply says "okay here you are" no matter if the memory is currently
> available or not. This is based on the assumption that most processes
> do not use all memory they allocate at all. Later on, when the process
> really accesses the memory a page fault is generated and linux only
> then allocates the memory for the process. But now it is too late to
> signal the process out-of-memory. So, linux somehow has to obtain
> memory no matter what. So, in short-of-memory conditions linux starts
> the OOM killer. It implements some heuristic that chooses some "best
> fitting" processes to be killed. These "best fitting" processes may be
> totally unrelated to the original problem.
>
> I had once a case where a perl program processed mailbox files (using
> Mail::Box) on a box where a postgres database ran. Unfortunately
> Mail::Box reads in the whole mailbox file. Normally our mailbox files
> were about 1-10mb and the program had worked for years. But suddenly
> we
> had one of >1Gb. Instead of signaling out-of-memory to the perl
> process
> linux killed the postgres database.
>
> Make sure you have enough swap space (at least the RAM size) before
> experimenting with those sysctls.
>
> Torsten
>
> --
> Need professional mod_perl support?
> Just hire me: torsten.foertsch[at]gmx.net

Mit freundlichen Grüßen
Rolf Schaufelberger

plusW GmbH
Stuttgarter Str. 26 Tel. 07183 30 21 36
73635 Rudersberg Fax 07183 30 21 85

www.plusw.de
www.mypixler.com
www.calendrino.de

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com