Mailing List Archive: ModPerl: ModPerl

print throwing intermittent Segfaults

Index | Next | Previous | View Threaded

denis.banovic at ncm

Nov 21, 2009, 1:43 AM

Post #1 of 6 (1550 views)
Permalink

print throwing intermittent Segfaults

Hi Everybody,

I'm having big problems with mod_perl throwing intermittent Segmentation faults our production machines on RHEL 4 & 5.
To be able to produce a core dump on this segfaults I've installed mod_dumpcore from this tutorial:
http://mituzas.lt/2009/09/26/getting-apache-core-dumps-in-linux/

gdb /usr/sbin/httpd core.1 produces following output:

#0 0x00b29f4b in XS_Apache__RequestRec_content_type () from /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/Apache/RequestRec/RequestRec.so

and very rarely ( 1 in 15 )

#0 0x003830b9 in apr_palloc () from /usr/lib/libapr-0.so.0

The content-type is set by
$r->content_type("text/html; charset=iso-8859-1") but this is not what is causing him to segfault...

By try and error I've figured out that the segfault happens when I do a
$r->print($mypagecontent);

I've even tried to do a
unless($r->connection->aborted) {
$r->print($mypagecontent);
}
but this didn't help either.

The segfault happens randomly, between 30 and 250 mod_perl requests. There is no specific request URL or script that causes him to segfault, it just happens after some time.
More load on the server means more segfaults.

From my Apache Config:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 200
MaxRequestsPerChild 15
</IfModule>

There are some additional Perl Modules that I've build from CPAN:
Compress-Zlib-2.004
Digest-MD5-2.39
Email-MIME-1.861
Email-MIME-ContentType-1.014
Email-MIME-Encodings-1.311
Email-Simple-2.004
Encode-Detect-1.01
ExtUtils-CBuilder-0.23
File-Slurp-9999.12
IO-Compress-Zlib-2.004
MIME-Base64-3.07
MIME-Types-1.24
Module-Build-0.2808
Pod-Escapes-1.04
Pod-Simple-3.07
String-Similarity-1.03
Template-Plugin-XML-Escape-0.02
Test-Pod-1.26
Test-Simple-0.80

Has anyone a hint where to start looking and what to do next to figure out why this segfault is happening?

Thanks

Denis

dietbuddha at gmail

Nov 21, 2009, 10:27 AM

Post #2 of 6 (1489 views)
Permalink

Re: print throwing intermittent Segfaults [In reply to]

This is the list of stuff I usually start with when I get a problem
that doesn't seem to be tied to a particular code path.

* code path - perhaps a particular code path is only being exercised
rarely, and it has a bug
* forking - when child dies, all open descriptors in it's name space
also get closed
* eval - always a good thing to look at when weird things happen
* persistancy - globals, closures, persistant objects,
serialized/restored objects, shared memory, shared objects (between
processes)

-wjt

denis.banovic at ncm

Nov 23, 2009, 12:27 AM

Post #3 of 6 (1472 views)
Permalink

AW: print throwing intermittent Segfaults [In reply to]

Hi Willian,

Thanks for your checklist, I've run through it, segfaults still there...
Right now it takes less then a minute from apache restart to the first segfault.
This is from the error_log from the RedHat 5 Production machine:

Apache2::RequestIO::print: (103) Software caused connection abort at

The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help?
Or has someone another hint?

Thanks

Denis

-----Ursprüngliche Nachricht-----
Von: William T [mailto:dietbuddha [at] gmail]
Gesendet: Samstag, 21. November 2009 19:28
An: Denis Banovic
Cc: modperl [at] perl
Betreff: Re: print throwing intermittent Segfaults

This is the list of stuff I usually start with when I get a problem that doesn't seem to be tied to a particular code path.

* code path - perhaps a particular code path is only being exercised rarely, and it has a bug
* forking - when child dies, all open descriptors in it's name space also get closed
* eval - always a good thing to look at when weird things happen
* persistancy - globals, closures, persistant objects, serialized/restored objects, shared memory, shared objects (between
processes)

-wjt

--------------------------------------------------------------------------------
Von: Denis Banovic [mailto:denis.banovic [at] ncm]
Gesendet: Samstag, 21. November 2009 10:43
An: modperl [at] perl
Betreff: print throwing intermittent Segfaults

Hi Everybody,

I'm having big problems with mod_perl throwing intermittent Segmentation faults our production machines on RHEL 4 & 5.
To be able to produce a core dump on this segfaults I've installed mod_dumpcore from this tutorial:
http://mituzas.lt/2009/09/26/getting-apache-core-dumps-in-linux/

gdb /usr/sbin/httpd core.1 produces following output:

#0 0x00b29f4b in XS_Apache__RequestRec_content_type () from /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/Apache/RequestRec/RequestRec.so

and very rarely ( 1 in 15 )

#0 0x003830b9 in apr_palloc () from /usr/lib/libapr-0.so.0

The content-type is set by
$r->content_type("text/html; charset=iso-8859-1") but this is not what is causing him to segfault...

By try and error I've figured out that the segfault happens when I do a
$r->print($mypagecontent);

I've even tried to do a
unless($r->connection->aborted) {
$r->print($mypagecontent);
}
but this didn't help either.

The segfault happens randomly, between 30 and 250 mod_perl requests. There is no specific request URL or script that causes him to segfault, it just happens after some time.
More load on the server means more segfaults.

From my Apache Config:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 200
MaxRequestsPerChild 15
</IfModule>

There are some additional Perl Modules that I've build from CPAN:
Compress-Zlib-2.004
Digest-MD5-2.39
Email-MIME-1.861
Email-MIME-ContentType-1.014
Email-MIME-Encodings-1.311
Email-Simple-2.004
Encode-Detect-1.01
ExtUtils-CBuilder-0.23
File-Slurp-9999.12
IO-Compress-Zlib-2.004
MIME-Base64-3.07
MIME-Types-1.24
Module-Build-0.2808
Pod-Escapes-1.04
Pod-Simple-3.07
String-Similarity-1.03
Template-Plugin-XML-Escape-0.02
Test-Pod-1.26
Test-Simple-0.80

Has anyone a hint where to start looking and what to do next to figure out why this segfault is happening?

Thanks

Denis

aw at ice-sa

Nov 23, 2009, 12:46 AM

Post #4 of 6 (1481 views)
Permalink

Re: AW: print throwing intermittent Segfaults [In reply to]

Denis Banovic wrote:
> Hi Willian,
>
> Thanks for your checklist, I've run through it, segfaults still there...
> Right now it takes less then a minute from apache restart to the first segfault.
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help?
> Or has someone another hint?
>
Just my grain of salt : in my own experience, 99% of the "segfault"
cases I have encountered, was when Apache or Perl tried to run a piece
of code not meant for this machine (such as a library meant for another
machine or another OS version).
Maybe one of the modules you are using installed a wrong library ?
In that sense, the guys from rackspace may be right, although I believe
that the CPAN modules don't generally contain object-code libraries, or
else they do compile them at installation.
So maybe it is a library from the RHEL repository which is wrong.

morten.bjornsvik at experian-da

Nov 23, 2009, 1:15 AM

Post #5 of 6 (1475 views)
Permalink

RE: AW: print throwing intermittent Segfaults [In reply to]

Hi

I've had a similar error I "fixed" it by adding an eval block around the
offending code which was tracked back to MASON.

http://rt.cpan.org/Public/Bug/Display.html?id=49031

We compile everything from scratch apache,perl,mod_perl,mason, all modules
by an automated build script.

Earlier when we run on mod_perl1.99 and the redhat stack it worked fine.
But then we had other worries :-)

--
Morten Bjoernsvik, Developer, Decision Analytics

-----Original Message-----
From: André Warnier [mailto:aw [at] ice-sa]
Sent: 23. november 2009 09:46
To: mod_perl list
Subject: Re: AW: print throwing intermittent Segfaults

Denis Banovic wrote:
> Hi Willian,
>
> Thanks for your checklist, I've run through it, segfaults still there...
> Right now it takes less then a minute from apache restart to the first segfault.
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help?
> Or has someone another hint?
>
Just my grain of salt : in my own experience, 99% of the "segfault"
cases I have encountered, was when Apache or Perl tried to run a piece
of code not meant for this machine (such as a library meant for another
machine or another OS version).
Maybe one of the modules you are using installed a wrong library ?
In that sense, the guys from rackspace may be right, although I believe
that the CPAN modules don't generally contain object-code libraries, or
else they do compile them at installation.
So maybe it is a library from the RHEL repository which is wrong.

khansen at charityweb

Dec 11, 2009, 11:08 AM

Post #6 of 6 (1374 views)
Permalink

Re: AW: print throwing intermittent Segfaults [In reply to]

Hello,

I started to see the following when I setup a new server on CentOS 5.4
and installed perl modules from CPAN on Dec. 4:

Denis Banovic wrote:
> This is from the error_log from the RedHat 5 Production machine:
>
> Apache2::RequestIO::print: (103) Software caused connection abort at
>
> The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help?
> Or has someone another hint?
>
>
I do not see this on another server with CentOS 5.2 and perl modules
primarily built Feb 09.

I think something has changed in either CentOS, Apache, or the perl
modules I use in the interim.

I think it has something to do with Apache children being killed either
because of size or request limit reached.

So, Denis, have you tried adjusting these:

> >From my Apache Config:
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 200
> MaxRequestsPerChild 15
> </IfModule>
>
>
The MaxRequetsPerChild especially seems awfully small.

Plus, you say you see the problem happening every 30 to 250 requests.
(StartServers)8 * (MaxRequestsPerChild)15 = 120. This sounds like
roughtly around the time you'd expect Apache children to die.

I had MaxRequests set at 2000 and saw the problem intermittently. I'm
going to set MaxRequests to 0 to turn it off and see if it goes away.

I wonder if something has changed in Apache recently? Or, mod_perl?

Both servers are running Apache 2.2.3. One is running mod_perl 2.0.2 and
the other one is 2.0.4. Judging on the version numbers...could this be a
bug in 2.0.4?

Take care,

Kurt Hansen

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads