Mailing List Archive: Apache: Users

Tons of Segfaults in Apache-Prefork 2.2.4

Index | Next | Previous | View Threaded

markus.meyer at koeln

Apr 17, 2008, 3:13 AM

Post #1 of 6 (4364 views)
Permalink

Tons of Segfaults in Apache-Prefork 2.2.4

Hi all,

we experience a lot of threads segfaulting with our Apaches. There are
about 300 dying threads per day. The error in syslog is the following:

Apr 16 23:51:41 foo-bar kernel: httpd[24629]: segfault at
00007fffae529f98 rip 00002b2ffd6408b7 rsp 00007fffae529ea0 error 6

I wish I could give you some kind of trace or core file. But we don't
get a core file since only threads die and not the root-process. I also
tried to strace the Apache but this slowed the server down too much. And
the error appears only on the live system with a very high load. Any
ideas welcome here.

At the same time we get broken TCP packets with a wrong window size
which looks like this:

Apr 17 00:14:17 foo-bar kernel: TCP: Treason uncloaked! Peer
84.175.98.78:2637/80 shrinks window 1785243455:1785255071. Repaired.

The information I found on the net are very contradictory, so I can't
estimate the impact on the system.

The box is used for dynamic webpages on PHP. Below are some system
information.
Sorry, this is not very much information but I'm still at the beginning
of my Apache leraning curve.

Thx and cheers,
---

Markus Meyer

Some system info:
# uname -a
Linux wkw-fra2 2.6.18-4-amd64 #1 SMP Fri May 4 00:37:33 UTC 2007 x86_64
GNU/Linux

from httpd.conf
================
ServerLimit 1800
StartServers 250
MinSpareServers 64
MaxSpareServers 128
MaxClients 1200

Timeout 60

KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 128
MaxRequestsPerChild 10000

LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule env_module modules/mod_env.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule asis_module modules/mod_asis.so
LoadModule info_module modules/mod_info.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule actions_module modules/mod_actions.so
LoadModule php5_module modules/libphp5.so

CoreDumpDirectory /var/log/apache/core
UseCanonicalName On
TypesConfig /etc/apache2/conf-prefork/mime.types
DefaultType text/plain
MIMEMagicFile /etc/apache2/conf-prefork/magic
HostnameLookups Off

PHP version
============
# ./php --version
PHP 5.2.3 (cli) (built: Jun 17 2007 21:46:02)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
with eAccelerator v0.9.5.2, Copyright (c) 2004-2006 eAccelerator,
by eAccelerator

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe [at] httpd
" from the digest: users-digest-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

markus.meyer at koeln

Apr 17, 2008, 7:34 AM

Post #2 of 6 (4322 views)
Permalink

Re: Tons of Segfaults in Apache-Prefork 2.2.4 [In reply to]

Markus Meyer schrieb:
> Hi all,
>
> we experience a lot of threads segfaulting with our Apaches. There are
> about 300 dying threads per day. The error in syslog is the following:
>
> Apr 16 23:51:41 foo-bar kernel: httpd[24629]: segfault at
> 00007fffae529f98 rip 00002b2ffd6408b7 rsp 00007fffae529ea0 error 6

BTW: The same appears with the actual Apache 2.2.8 too.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe [at] httpd
" from the digest: users-digest-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

nick at webthing

Apr 17, 2008, 8:25 AM

Post #3 of 6 (4281 views)
Permalink

Re: Tons of Segfaults in Apache-Prefork 2.2.4 [In reply to]

On Thu, 17 Apr 2008 12:13:04 +0200
Markus Meyer <markus.meyer [at] koeln> wrote:

> Hi all,
>
> we experience a lot of threads segfaulting with our Apaches. There
> are about 300 dying threads per day. The error in syslog is the
> following:
>
> Apr 16 23:51:41 foo-bar kernel: httpd[24629]: segfault at
> 00007fffae529f98 rip 00002b2ffd6408b7 rsp 00007fffae529ea0 error 6
>
> I wish I could give you some kind of trace or core file. But we don't
> get a core file since only threads die and not the root-process. I

Hmmm? Your subject line says prefork. With prefork, there are
no threads. Anyway, no matter.

You should get cores from a segfault. Have you enabled them?
(See Coredumpdirectory, and check limitations imposed by your
operating system and shell).

> also tried to strace the Apache but this slowed the server down too
> much. And the error appears only on the live system with a very high
> load. Any ideas welcome here.

We recently (for 2.2.9) fixed a bug that fits that description:
http://issues.apache.org/bugzilla/show_bug.cgi?id=44402
But that's not relevant to prefork.

> At the same time we get broken TCP packets with a wrong window size
> which looks like this:

Um, sounds like a problem in the network, below the level of apache.

> The box is used for dynamic webpages on PHP. Below are some system
> information.

PHP is always a prime suspect for this kind of problem.
If you're using mod_php, make sure you really are running prefork,
as PHP+threads is a classic recipe for random segfaults.

> Sorry, this is not very much information but I'm still at the
> beginning of my Apache leraning curve.

Contradictory information is more of a problem.

--
Nick Kew

Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe [at] httpd
" from the digest: users-digest-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

markus.meyer at koeln

Apr 17, 2008, 9:16 AM

Post #4 of 6 (4280 views)
Permalink

Re: Tons of Segfaults in Apache-Prefork 2.2.4 [In reply to]

Nick Kew schrieb:
> On Thu, 17 Apr 2008 12:13:04 +0200

> Hmmm? Your subject line says prefork. With prefork, there are
> no threads. Anyway, no matter.

Stupid me. This happens when having developers crying in your ear all
the time ;)

> You should get cores from a segfault. Have you enabled them?
> (See Coredumpdirectory, and check limitations imposed by your
> operating system and shell).

I enabled CoreDumpDirectory in httpd.conf, made a "ulimit -c unlimited"
and started Apache. When I had a segfault there was no corefile in the
specified directory and Apache was still running. Only the forked
process was gone and a new was spawned. After two days or sometimes a
week Apache gets really slow and can handle only a fourth or less of
it's usual traffic.

> We recently (for 2.2.9) fixed a bug that fits that description:
> http://issues.apache.org/bugzilla/show_bug.cgi?id=44402
> But that's not relevant to prefork.

Currently the page is not reachable.

> Um, sounds like a problem in the network, below the level of apache.

I'm looking into this too.

> PHP is always a prime suspect for this kind of problem.
> If you're using mod_php, make sure you really are running prefork,
> as PHP+threads is a classic recipe for random segfaults.

*lol*

> Contradictory information is more of a problem.

I do my best.

Cheers,

Markus Meyer

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe [at] httpd
" from the digest: users-digest-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

joshua at slive

Apr 17, 2008, 10:00 AM

Post #5 of 6 (4288 views)
Permalink

Re: Tons of Segfaults in Apache-Prefork 2.2.4 [In reply to]

On Thu, Apr 17, 2008 at 12:16 PM, Markus Meyer <markus.meyer [at] koeln> wrote:

> I enabled CoreDumpDirectory in httpd.conf, made a "ulimit -c unlimited" and
> started Apache. When I had a segfault there was no corefile in the specified
> directory and Apache was still running. Only the forked process was gone and
> a new was spawned. After two days or sometimes a week Apache gets really
> slow and can handle only a fourth or less of it's usual traffic.

It is expected that apache should still be running, but you should be
able to get a core file. Without the core file, it is almost
impossible to debug a segfault. There is a little more guidance here:
http://httpd.apache.org/dev/debugging.html#crashes

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe [at] httpd
" from the digest: users-digest-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

markus.meyer at koeln

Apr 18, 2008, 6:12 AM

Post #6 of 6 (4271 views)
Permalink

Re: Tons of Segfaults in Apache-Prefork 2.2.4 [In reply to]

Joshua Slive schrieb:

Hi Joshua,

> http://httpd.apache.org/dev/debugging.html#crashes

Somewhere I must have forgot something. I did the same procedure again
and now I got a coredump. Here it is:

Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from
/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/eaccelerator.so...done.
Loaded symbols for
/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/eaccelerator.so
Reading symbols from
/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/memcache-2.1.2.so...done.
Loaded symbols for
/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/memcache-2.1.2.so
Reading symbols from
/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/ssh2.so...done.
Loaded symbols for
/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/ssh2.so
Reading symbols from /usr/local/lib/libssh2.so.1...done.
Loaded symbols for /usr/local/lib/libssh2.so.1
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Core was generated by `/usr/local/apache2-prefork/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
#0 0x00002b8fd9403fdc in zif_memcache_get (ht=<value optimized out>,
return_value=0x2b8fdd94cbd8, return_value_ptr=<value optimized out>,
this_ptr=<value optimized out>,
return_value_used=<value optimized out>) at
/update/memcache-2.1.2/memcache.c:1153
1153 /update/memcache-2.1.2/memcache.c: No such file or directory.
in /update/memcache-2.1.2/memcache.c

Great. I guess I have take a look at the Memcache website.

All the noise and now that. Sorry for the fuzz.

Cheers,

Markus Meyer

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe [at] httpd
" from the digest: users-digest-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads