Mailing List Archive: Apache: Users

Apache 2.4.1- not working (Fatal error... Apache is exiting)

Index | Next | Previous | View Threaded

simran4u2 at gmail

Jun 11, 2012, 3:21 AM

Post #1 of 7 (1635 views)
Permalink

Apache 2.4.1- not working (Fatal error... Apache is exiting)

Hi

I am using apache 2.4.1 with OpenSSL 1.0.1 and modjk 1.2.35 on Red Hat
Enterprise Linux Server release 6.1 :

When running for long time with lot of request\stability Test on Apache
httpd setup i m getting sequence of errors here is details of error logs:

..
[Fri Jun 08 17:19:57.762560 2012] [mpm_worker:notice] [pid 24632:tid
139857001494336] AH00292: Apache/2.4.1 (Unix) OpenSSL/1.0.1 mod_jk/1.2.35
configured -- resuming normal operations
[Fri Jun 08 17:19:57.762591 2012] [core:notice] [pid 24632:tid
139857001494336] AH00094: Command line: '/opt/balancer/bin/httpd'
*** glibc detected *** /opt/balancer/bin/httpd: free(): invalid pointer:
0x00007f32b00083e0 ***
*** glibc detected *** /opt/balancer/bin/httpd: double free or corruption
(fasttop): 0x00007f32b001afa0 ***
*** glibc detected *** /opt/balancer/bin/httpd: double free or corruption
(fasttop): 0x00007f32b001b230 ***
*** glibc detected *** /opt//balancer/bin/httpd: double free or corruption
(fasttop): 0x00007f32a801f400 ***
[Sat Jun 09 05:15:15.534017 2012] [core:notice] [pid 24632:tid
139857001494336] AH00052: child pid 19947 exit signal Segmentation fault
(11)
[Sat Jun 09 05:15:15.534081 2012] [core:notice] [pid 24632:tid
139857001494336] AH00052: child pid 19948 exit signal Segmentation fault
(11)
[Sat Jun 09 05:15:15.536082 2012] [mpm_worker:alert] [pid 20059:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:15:15.536097 2012] [mpm_worker:alert] [pid 20060:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:15:15.536098 2012] [mpm_worker:alert] [pid 20062:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:15:15.536107 2012] [mpm_worker:alert] [pid 20058:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:15:18.538791 2012] [core:notice] [pid 24632:tid
139857001494336] AH00052: child pid 20226 exit signal Segmentation fault
(11)
[Sat Jun 09 05:15:19.541181 2012] [core:notice] [pid 24632:tid
139857001494336] AH00052: child pid 20280 exit signal Segmentation fault
(11)
[Sat Jun 09 05:15:19.542694 2012] [mpm_worker:alert] [pid 20388:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:15:19.542689 2012] [mpm_worker:alert] [pid 20386:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
...
..
.
.

[Sat Jun 09 05:59:41.472983 2012] [mpm_worker:alert] [pid 654:tid
139857001494336] (11)Resource temporarily unavailable: AH00282:
apr_thread_create: unable to create worker thread
[Sat Jun 09 05:59:43.476213 2012] [mpm_worker:alert] [pid 657:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:59:44.477741 2012] [mpm_worker:alert] [pid 661:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:59:44.477886 2012] [mpm_worker:alert] [pid 662:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:59:46.480950 2012] [mpm_worker:alert] [pid 670:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:59:47.482508 2012] [mpm_worker:alert] [pid 676:tid
139856935798528] (11)Resource temporarily unavailable: apr_thread_create:
unable to create worker thread
[Sat Jun 09 05:59:47.482550 2012] [mpm_worker:alert] [pid 677:tid
139857001494336] (11)Resource temporarily unavailable: AH00282:
apr_thread_create: unable to create worker thread
[Sat Jun 09 05:59:49.485424 2012] [mpm_worker:alert] [pid 691:tid
139857001494336] (11)Resource temporarily unavailable: AH00282:
apr_thread_create: unable to create worker thread
[Sat Jun 09 05:59:50.487014 2012] [mpm_worker:alert] [pid 692:tid
139857001494336] (11)Resource temporarily unavailable: AH00282:
apr_thread_create: unable to create worker thread
[Sat Jun 09 05:59:50.487017 2012] [unixd:alert] [pid 693:tid
139857001494336] (11)Resource temporarily unavailable: AH02162: setuid:
unable to change to uid: 500
[Sat Jun 09 05:59:51.487946 2012] [core:alert] [pid 24632:tid
139857001494336] AH00050: Child 693 returned a Fatal error... Apache is
exiting!
[Sat Jun 09 05:59:54.837738 2012] [core:warn] [pid 24632:tid
139857001494336] AH00045: child process 24640 still did not exit, sending a
SIGTERM
[Sat Jun 09 05:59:54.837805 2012] [core:warn] [pid 24632:tid
139857001494336] AH00045: child process 24641 still did not exit, sending a
SIGTERM
[Sat Jun 09 05:59:54.837840 2012] [core:warn] [pid 24632:tid
139857001494336] AH00045: child process 24642 still did not exit, sending a
SIGTERM
[Sat Jun 09 05:59:54.837867 2012] [core:warn] [pid 24632:tid
139857001494336] AH00045: child process 24647 still did not exit, sending a
SIGTERM
[Sat Jun 09 05:59:54.837897 2012] [core:warn] [pid 24632:tid
139857001494336] AH00045: child process 24653 still did not exit, sending a
SIGTERM

Any idea what is issue?

here is server details:
Server version: Apache/2.4.1 (Unix)
Server built: Mar 30 2012 06:12:48
Server's Module Magic Number: 20120211:0
Server loaded: APR 1.4.6, APR-UTIL 1.4.1
Compiled using: APR 1.4.6, APR-UTIL 1.4.1
Architecture: 64-bit
Server MPM: worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/opt//balancer"
-D SUEXEC_BIN="/opt//balancer/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
....
<IfModule worker.c>
ServerLimit 60
StartServers 5
MaxClients 2500
MinSpareThreads 100
MaxSpareThreads 1000
ThreadsPerChild 50
MaxRequestsPerChild 0
</IfModule>

Regards
Harsimran

nick at webthing

Jun 11, 2012, 4:19 AM

Post #2 of 7 (1505 views)
Permalink

Re: Apache 2.4.1- not working (Fatal error... Apache is exiting) [In reply to]

On Mon, 11 Jun 2012 15:51:18 +0530
Harsimranjit singh Kler <simran4u2 [at] gmail> wrote:

> [chop - crash report]

First check: presumably you rebuilt (or re-installed from package)
your everything that's running in httpd, when you moved to 2.4?
I mean, your mod_jk and any other external modules you have?

Second check: is this happening consistently? The first error
appears to be happening at startup, but what about the others?
Is the time random or were these other startups?

> Any idea what is issue?

Are you familiar with gdb (or do you know anyone who is)?
A traceback from a core, or (better) a live run, is the
best diagnostic tool.

Failing that, you might get some more information from Jeff's
diagnostic modules at http://people.apache.org/~trawick/

--
Nick Kew

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

simran4u2 at gmail

Jun 11, 2012, 4:48 AM

Post #3 of 7 (1500 views)
Permalink

Re: Apache 2.4.1- not working (Fatal error... Apache is exiting) [In reply to]

Hi

Yes i rebuild everything for 2.4.All external module openssl,modjk
,apr,pcre etc which are supported and compatibile are upgraded.
Its functional for basic requests and working normal.

Currently it is not happening consistently.

I have setup like(Node1
->httpd+modjk,Node2->worker1(tomcat),worker2(tomcat)).After 12 hour
successfull run Node2 went out of memory and worker1 and worker2 stop
responding.As in logs intial error came at that time when Node2 gone and
after 1 hour its showing apache exiting.So am not sure why worker fail
cause apache httpd to exit?

i am not familiar with gdb and can you provide details of diagnostic tool
to debug this.

On Mon, Jun 11, 2012 at 4:49 PM, Nick Kew <nick [at] webthing> wrote:

> On Mon, 11 Jun 2012 15:51:18 +0530
> Harsimranjit singh Kler <simran4u2 [at] gmail> wrote:
>
> > [chop - crash report]
>
> First check: presumably you rebuilt (or re-installed from package)
> your everything that's running in httpd, when you moved to 2.4?
> I mean, your mod_jk and any other external modules you have?
>
> Second check: is this happening consistently? The first error
> appears to be happening at startup, but what about the others?
> Is the time random or were these other startups?
>
> > Any idea what is issue?
>
> Are you familiar with gdb (or do you know anyone who is)?
> A traceback from a core, or (better) a live run, is the
> best diagnostic tool.
>
> Failing that, you might get some more information from Jeff's
> diagnostic modules at http://people.apache.org/~trawick/
>
>
> --
> Nick Kew
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>
>

simran4u2 at gmail

Jun 11, 2012, 11:00 PM

Post #4 of 7 (1493 views)
Permalink

Re: Apache 2.4.1- not working (Fatal error... Apache is exiting) [In reply to]

http://people.apache.org/~trawick/

This link also not much helpful.

On Mon, Jun 11, 2012 at 5:18 PM, Harsimranjit singh Kler <
simran4u2 [at] gmail> wrote:

> Hi
>
>
> Yes i rebuild everything for 2.4.All external module openssl,modjk
> ,apr,pcre etc which are supported and compatibile are upgraded.
> Its functional for basic requests and working normal.
>
> Currently it is not happening consistently.
>
> I have setup like(Node1
> ->httpd+modjk,Node2->worker1(tomcat),worker2(tomcat)).After 12 hour
> successfull run Node2 went out of memory and worker1 and worker2 stop
> responding.As in logs intial error came at that time when Node2 gone and
> after 1 hour its showing apache exiting.So am not sure why worker fail
> cause apache httpd to exit?
>
>
> i am not familiar with gdb and can you provide details of diagnostic tool
> to debug this.
>
> On Mon, Jun 11, 2012 at 4:49 PM, Nick Kew <nick [at] webthing> wrote:
>
>> On Mon, 11 Jun 2012 15:51:18 +0530
>> Harsimranjit singh Kler <simran4u2 [at] gmail> wrote:
>>
>> > [chop - crash report]
>>
>> First check: presumably you rebuilt (or re-installed from package)
>> your everything that's running in httpd, when you moved to 2.4?
>> I mean, your mod_jk and any other external modules you have?
>>
>> Second check: is this happening consistently? The first error
>> appears to be happening at startup, but what about the others?
>> Is the time random or were these other startups?
>>
>> > Any idea what is issue?
>>
>> Are you familiar with gdb (or do you know anyone who is)?
>> A traceback from a core, or (better) a live run, is the
>> best diagnostic tool.
>>
>> Failing that, you might get some more information from Jeff's
>> diagnostic modules at http://people.apache.org/~trawick/
>>
>>
>> --
>> Nick Kew
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe [at] httpd
>> For additional commands, e-mail: users-help [at] httpd
>>
>>
>

simran4u2 at gmail

Jun 12, 2012, 3:57 AM

Post #5 of 7 (1491 views)
Permalink

Re: Apache 2.4.1- not working (Fatal error... Apache is exiting) [In reply to]

Hi Experts,
I am able to reproduce same problem and logs many time now with ab tool

like:
ab -n 20000 -c 800 https://10.48.11.85:10002/index.html
(I am running lb with ssl on 10002 port)

I stuck on this.
is there any problem in configurations? or hardware problem?.Is there any
thread\memory configuration missing.
please help me out.

On Mon, Jun 11, 2012 at 4:49 PM, Nick Kew <nick [at] webthing> wrote:

> On Mon, 11 Jun 2012 15:51:18 +0530
> Harsimranjit singh Kler <simran4u2 [at] gmail> wrote:
>
> > [chop - crash report]
>
> First check: presumably you rebuilt (or re-installed from package)
> your everything that's running in httpd, when you moved to 2.4?
> I mean, your mod_jk and any other external modules you have?
>
> Second check: is this happening consistently? The first error
> appears to be happening at startup, but what about the others?
> Is the time random or were these other startups?
>
> > Any idea what is issue?
>
> Are you familiar with gdb (or do you know anyone who is)?
> A traceback from a core, or (better) a live run, is the
> best diagnostic tool.
>
> Failing that, you might get some more information from Jeff's
> diagnostic modules at http://people.apache.org/~trawick/
>
>
> --
> Nick Kew
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>
>

trawick at gmail

Jun 12, 2012, 4:37 AM

Post #6 of 7 (1499 views)
Permalink

Re: Apache 2.4.1- not working (Fatal error... Apache is exiting) [In reply to]

On Tue, Jun 12, 2012 at 2:00 AM, Harsimranjit singh Kler
<simran4u2 [at] gmail> wrote:
> �http://people.apache.org/~trawick/
>
> This link also not much helpful.

http://people.apache.org/~trawick/exception_hook.html

Synopsis: If you build Apache httpd with a certain configure option on
Unix, you can have these two modules capture diagnostic information at
the time of a crash.

>
>
> On Mon, Jun 11, 2012 at 5:18 PM, Harsimranjit singh Kler
> <simran4u2 [at] gmail> wrote:
>>
>> Hi
>>
>>
>> Yes i rebuild everything for 2.4.All external module openssl,modjk
>> ,apr,pcre etc which are supported and compatibile are upgraded.
>> Its functional for basic requests and working normal.
>>
>> Currently it is not happening consistently.
>>
>> I have setup like(Node1
>> ->httpd+modjk,Node2->worker1(tomcat),worker2(tomcat)).After 12 hour
>> successfull run Node2 went out of memory and worker1 and worker2 stop
>> responding.As in logs intial error came at that time when Node2 gone�and
>> after 1 hour its showing apache exiting.So am not sure why worker fail cause
>> apache httpd to exit?
>>
>>
>> i am not familiar with gdb and can you provide details of diagnostic tool
>> to debug this.
>>
>> On Mon, Jun 11, 2012 at 4:49 PM, Nick Kew <nick [at] webthing> wrote:
>>>
>>> On Mon, 11 Jun 2012 15:51:18 +0530
>>> Harsimranjit singh Kler <simran4u2 [at] gmail> wrote:
>>>
>>> > [chop - crash report]
>>>
>>> First check: presumably you rebuilt (or re-installed from package)
>>> your everything that's running in httpd, when you moved to 2.4?
>>> I mean, your mod_jk and any other external modules you have?
>>>
>>> Second check: is this happening consistently? �The first error
>>> appears to be happening at startup, but what about the others?
>>> Is the time random or were these other startups?
>>>
>>> > Any idea what is issue?
>>>
>>> Are you familiar with gdb (or do you know anyone who is)?
>>> A traceback from a core, or (better) a live run, is the
>>> best diagnostic tool.
>>>
>>> Failing that, you might get some more information from Jeff's
>>> diagnostic modules at http://people.apache.org/~trawick/
>>>
>>>
>>> --
>>> Nick Kew
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe [at] httpd
>>> For additional commands, e-mail: users-help [at] httpd
>>>
>>
>

--
Born in Roswell... married an alien...
http://emptyhammock.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

simran4u2 at gmail

Jun 22, 2012, 11:51 AM

Post #7 of 7 (1405 views)
Permalink

Re: Apache 2.4.1- not working (Fatal error... Apache is exiting) [In reply to]

On Tue, Jun 12, 2012 at 4:27 PM, Harsimranjit singh Kler <
simran4u2 [at] gmail> wrote:

> Hi Experts,
> I am able to reproduce same problem and logs many time now with ab tool
>
> like:
> ab -n 20000 -c 800 https://10.48.11.85:10002/index.html
> (I am running lb with ssl on 10002 port)
>
> I stuck on this.
> is there any problem in configurations? or hardware problem?.Is there any
> thread\memory configuration missing.
> please help me out.
>
> >I am able to solve this by increasing max user processes value in Red
> hat Linux 6.Changed these values:
>
> /etc/security/ limits.conf
>
> * soft nproc 385906
>
> Change 90-nproc.conf file
>
> /etc/security/limits.d/ 90-nproc.conf
>
> * soft nproc 385906
>
> check with ulimit -a
>

> May be help for someone else whos is facing same problem

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads