Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Apache: Users

Apache HTTPD Sporadic High CPU

 

 

Apache users RSS feed   Index | Next | Previous | View Threaded


bbawcombe at promnetwork

Apr 10, 2012, 2:51 PM

Post #1 of 5 (848 views)
Permalink
Apache HTTPD Sporadic High CPU

We are sporadically experiencing high CPU with our Apache instances.
The CPU averages about 5% utilization normally and has run fine for over
a year without an issue but over the past month we see spikes up to 100%
utilization that never drop until the instance is bounced. We can go up
to a week before we see the issue but we've also seen it happen multiple
times in a day. The logs are not indicating any errors for further
investigation and we only have this issue in our Production environment.


Things we've tried:
-We've searched numerous topics on Apache HTTPD and high CPU and with
regards to tuning parameters but considering we haven't had an increase
in usage and our average CPU utilization is normally low we are
reluctant to make tuning changes without understanding what else we may
impact.

-We ran prstat and the one anomaly we've seen is the Involuntary Context
Switching (ICX) seems to be high when the CPU spikes. Normally ICS is
in the single digits but when the CPU spikes we see thousands. From
observation it appears that the ICX value and CPU utilization increase
and drop simultaneously.

Information regarding our environment:
- Apache HTTPD 2.2.20
- MOD_PROXY as a reverse proxy
- Solaris 10
- 2 servers for high availability and have experienced the issue on both
servers

One other note is that we are using CA's SiteMinder (authentication and
authorization) agent for Apache. We've exhausted all of the support CA
will give us regarding this issue and they have no other known issues
from their customer base.

Thanks,
Bill




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd


trawick at gmail

Apr 10, 2012, 4:17 PM

Post #2 of 5 (810 views)
Permalink
Re: Apache HTTPD Sporadic High CPU [In reply to]

On Tue, Apr 10, 2012 at 5:51 PM, Bill Bawcombe
<bbawcombe [at] promnetwork> wrote:
> We are sporadically experiencing high CPU with our Apache instances.
> The CPU averages about 5% utilization normally and has run fine for over
> a year without an issue but over the past month we see spikes up to 100%
> utilization that never drop until the instance is bounced.  We can go up
> to a week before we see the issue but we've also seen it happen multiple
> times in a day.  The logs are not indicating any errors for further
> investigation and we only have this issue in our Production environment.
>
>
> Things we've tried:
> -We've searched numerous topics on Apache HTTPD and high CPU and with
> regards to tuning parameters but considering we haven't had an increase
> in usage and our average CPU utilization is normally low we are
> reluctant to make tuning changes without understanding what else we may
> impact.
>
> -We ran prstat and the one anomaly we've seen is the Involuntary Context
> Switching (ICX) seems to be high when the CPU spikes.  Normally ICS is
> in the single digits but when the CPU spikes we see thousands.  From
> observation it appears that the ICX value and CPU utilization increase
> and drop simultaneously.
>
> Information regarding our environment:
> - Apache HTTPD 2.2.20
> - MOD_PROXY as a reverse proxy
> - Solaris 10
> - 2 servers for high availability and have experienced the issue on both
> servers
>
> One other note is that we are using CA's SiteMinder (authentication and
> authorization) agent for Apache.  We've exhausted all of the support CA
> will give us regarding this issue and they have no other known issues
> from their customer base.

What kind of information did CA gather? SiteMinder has its own
threads in the httpd processes, and of course SiteMinder can run on
the normal httpd threads.

One obvious thing to try is to get a handful of pstacks of a high CPU
process at 10 second intervals when the problem occurs and see if you
can find anything out of the ordinary there. You can also point truss
to particular LWPs (as shown by pstack) for threads that look
suspicious.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd


bbawcombe at promnetwork

Apr 10, 2012, 6:07 PM

Post #3 of 5 (812 views)
Permalink
RE: Apache HTTPD Sporadic High CPU [In reply to]

CA wasn't much help at all but that is pretty typical.

We are requesting sudo access to pstack and truss and a few other utilities as my team is application specific and we don't have root access. I was hoping someone may have run into an issue like this before but the one thread I've found on here and other places regarding high CPU and ICX has gone unanswered since 2005.

One solution we've researched is to reduce the number of threads spawned by apache. Our thinking is that the number of threads generated is too large to be handled by the server. This is governed by the "ThreadsPerChild" parameter. The current value is 25 and we were thinking of reducing the value to 20.

I should add that we are running in Worker MPM.

Bill

-----Original Message-----
From: Jeff Trawick [mailto:trawick [at] gmail]
Sent: Tuesday, April 10, 2012 7:17 PM
To: users [at] httpd
Subject: Re: [users [at] http] Apache HTTPD Sporadic High CPU

On Tue, Apr 10, 2012 at 5:51 PM, Bill Bawcombe <bbawcombe [at] promnetwork> wrote:
> We are sporadically experiencing high CPU with our Apache instances.
> The CPU averages about 5% utilization normally and has run fine for
> over a year without an issue but over the past month we see spikes up
> to 100% utilization that never drop until the instance is bounced.  We
> can go up to a week before we see the issue but we've also seen it
> happen multiple times in a day.  The logs are not indicating any
> errors for further investigation and we only have this issue in our Production environment.
>
>
> Things we've tried:
> -We've searched numerous topics on Apache HTTPD and high CPU and with
> regards to tuning parameters but considering we haven't had an
> increase in usage and our average CPU utilization is normally low we
> are reluctant to make tuning changes without understanding what else
> we may impact.
>
> -We ran prstat and the one anomaly we've seen is the Involuntary
> Context Switching (ICX) seems to be high when the CPU spikes.  
> Normally ICS is in the single digits but when the CPU spikes we see
> thousands.  From observation it appears that the ICX value and CPU
> utilization increase and drop simultaneously.
>
> Information regarding our environment:
> - Apache HTTPD 2.2.20
> - MOD_PROXY as a reverse proxy
> - Solaris 10
> - 2 servers for high availability and have experienced the issue on
> both servers
>
> One other note is that we are using CA's SiteMinder (authentication
> and
> authorization) agent for Apache.  We've exhausted all of the support
> CA will give us regarding this issue and they have no other known
> issues from their customer base.

What kind of information did CA gather? SiteMinder has its own threads in the httpd processes, and of course SiteMinder can run on the normal httpd threads.

One obvious thing to try is to get a handful of pstacks of a high CPU process at 10 second intervals when the problem occurs and see if you can find anything out of the ordinary there. You can also point truss to particular LWPs (as shown by pstack) for threads that look suspicious.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd


trawick at gmail

Apr 10, 2012, 6:24 PM

Post #4 of 5 (810 views)
Permalink
Re: Apache HTTPD Sporadic High CPU [In reply to]

On Tue, Apr 10, 2012 at 9:07 PM, Bill Bawcombe
<bbawcombe [at] promnetwork> wrote:
> CA wasn't much help at all but that is pretty typical.
>
> We are requesting sudo access to pstack and truss and a few other utilities as my team is application specific and we don't have root access.  I was hoping someone may have run into an issue like this before but the one thread I've found on here and other places regarding high CPU and ICX has gone unanswered since 2005.
>
> One solution we've researched is to reduce the number of threads spawned by apache. Our thinking is that the number of threads generated is too large to be handled by the server.  This is governed by the "ThreadsPerChild" parameter.  The current value is 25 and we were thinking of reducing the value to 20.
>
> I should add that we are running in Worker MPM.

I doubt that the number of worker threads is the issue, and reducing
from 25 to 20 isn't much difference anyway. (Another way to look at
it is that you'll end up with more SiteMinder threads as you increase
the number of child processes ;) )

I hope your APR is >= 1.3.10. (I think one of the symptoms from old
bugs in Solaris event port support was looping.)

>
> Bill
>
> -----Original Message-----
> From: Jeff Trawick [mailto:trawick [at] gmail]
> Sent: Tuesday, April 10, 2012 7:17 PM
> To: users [at] httpd
> Subject: Re: [users [at] http] Apache HTTPD Sporadic High CPU
>
> On Tue, Apr 10, 2012 at 5:51 PM, Bill Bawcombe <bbawcombe [at] promnetwork> wrote:
>> We are sporadically experiencing high CPU with our Apache instances.
>> The CPU averages about 5% utilization normally and has run fine for
>> over a year without an issue but over the past month we see spikes up
>> to 100% utilization that never drop until the instance is bounced.  We
>> can go up to a week before we see the issue but we've also seen it
>> happen multiple times in a day.  The logs are not indicating any
>> errors for further investigation and we only have this issue in our Production environment.
>>
>>
>> Things we've tried:
>> -We've searched numerous topics on Apache HTTPD and high CPU and with
>> regards to tuning parameters but considering we haven't had an
>> increase in usage and our average CPU utilization is normally low we
>> are reluctant to make tuning changes without understanding what else
>> we may impact.
>>
>> -We ran prstat and the one anomaly we've seen is the Involuntary
>> Context Switching (ICX) seems to be high when the CPU spikes.
>> Normally ICS is in the single digits but when the CPU spikes we see
>> thousands.  From observation it appears that the ICX value and CPU
>> utilization increase and drop simultaneously.
>>
>> Information regarding our environment:
>> - Apache HTTPD 2.2.20
>> - MOD_PROXY as a reverse proxy
>> - Solaris 10
>> - 2 servers for high availability and have experienced the issue on
>> both servers
>>
>> One other note is that we are using CA's SiteMinder (authentication
>> and
>> authorization) agent for Apache.  We've exhausted all of the support
>> CA will give us regarding this issue and they have no other known
>> issues from their customer base.
>
> What kind of information did CA gather?  SiteMinder has its own threads in the httpd processes, and of course SiteMinder can run on the normal httpd threads.
>
> One obvious thing to try is to get a handful of pstacks of a high CPU process at 10 second intervals when the problem occurs and see if you can find anything out of the ordinary there.  You can also point truss to particular LWPs (as shown by pstack) for threads that look suspicious.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>



--
Born in Roswell... married an alien...

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd


bbawcombe at promnetwork

Apr 11, 2012, 10:57 AM

Post #5 of 5 (805 views)
Permalink
RE: Apache HTTPD Sporadic High CPU [In reply to]

We checked our APR version (1.4.5) and we should be OK for the bug you mention.

Here is the output from our httpd -V if this helps:

Server version: Apache/2.2.20 (Unix)
Server built: Sep 30 2011 13:13:01
Server's Module Magic Number: 20051115:28
Server loaded: APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture: 32-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_PROC_PTHREAD_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/opt/apache/cdars_2.2.20"
-D SUEXEC_BIN="/opt/apache/cdars_2.2.20/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"



-----Original Message-----
From: Jeff Trawick [mailto:trawick [at] gmail]
Sent: Tuesday, April 10, 2012 9:24 PM
To: users [at] httpd
Subject: Re: [users [at] http] Apache HTTPD Sporadic High CPU

On Tue, Apr 10, 2012 at 9:07 PM, Bill Bawcombe <bbawcombe [at] promnetwork> wrote:
> CA wasn't much help at all but that is pretty typical.
>
> We are requesting sudo access to pstack and truss and a few other utilities as my team is application specific and we don't have root access.  I was hoping someone may have run into an issue like this before but the one thread I've found on here and other places regarding high CPU and ICX has gone unanswered since 2005.
>
> One solution we've researched is to reduce the number of threads spawned by apache. Our thinking is that the number of threads generated is too large to be handled by the server.  This is governed by the "ThreadsPerChild" parameter.  The current value is 25 and we were thinking of reducing the value to 20.
>
> I should add that we are running in Worker MPM.

I doubt that the number of worker threads is the issue, and reducing from 25 to 20 isn't much difference anyway. (Another way to look at it is that you'll end up with more SiteMinder threads as you increase the number of child processes ;) )

I hope your APR is >= 1.3.10. (I think one of the symptoms from old bugs in Solaris event port support was looping.)

>
> Bill
>
> -----Original Message-----
> From: Jeff Trawick [mailto:trawick [at] gmail]
> Sent: Tuesday, April 10, 2012 7:17 PM
> To: users [at] httpd
> Subject: Re: [users [at] http] Apache HTTPD Sporadic High CPU
>
> On Tue, Apr 10, 2012 at 5:51 PM, Bill Bawcombe <bbawcombe [at] promnetwork> wrote:
>> We are sporadically experiencing high CPU with our Apache instances.
>> The CPU averages about 5% utilization normally and has run fine for
>> over a year without an issue but over the past month we see spikes up
>> to 100% utilization that never drop until the instance is bounced.  
>> We can go up to a week before we see the issue but we've also seen it
>> happen multiple times in a day.  The logs are not indicating any
>> errors for further investigation and we only have this issue in our Production environment.
>>
>>
>> Things we've tried:
>> -We've searched numerous topics on Apache HTTPD and high CPU and with
>> regards to tuning parameters but considering we haven't had an
>> increase in usage and our average CPU utilization is normally low we
>> are reluctant to make tuning changes without understanding what else
>> we may impact.
>>
>> -We ran prstat and the one anomaly we've seen is the Involuntary
>> Context Switching (ICX) seems to be high when the CPU spikes.
>> Normally ICS is in the single digits but when the CPU spikes we see
>> thousands.  From observation it appears that the ICX value and CPU
>> utilization increase and drop simultaneously.
>>
>> Information regarding our environment:
>> - Apache HTTPD 2.2.20
>> - MOD_PROXY as a reverse proxy
>> - Solaris 10
>> - 2 servers for high availability and have experienced the issue on
>> both servers
>>
>> One other note is that we are using CA's SiteMinder (authentication
>> and
>> authorization) agent for Apache.  We've exhausted all of the support
>> CA will give us regarding this issue and they have no other known
>> issues from their customer base.
>
> What kind of information did CA gather?  SiteMinder has its own threads in the httpd processes, and of course SiteMinder can run on the normal httpd threads.
>
> One obvious thing to try is to get a handful of pstacks of a high CPU process at 10 second intervals when the problem occurs and see if you can find anything out of the ordinary there.  You can also point truss to particular LWPs (as shown by pstack) for threads that look suspicious.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>



--
Born in Roswell... married an alien...

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

Apache users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.