Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ModPerl: ModPerl

High CPU utilization on RHEL5.6/CentOS5.6

 

 

ModPerl modperl RSS feed   Index | Next | Previous | View Threaded


christopher.stanton at codaxus

Jul 25, 2011, 3:43 PM

Post #1 of 11 (920 views)
Permalink
High CPU utilization on RHEL5.6/CentOS5.6

I have a MJPEG streaming system which uses mod_perl in the web
interface to supply the final stream to the client.

I am seeing high cpu utilization under RHEL5.6 which I don't see on
FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
different hardware, but not that different. And on EL5 the httpd
session is way higher (cpu wise) than the supplying proxy. On FC14 on
my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.

I think this is an issue with that platform, but I haven't seen
any/many reports. Does anyone have a sec to look at the source? I
don't think I have implemented this is a horrible inefficient way but
who knows. It is true, I could have byte markers and then know the
size of the JPEG I am expecting rather than just splitting on the
boundary field, but in tests (and production) it seemed to work ok.

http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl

Any comments or critiques would be appreciated,
Christopher

----------------------------------------------------------------------

On FC14:
Requires: httpd >= 2.2.17-1
Requires: mod_perl >= 2.0.4-11
Requires: mod_ssl >= 1:2.2.17-1
Requires: perl >= 5.12.3-141
Requires: perl-CGI >= 3.51-1
Requires: perl-Digest-SHA1 >= 2.12-4
Requires: perl-XML-Filter-BufferText >= 1.01-9
Requires: perl-XML-Simple >= 2.18-7
Requires: perl-XML-Validator-Schema >= 1.10-5
Requires: perl-Tree-DAG_Node >= 1.06-8
Requires: perl-LDAP >= 0.40-2
Requires: perl-Crypt-SSLeay >= 0.58-1
Requires: ffmpeg >= 0.6-4

On EL5:
Requires: httpd >= 2.2.3-45
Requires: mod_perl >= 2.0.4-6
Requires: mod_ssl >= 2.2.3-45
Requires: perl >= 5.8.8-32
Requires: perl-Digest-SHA1 >= 2.11-1
Requires: perl-XML-Filter-BufferText >= 1.01
Requires: perl-XML-Simple >= 2.14-4
Requires: perl-XML-Validator-Schema >= 1.10-1
Requires: perl-Tree-DAG_Node >= 1.06
Requires: perl-LDAP >= 0.33-3
Requires: perl-Crypt-SSLeay >= 0.51-11
Requires: ffmpeg >= 0.6.1-1


fred at redhotpenguin

Jul 25, 2011, 3:55 PM

Post #2 of 11 (908 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

This may not be related, but when I was working with Centos 5 in a
Vmware environment, I ran into an issue of high cpu since the default
clock rate with Linux 2.6 is 1000Hz. I changed to a 100Hz clock rate
and got much lower cpu usage.

I'm not sure if that is your issue, perhaps FC14 was using less than a
1000Hz clock rate. You might try a tickless kernel, or dig further
into what clock rate your kernel is using.

I don't know where the authoritative source for this information is,
but here's what I found with a search:

http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare

On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
<christopher.stanton [at] codaxus> wrote:
> I have a MJPEG streaming system which uses mod_perl in the web
> interface to supply the final stream to the client.
>
> I am seeing high cpu utilization under RHEL5.6 which I don't see on
> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
> different hardware, but not that different. And on EL5 the httpd
> session is way higher (cpu wise) than the supplying proxy. On FC14 on
> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>
> I think this is an issue with that platform, but I haven't seen
> any/many reports. Does anyone have a sec to look at the source? I
> don't think I have implemented this is a horrible inefficient way but
> who knows. It is true, I could have byte markers and then know the
> size of the JPEG I am expecting rather than just splitting on the
> boundary field, but in tests (and production) it seemed to work ok.
>
> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl
>
> Any comments or critiques would be appreciated,
> Christopher
>
> ----------------------------------------------------------------------
>
> On FC14:
> Requires: httpd >= 2.2.17-1
> Requires: mod_perl >= 2.0.4-11
> Requires: mod_ssl >= 1:2.2.17-1
> Requires: perl >= 5.12.3-141
> Requires: perl-CGI >= 3.51-1
> Requires: perl-Digest-SHA1 >= 2.12-4
> Requires: perl-XML-Filter-BufferText >= 1.01-9
> Requires: perl-XML-Simple >= 2.18-7
> Requires: perl-XML-Validator-Schema >= 1.10-5
> Requires: perl-Tree-DAG_Node >= 1.06-8
> Requires: perl-LDAP >= 0.40-2
> Requires: perl-Crypt-SSLeay >= 0.58-1
> Requires: ffmpeg >= 0.6-4
>
> On EL5:
> Requires: httpd >= 2.2.3-45
> Requires: mod_perl >= 2.0.4-6
> Requires: mod_ssl >= 2.2.3-45
> Requires: perl >= 5.8.8-32
> Requires: perl-Digest-SHA1 >= 2.11-1
> Requires: perl-XML-Filter-BufferText >= 1.01
> Requires: perl-XML-Simple >= 2.14-4
> Requires: perl-XML-Validator-Schema >= 1.10-1
> Requires: perl-Tree-DAG_Node >= 1.06
> Requires: perl-LDAP >= 0.33-3
> Requires: perl-Crypt-SSLeay >= 0.51-11
> Requires: ffmpeg >= 0.6.1-1
>


christopher.stanton at codaxus

Jul 25, 2011, 4:02 PM

Post #3 of 11 (911 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

Neither install is running within VMWare.

On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer <fred [at] redhotpenguin> wrote:
> This may not be related, but when I was working with Centos 5 in a
> Vmware environment, I ran into an issue of high cpu since the default
> clock rate with Linux 2.6 is 1000Hz.  I changed to a 100Hz clock rate
> and got much lower cpu usage.
>
> I'm not sure if that is your issue, perhaps FC14 was using less than a
> 1000Hz clock rate.  You might try a tickless kernel, or dig further
> into what clock rate your kernel is using.
>
> I don't know where the authoritative source for this information is,
> but here's what I found with a search:
>
> http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare
>
> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
> <christopher.stanton [at] codaxus> wrote:
>> I have a MJPEG streaming system which uses mod_perl in the web
>> interface to supply the final stream to the client.
>>
>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>> different hardware, but not that different. And on EL5 the httpd
>> session is way higher (cpu wise) than the supplying proxy. On FC14 on
>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>>
>> I think this is an issue with that platform, but I haven't seen
>> any/many reports. Does anyone have a sec to look at the source? I
>> don't think I have implemented this is a horrible inefficient way but
>> who knows. It is true, I could have byte markers and then know the
>> size of the JPEG I am expecting rather than just splitting on the
>> boundary field, but in tests (and production) it seemed to work ok.
>>
>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl
>>
>> Any comments or critiques would be appreciated,
>> Christopher
>>
>> ----------------------------------------------------------------------
>>
>> On FC14:
>> Requires: httpd >= 2.2.17-1
>> Requires: mod_perl >= 2.0.4-11
>> Requires: mod_ssl >= 1:2.2.17-1
>> Requires: perl >= 5.12.3-141
>> Requires: perl-CGI >= 3.51-1
>> Requires: perl-Digest-SHA1 >= 2.12-4
>> Requires: perl-XML-Filter-BufferText >= 1.01-9
>> Requires: perl-XML-Simple >= 2.18-7
>> Requires: perl-XML-Validator-Schema >= 1.10-5
>> Requires: perl-Tree-DAG_Node >= 1.06-8
>> Requires: perl-LDAP >= 0.40-2
>> Requires: perl-Crypt-SSLeay >= 0.58-1
>> Requires: ffmpeg >= 0.6-4
>>
>> On EL5:
>> Requires: httpd >= 2.2.3-45
>> Requires: mod_perl >= 2.0.4-6
>> Requires: mod_ssl >= 2.2.3-45
>> Requires: perl >= 5.8.8-32
>> Requires: perl-Digest-SHA1 >= 2.11-1
>> Requires: perl-XML-Filter-BufferText >= 1.01
>> Requires: perl-XML-Simple >= 2.14-4
>> Requires: perl-XML-Validator-Schema >= 1.10-1
>> Requires: perl-Tree-DAG_Node >= 1.06
>> Requires: perl-LDAP >= 0.33-3
>> Requires: perl-Crypt-SSLeay >= 0.51-11
>> Requires: ffmpeg >= 0.6.1-1
>>
>


fred at redhotpenguin

Jul 25, 2011, 4:43 PM

Post #4 of 11 (907 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

On Mon, Jul 25, 2011 at 4:02 PM, Christopher Stanton
<christopher.stanton [at] codaxus> wrote:
> Neither install is running within VMWare.

Might be worthwhile trying a different rate though, or going tickless.
Perl 5.8.8 isn't that much (if at all) slower than 5.12. Looks like
the FC14 kernel is tickless:

http://docs.fedoraproject.org/en-US/Fedora/14/pdf/Power_Management_Guide/Fedora-14-Power_Management_Guide-en-US.pdf


> On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer <fred [at] redhotpenguin> wrote:
>> This may not be related, but when I was working with Centos 5 in a
>> Vmware environment, I ran into an issue of high cpu since the default
>> clock rate with Linux 2.6 is 1000Hz.  I changed to a 100Hz clock rate
>> and got much lower cpu usage.
>>
>> I'm not sure if that is your issue, perhaps FC14 was using less than a
>> 1000Hz clock rate.  You might try a tickless kernel, or dig further
>> into what clock rate your kernel is using.
>>
>> I don't know where the authoritative source for this information is,
>> but here's what I found with a search:
>>
>> http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare
>>
>> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
>> <christopher.stanton [at] codaxus> wrote:
>>> I have a MJPEG streaming system which uses mod_perl in the web
>>> interface to supply the final stream to the client.
>>>
>>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>>> different hardware, but not that different. And on EL5 the httpd
>>> session is way higher (cpu wise) than the supplying proxy. On FC14 on
>>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
>>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>>>
>>> I think this is an issue with that platform, but I haven't seen
>>> any/many reports. Does anyone have a sec to look at the source? I
>>> don't think I have implemented this is a horrible inefficient way but
>>> who knows. It is true, I could have byte markers and then know the
>>> size of the JPEG I am expecting rather than just splitting on the
>>> boundary field, but in tests (and production) it seemed to work ok.
>>>
>>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl
>>>
>>> Any comments or critiques would be appreciated,
>>> Christopher
>>>
>>> ----------------------------------------------------------------------
>>>
>>> On FC14:
>>> Requires: httpd >= 2.2.17-1
>>> Requires: mod_perl >= 2.0.4-11
>>> Requires: mod_ssl >= 1:2.2.17-1
>>> Requires: perl >= 5.12.3-141
>>> Requires: perl-CGI >= 3.51-1
>>> Requires: perl-Digest-SHA1 >= 2.12-4
>>> Requires: perl-XML-Filter-BufferText >= 1.01-9
>>> Requires: perl-XML-Simple >= 2.18-7
>>> Requires: perl-XML-Validator-Schema >= 1.10-5
>>> Requires: perl-Tree-DAG_Node >= 1.06-8
>>> Requires: perl-LDAP >= 0.40-2
>>> Requires: perl-Crypt-SSLeay >= 0.58-1
>>> Requires: ffmpeg >= 0.6-4
>>>
>>> On EL5:
>>> Requires: httpd >= 2.2.3-45
>>> Requires: mod_perl >= 2.0.4-6
>>> Requires: mod_ssl >= 2.2.3-45
>>> Requires: perl >= 5.8.8-32
>>> Requires: perl-Digest-SHA1 >= 2.11-1
>>> Requires: perl-XML-Filter-BufferText >= 1.01
>>> Requires: perl-XML-Simple >= 2.14-4
>>> Requires: perl-XML-Validator-Schema >= 1.10-1
>>> Requires: perl-Tree-DAG_Node >= 1.06
>>> Requires: perl-LDAP >= 0.33-3
>>> Requires: perl-Crypt-SSLeay >= 0.51-11
>>> Requires: ffmpeg >= 0.6.1-1
>>>
>>
>


christopher.stanton at codaxus

Jul 25, 2011, 5:24 PM

Post #5 of 11 (909 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

Before I start playing mix-and-match with kernels, I would prefer some
feedback on mod_perl and the script's code.

It doesn't sleep in the main loop, but it should block on
my @ready = $select->can_read(1);
and not block on
if ( $sock->read(my $data, POSIX::BUFSIZ*10) ){

since the socket is set with Blocking => 0.

A JPEG (in a 25fps stream) arrives every 40,000 microseconds.

Christopher


On Mon, Jul 25, 2011 at 6:43 PM, Fred Moyer <fred [at] redhotpenguin> wrote:
> On Mon, Jul 25, 2011 at 4:02 PM, Christopher Stanton
> <christopher.stanton [at] codaxus> wrote:
>> Neither install is running within VMWare.
>
> Might be worthwhile trying a different rate though, or going tickless.
>  Perl 5.8.8 isn't that much (if at all) slower than 5.12.  Looks like
> the FC14 kernel is tickless:
>
> http://docs.fedoraproject.org/en-US/Fedora/14/pdf/Power_Management_Guide/Fedora-14-Power_Management_Guide-en-US.pdf
>
>
>> On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer <fred [at] redhotpenguin> wrote:
>>> This may not be related, but when I was working with Centos 5 in a
>>> Vmware environment, I ran into an issue of high cpu since the default
>>> clock rate with Linux 2.6 is 1000Hz.  I changed to a 100Hz clock rate
>>> and got much lower cpu usage.
>>>
>>> I'm not sure if that is your issue, perhaps FC14 was using less than a
>>> 1000Hz clock rate.  You might try a tickless kernel, or dig further
>>> into what clock rate your kernel is using.
>>>
>>> I don't know where the authoritative source for this information is,
>>> but here's what I found with a search:
>>>
>>> http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare
>>>
>>> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
>>> <christopher.stanton [at] codaxus> wrote:
>>>> I have a MJPEG streaming system which uses mod_perl in the web
>>>> interface to supply the final stream to the client.
>>>>
>>>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>>>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>>>> different hardware, but not that different. And on EL5 the httpd
>>>> session is way higher (cpu wise) than the supplying proxy. On FC14 on
>>>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
>>>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>>>>
>>>> I think this is an issue with that platform, but I haven't seen
>>>> any/many reports. Does anyone have a sec to look at the source? I
>>>> don't think I have implemented this is a horrible inefficient way but
>>>> who knows. It is true, I could have byte markers and then know the
>>>> size of the JPEG I am expecting rather than just splitting on the
>>>> boundary field, but in tests (and production) it seemed to work ok.
>>>>
>>>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl
>>>>
>>>> Any comments or critiques would be appreciated,
>>>> Christopher
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> On FC14:
>>>> Requires: httpd >= 2.2.17-1
>>>> Requires: mod_perl >= 2.0.4-11
>>>> Requires: mod_ssl >= 1:2.2.17-1
>>>> Requires: perl >= 5.12.3-141
>>>> Requires: perl-CGI >= 3.51-1
>>>> Requires: perl-Digest-SHA1 >= 2.12-4
>>>> Requires: perl-XML-Filter-BufferText >= 1.01-9
>>>> Requires: perl-XML-Simple >= 2.18-7
>>>> Requires: perl-XML-Validator-Schema >= 1.10-5
>>>> Requires: perl-Tree-DAG_Node >= 1.06-8
>>>> Requires: perl-LDAP >= 0.40-2
>>>> Requires: perl-Crypt-SSLeay >= 0.58-1
>>>> Requires: ffmpeg >= 0.6-4
>>>>
>>>> On EL5:
>>>> Requires: httpd >= 2.2.3-45
>>>> Requires: mod_perl >= 2.0.4-6
>>>> Requires: mod_ssl >= 2.2.3-45
>>>> Requires: perl >= 5.8.8-32
>>>> Requires: perl-Digest-SHA1 >= 2.11-1
>>>> Requires: perl-XML-Filter-BufferText >= 1.01
>>>> Requires: perl-XML-Simple >= 2.14-4
>>>> Requires: perl-XML-Validator-Schema >= 1.10-1
>>>> Requires: perl-Tree-DAG_Node >= 1.06
>>>> Requires: perl-LDAP >= 0.33-3
>>>> Requires: perl-Crypt-SSLeay >= 0.51-11
>>>> Requires: ffmpeg >= 0.6.1-1
>>>>
>>>
>>
>


perrin at elem

Jul 26, 2011, 10:43 AM

Post #6 of 11 (890 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

On Mon, Jul 25, 2011 at 6:43 PM, Christopher Stanton
<christopher.stanton [at] codaxus> wrote:
> I am seeing high cpu utilization under RHEL5.6 which I don't see on
> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
> different hardware, but not that different.

Well, you've changed hardware and lots of software versions, so there
are many things it could be. The only one that seems unlikely is your
code, since that didn't change.

I would suspect the major downgrade in Perl as the culprit. You went
from 5.12 to 5.8. To test this, I suggest you install 5.12 on your
new system and compile mod_perl against it. If you don't want to
touch the system mod_perl and apache you can install your own in a
different location. I use perlbrew to try out new Perl versions
locally.

- Perrin


christopher.stanton at codaxus

Jul 26, 2011, 10:58 AM

Post #7 of 11 (891 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

I will see if I can get Perl 5.12 and an associated mod_perl up and
running on the problem machine and see if the CPU utilization goes
down.

Long long ago in a install far far away I did not run into this issue
on FC4 which has very similar packages other than apache (2.0 rather
than 2.2):
Requires: httpd >= 2.0.54-0; perl >= 5.8.6-0; perl-Digest-SHA1 >=
2.10-0; mod_perl >= 2.0.1-0;

On Tue, Jul 26, 2011 at 12:43 PM, Perrin Harkins <perrin [at] elem> wrote:
> On Mon, Jul 25, 2011 at 6:43 PM, Christopher Stanton
> <christopher.stanton [at] codaxus> wrote:
>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>> different hardware, but not that different.
>
> Well, you've changed hardware and lots of software versions, so there
> are many things it could be.  The only one that seems unlikely is your
> code, since that didn't change.
>
> I would suspect the major downgrade in Perl as the culprit.  You went
> from 5.12 to 5.8.  To test this, I suggest you install 5.12 on your
> new system and compile mod_perl against it.  If you don't want to
> touch the system mod_perl and apache you can install your own in a
> different location.  I use perlbrew to try out new Perl versions
> locally.
>
> - Perrin
>


davehodg at gmail

Jul 27, 2011, 5:22 AM

Post #8 of 11 (887 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

On 26 Jul 2011, at 18:58, Christopher Stanton wrote:

> I will see if I can get Perl 5.12 and an associated mod_perl up and
> running on the problem machine and see if the CPU utilization goes
> down.

Is benchmarking out of the question?


christopher.stanton at codaxus

Jul 27, 2011, 5:36 AM

Post #9 of 11 (888 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

Which benchmark module would work best with mod_perl, Perl 5.8 and Perl 5.12?

On Wed, Jul 27, 2011 at 7:22 AM, Dave Hodgkinson <davehodg [at] gmail> wrote:
>
> On 26 Jul 2011, at 18:58, Christopher Stanton wrote:
>
>> I will see if I can get Perl 5.12 and an associated mod_perl up and
>> running on the problem machine and see if the CPU utilization goes
>> down.
>
> Is benchmarking out of the question?
>
>


perrin at elem

Aug 1, 2011, 7:48 AM

Post #10 of 11 (851 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

On Wed, Jul 27, 2011 at 8:36 AM, Christopher Stanton
<christopher.stanton [at] codaxus> wrote:
> Which benchmark module would work best with mod_perl, Perl 5.8 and Perl 5.12?

For web apps, I usually write my own test with LWP (or HTTP::Async if
you want to simulate a lot of users). To see where things are slow,
use Devel::NYTProf.

- Perrin


davehodg at gmail

Aug 1, 2011, 10:13 AM

Post #11 of 11 (852 views)
Permalink
Re: High CPU utilization on RHEL5.6/CentOS5.6 [In reply to]

Wait. Is this the one with the horrible bug that knackers OO?

On 25 Jul 2011, at 23:43, Christopher Stanton wrote:

> Requires: perl >= 5.8.8-32

ModPerl modperl RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.