Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Users

Mythbackend high cpu?

 

 

MythTV users RSS feed   Index | Next | Previous | View Threaded


tom at redpepperracing

Jun 17, 2012, 7:25 AM

Post #1 of 13 (1957 views)
Permalink
Mythbackend high cpu?

Is anyone else seeing extremely high CPU usage on their backend when
playing back *any* content on a remote front end? I just noticed that
during playback, the CPU usage of my backend stays around 175-200% on
a 2 x Dual core AMD system. I can't imagine what it would be doing
other than just streaming the file, is anyone else seeing this, or can
explain it?

I am on close to current master:

MythTV Version : v0.26-pre-629-g181641a-dirty
MythTV Branch : master
Network Protocol : 75
Library API : 0.26.20120614-1
QT Version : 4.6.3
Options compiled in:
linux profile use_hidesyms using_alsa using_oss using_backend
using_bindings_perl using_bindings_python using_bindings_php using_dvb
using_frontend using_hdhomerun using_ceton using_hdpvr using_iptv
using_ivtv using_joystick_menu using_libcrypto using_libdns_sd
using_libxml2 using_lirc using_mheg using_opengl_video using_qtwebkit
using_qtscript using_qtdbus using_v4l2 using_v4l1 using_x11
using_xrandr using_bindings_perl using_bindings_python
using_bindings_php using_mythtranscode using_opengl
using_ffmpeg_threads using_live using_mheg using_libxml2

Thanks
Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


tom at redpepperracing

Jun 17, 2012, 11:07 AM

Post #2 of 13 (1903 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Sun, Jun 17, 2012 at 10:25 AM, Tom Lichti <tom [at] redpepperracing> wrote:
> Is anyone else seeing extremely high CPU usage on their backend when
> playing back *any* content on a remote front end? I just noticed that
> during playback, the CPU usage of my backend stays around 175-200% on
> a 2 x Dual core AMD system. I can't imagine what it would be doing
> other than just streaming the file, is anyone else seeing this, or can
> explain it?
>
> I am on close to current master:
>
> MythTV Version : v0.26-pre-629-g181641a-dirty
> MythTV Branch : master
> Network Protocol : 75
> Library API : 0.26.20120614-1
> QT Version : 4.6.3
> Options compiled in:
>  linux profile use_hidesyms using_alsa using_oss using_backend
> using_bindings_perl using_bindings_python using_bindings_php using_dvb
> using_frontend using_hdhomerun using_ceton using_hdpvr using_iptv
> using_ivtv using_joystick_menu using_libcrypto using_libdns_sd
> using_libxml2 using_lirc using_mheg using_opengl_video using_qtwebkit
> using_qtscript using_qtdbus using_v4l2 using_v4l1 using_x11
> using_xrandr using_bindings_perl using_bindings_python
> using_bindings_php using_mythtranscode using_opengl
> using_ffmpeg_threads using_live using_mheg using_libxml2

Hmmm....looks like it may have been a memory leak of some sort, I
started getting OOM killers and lots of swapping. I'll keep an eye on
it and see if it happens again.

Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


tom at redpepperracing

Jun 17, 2012, 1:07 PM

Post #3 of 13 (1899 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Sun, Jun 17, 2012 at 2:07 PM, Tom Lichti <tom [at] redpepperracing> wrote:
> On Sun, Jun 17, 2012 at 10:25 AM, Tom Lichti <tom [at] redpepperracing> wrote:
>> Is anyone else seeing extremely high CPU usage on their backend when
>> playing back *any* content on a remote front end? I just noticed that
>> during playback, the CPU usage of my backend stays around 175-200% on
>> a 2 x Dual core AMD system. I can't imagine what it would be doing
>> other than just streaming the file, is anyone else seeing this, or can
>> explain it?
>>
>> I am on close to current master:
>>
>> MythTV Version : v0.26-pre-629-g181641a-dirty
>> MythTV Branch : master
>> Network Protocol : 75
>> Library API : 0.26.20120614-1
>> QT Version : 4.6.3
>> Options compiled in:
>>  linux profile use_hidesyms using_alsa using_oss using_backend
>> using_bindings_perl using_bindings_python using_bindings_php using_dvb
>> using_frontend using_hdhomerun using_ceton using_hdpvr using_iptv
>> using_ivtv using_joystick_menu using_libcrypto using_libdns_sd
>> using_libxml2 using_lirc using_mheg using_opengl_video using_qtwebkit
>> using_qtscript using_qtdbus using_v4l2 using_v4l1 using_x11
>> using_xrandr using_bindings_perl using_bindings_python
>> using_bindings_php using_mythtranscode using_opengl
>> using_ffmpeg_threads using_live using_mheg using_libxml2
>
> Hmmm....looks like it may have been a memory leak of some sort, I
> started getting OOM killers and lots of swapping. I'll keep an eye on
> it and see if it happens again.
>

I love talking to myself. Now with one frontend playing back and two
recordings (1 HDHR and 1 HD-PVR on a slave BE) the BE CPU is over
250%. Lots of free memory (overall system memory is 8GB), so the OOM
doesn't explain it.

Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


joe at thefrys

Jun 18, 2012, 12:48 AM

Post #4 of 13 (1903 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

>
>
> >> Is anyone else seeing extremely high CPU usage on their backend when
> >> playing back *any* content on a remote front end? I just noticed that
> >> during playback, the CPU usage of my backend stays around 175-200% on
> >> a 2 x Dual core AMD system. I can't imagine what it would be doing
> >> other than just streaming the file, is anyone else seeing this, or can
> >> explain it?
> >>
> >> I am on close to current master:
> >>
> >> MythTV Version : v0.26-pre-629-g181641a-dirty
> >> MythTV Branch : master
> >> Network Protocol : 75
> >> Library API : 0.26.20120614-1
> >> QT Version : 4.6.3
> >> Options compiled in:
> >> linux profile use_hidesyms using_alsa using_oss using_backend
> >> using_bindings_perl using_bindings_python using_bindings_php using_dvb
> >> using_frontend using_hdhomerun using_ceton using_hdpvr using_iptv
> >> using_ivtv using_joystick_menu using_libcrypto using_libdns_sd
> >> using_libxml2 using_lirc using_mheg using_opengl_video using_qtwebkit
> >> using_qtscript using_qtdbus using_v4l2 using_v4l1 using_x11
> >> using_xrandr using_bindings_perl using_bindings_python
> >> using_bindings_php using_mythtranscode using_opengl
> >> using_ffmpeg_threads using_live using_mheg using_libxml2
> >
> > Hmmm....looks like it may have been a memory leak of some sort, I
> > started getting OOM killers and lots of swapping. I'll keep an eye on
> > it and see if it happens again.
> >
>
> I love talking to myself. Now with one frontend playing back and two
> recordings (1 HDHR and 1 HD-PVR on a slave BE) the BE CPU is over
> 250%. Lots of free memory (overall system memory is 8GB), so the OOM
> doesn't explain it.
>
>
The only things I can think of that MAY cause excessive CPU usage on the
backend, explicitly during playback/recording are:

1. Jobs (transcode/commercial flagging)
2. Degraded software RAID array.
3. non-DMA disk IO.
4. Encrypted/compressed drive
5. non-DMA network controller, or some on chipset NIC and a really horrible
network causing a ton of errors?

First try doing a couple of IO intensive tasks to confirm if it's mythtv
(perhaps use VLC to stream from the HDHR to a file?). I suspect it's
something on the system itself, not mythtv.

If I am right I would start by verifying all RAID arrays are clean, and if
all looks good, do a full shutdown. Then disconnect power and let it sit
for 15 minutes so any charged caps can dissipate before plugging in and
turning on. This has fixed countless weird issues for me in the past,
primarily with NIC's that support WOL. Seems as though many components of
your system don't truly reset anymore until the power is cut completely.

Good luck


tom at redpepperracing

Jun 18, 2012, 6:21 AM

Post #5 of 13 (1906 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Mon, Jun 18, 2012 at 3:48 AM, Joseph Fry <joe [at] thefrys> wrote:
>>
>> >> Is anyone else seeing extremely high CPU usage on their backend when
>> >> playing back *any* content on a remote front end? I just noticed that
>> >> during playback, the CPU usage of my backend stays around 175-200% on
>> >> a 2 x Dual core AMD system. I can't imagine what it would be doing
>> >> other than just streaming the file, is anyone else seeing this, or can
>> >> explain it?
>> >>
>> >> I am on close to current master:
>> >>
>> >> MythTV Version : v0.26-pre-629-g181641a-dirty
>> >> MythTV Branch : master
>> >> Network Protocol : 75
>> >> Library API : 0.26.20120614-1
>> >> QT Version : 4.6.3
>> >> Options compiled in:
>> >>  linux profile use_hidesyms using_alsa using_oss using_backend
>> >> using_bindings_perl using_bindings_python using_bindings_php using_dvb
>> >> using_frontend using_hdhomerun using_ceton using_hdpvr using_iptv
>> >> using_ivtv using_joystick_menu using_libcrypto using_libdns_sd
>> >> using_libxml2 using_lirc using_mheg using_opengl_video using_qtwebkit
>> >> using_qtscript using_qtdbus using_v4l2 using_v4l1 using_x11
>> >> using_xrandr using_bindings_perl using_bindings_python
>> >> using_bindings_php using_mythtranscode using_opengl
>> >> using_ffmpeg_threads using_live using_mheg using_libxml2
>> >
>> > Hmmm....looks like it may have been a memory leak of some sort, I
>> > started getting OOM killers and lots of swapping. I'll keep an eye on
>> > it and see if it happens again.
>> >
>>
>> I love talking to myself. Now with one frontend playing back and two
>> recordings (1 HDHR and 1 HD-PVR on a slave BE) the BE CPU is over
>> 250%. Lots of free memory (overall system memory is 8GB), so the OOM
>> doesn't explain it.
>>
>
> The only things I can think of that MAY cause excessive CPU usage on the
> backend, explicitly during playback/recording are:
>
> 1. Jobs (transcode/commercial flagging)
> 2. Degraded software RAID array.
> 3. non-DMA disk IO.
> 4. Encrypted/compressed drive
> 5. non-DMA network controller, or some on chipset NIC and a really horrible
> network causing a ton of errors?
>
> First try doing a couple of IO intensive tasks to confirm if it's mythtv
> (perhaps use VLC to stream from the HDHR to a file?).  I suspect it's
> something on the system itself, not mythtv.
>
> If I am right I would start by verifying all RAID arrays are clean, and if
> all looks good, do a full shutdown.  Then disconnect power and let it sit
> for 15 minutes so any charged caps can dissipate before plugging in and
> turning on.  This has fixed countless weird issues for me in the past,
> primarily with NIC's that support WOL.  Seems as though many components of
> your system don't truly reset anymore until the power is cut completely.

1,2 and 4 are definitely not a factor, 3 and 5 I haven't looked at, I
will look at those, as well as a restart, but it's been running fine
for months, it only started with the a recent git pull that I did. If
I rule out everything else, I'll try to track down the commit that may
have caused it. I have my suspicions... :)

Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


tom at redpepperracing

Jun 19, 2012, 6:14 AM

Post #6 of 13 (1894 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Mon, Jun 18, 2012 at 9:21 AM, Tom Lichti <tom [at] redpepperracing> wrote:
> On Mon, Jun 18, 2012 at 3:48 AM, Joseph Fry <joe [at] thefrys> wrote:
>>>
>>> >> Is anyone else seeing extremely high CPU usage on their backend when
>>> >> playing back *any* content on a remote front end? I just noticed that
>>> >> during playback, the CPU usage of my backend stays around 175-200% on
>>> >> a 2 x Dual core AMD system. I can't imagine what it would be doing
>>> >> other than just streaming the file, is anyone else seeing this, or can
>>> >> explain it?
>>> >>
>>> >> I am on close to current master:
>>> >>
>>> >> MythTV Version : v0.26-pre-629-g181641a-dirty
>>> >> MythTV Branch : master
>>> >> Network Protocol : 75
>>> >> Library API : 0.26.20120614-1
>>> >> QT Version : 4.6.3
>>> >> Options compiled in:
>>> >>  linux profile use_hidesyms using_alsa using_oss using_backend
>>> >> using_bindings_perl using_bindings_python using_bindings_php using_dvb
>>> >> using_frontend using_hdhomerun using_ceton using_hdpvr using_iptv
>>> >> using_ivtv using_joystick_menu using_libcrypto using_libdns_sd
>>> >> using_libxml2 using_lirc using_mheg using_opengl_video using_qtwebkit
>>> >> using_qtscript using_qtdbus using_v4l2 using_v4l1 using_x11
>>> >> using_xrandr using_bindings_perl using_bindings_python
>>> >> using_bindings_php using_mythtranscode using_opengl
>>> >> using_ffmpeg_threads using_live using_mheg using_libxml2
>>> >
>>> > Hmmm....looks like it may have been a memory leak of some sort, I
>>> > started getting OOM killers and lots of swapping. I'll keep an eye on
>>> > it and see if it happens again.
>>> >
>>>
>>> I love talking to myself. Now with one frontend playing back and two
>>> recordings (1 HDHR and 1 HD-PVR on a slave BE) the BE CPU is over
>>> 250%. Lots of free memory (overall system memory is 8GB), so the OOM
>>> doesn't explain it.
>>>
>>
>> The only things I can think of that MAY cause excessive CPU usage on the
>> backend, explicitly during playback/recording are:
>>
>> 1. Jobs (transcode/commercial flagging)
>> 2. Degraded software RAID array.
>> 3. non-DMA disk IO.
>> 4. Encrypted/compressed drive
>> 5. non-DMA network controller, or some on chipset NIC and a really horrible
>> network causing a ton of errors?
>>
>> First try doing a couple of IO intensive tasks to confirm if it's mythtv
>> (perhaps use VLC to stream from the HDHR to a file?).  I suspect it's
>> something on the system itself, not mythtv.
>>
>> If I am right I would start by verifying all RAID arrays are clean, and if
>> all looks good, do a full shutdown.  Then disconnect power and let it sit
>> for 15 minutes so any charged caps can dissipate before plugging in and
>> turning on.  This has fixed countless weird issues for me in the past,
>> primarily with NIC's that support WOL.  Seems as though many components of
>> your system don't truly reset anymore until the power is cut completely.
>
> 1,2 and 4 are definitely not a factor, 3 and 5 I haven't looked at, I
> will look at those, as well as a restart, but it's been running fine
> for months, it only started with the a recent git pull that I did. If
> I rule out everything else, I'll try to track down the commit that may
> have caused it. I have my suspicions... :)

There is definitely a massive memory leak in the back end logging
system, I have opened ticket 10846 with a valgrind log attached.

Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


gjhurlbu at gmail

Jun 20, 2012, 10:13 AM

Post #7 of 13 (1864 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

> There is definitely a massive memory leak in the back end logging
> system, I have opened ticket 10846 with a valgrind log attached.

==4709== LEAK SUMMARY:
==4709== definitely lost: 469,232 bytes in 44,198 blocks
==4709== indirectly lost: 4,918 bytes in 23 blocks
==4709== possibly lost: 31,430,386 bytes in 134,727 blocks

That's hardly a "massive leak"

I will be looking into this anyways. There is definitely something
odd going on.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


tom at redpepperracing

Jun 20, 2012, 10:32 AM

Post #8 of 13 (1866 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Wed, Jun 20, 2012 at 1:13 PM, Gavin Hurlbut <gjhurlbu [at] gmail> wrote:
>> There is definitely a massive memory leak in the back end logging
>> system, I have opened ticket 10846 with a valgrind log attached.
>
> ==4709== LEAK SUMMARY:
> ==4709==    definitely lost: 469,232 bytes in 44,198 blocks
> ==4709==    indirectly lost: 4,918 bytes in 23 blocks
> ==4709==      possibly lost: 31,430,386 bytes in 134,727 blocks
>
> That's hardly a "massive leak"
>
> I will be looking into this anyways.  There is definitely something
> odd going on.

I was seeing ~1GB lost per hour, I think that's fairly substantial. I
did a quick build last night and it has definitely helped, tonight I
hope to do a full clean and rebuild as per Daniel's earlier note just
to make sure I'm on a clean slate.

Thanks
Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


gjhurlbu at gmail

Jun 20, 2012, 11:58 AM

Post #9 of 13 (1861 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

> I was seeing ~1GB lost per hour, I think that's fairly substantial. I
> did a quick build last night and it has definitely helped, tonight I
> hope to do a full clean and rebuild as per Daniel's earlier note just
> to make sure I'm on a clean slate.

I doubt it very much. I think you were seeing 1GB/hr *growth*.
Memory usage != memory leakage. Your actual memory leakage is quite
small, although still much larger than I'd prefer.

We are actively looking into it as I think we are all getting hit by
it to some extent.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


tom at redpepperracing

Jun 20, 2012, 12:17 PM

Post #10 of 13 (1901 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Wed, Jun 20, 2012 at 2:58 PM, Gavin Hurlbut <gjhurlbu [at] gmail> wrote:
>> I was seeing ~1GB lost per hour, I think that's fairly substantial. I
>> did a quick build last night and it has definitely helped, tonight I
>> hope to do a full clean and rebuild as per Daniel's earlier note just
>> to make sure I'm on a clean slate.
>
> I doubt it very much.  I think you were seeing 1GB/hr *growth*.
> Memory usage != memory leakage.  Your actual memory leakage is quite
> small, although still much larger than I'd prefer.
>
> We are actively looking into it as I think we are all getting hit by
> it to some extent.

Ah, yes, that is correct. Although the growth was still quite a bit
larger than I would expect, after a normal evening of viewing the RAM
(all 8GB) was completely exhausted on my server. As I said, once I've
done a full clean and rebuild I'll be better able to gauge the current
status.

Thanks
Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


joe at thefrys

Jun 21, 2012, 8:55 PM

Post #11 of 13 (1830 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

>
> >> I was seeing ~1GB lost per hour, I think that's fairly substantial. I
> >> did a quick build last night and it has definitely helped, tonight I
> >> hope to do a full clean and rebuild as per Daniel's earlier note just
> >> to make sure I'm on a clean slate.
> >
> > I doubt it very much. I think you were seeing 1GB/hr *growth*.
> > Memory usage != memory leakage. Your actual memory leakage is quite
> > small, although still much larger than I'd prefer.
> >
> > We are actively looking into it as I think we are all getting hit by
> > it to some extent.
>
> Ah, yes, that is correct. Although the growth was still quite a bit
> larger than I would expect, after a normal evening of viewing the RAM
> (all 8GB) was completely exhausted on my server. As I said, once I've
> done a full clean and rebuild I'll be better able to gauge the current
> status.
>

How is this memory leak causing high CPU usage. I can see it swapping, but
swapping shouldn't impact the CPU, unless your swapping to software raid or
something silly like that.


stichnot at gmail

Jun 21, 2012, 11:38 PM

Post #12 of 13 (1836 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Thu, Jun 21, 2012 at 8:55 PM, Joseph Fry <joe [at] thefrys> wrote:
> How is this memory leak causing high CPU usage.  I can see it swapping, but
> swapping shouldn't impact the CPU, unless your swapping to software raid or
> something silly like that.

With the right kind of memory leak, an ever-increasing amount of time
can be spent in the malloc/free code due to heap fragmentation.

Jim
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users


tom at redpepperracing

Jun 22, 2012, 8:47 AM

Post #13 of 13 (1823 views)
Permalink
Re: Mythbackend high cpu? [In reply to]

On Fri, Jun 22, 2012 at 2:38 AM, Jim Stichnoth <stichnot [at] gmail> wrote:
> On Thu, Jun 21, 2012 at 8:55 PM, Joseph Fry <joe [at] thefrys> wrote:
>> How is this memory leak causing high CPU usage.  I can see it swapping, but
>> swapping shouldn't impact the CPU, unless your swapping to software raid or
>> something silly like that.
>
> With the right kind of memory leak, an ever-increasing amount of time
> can be spent in the malloc/free code due to heap fragmentation.

I did a complete clean and rebuild as recommended by Daniel in the
other thread, but I have not had a chance to really validate it yet,
but I haven't had any complaints from the family, so that's a good
sign. I will do a full validation tonight.

Tom
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users

MythTV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.