Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Dev

Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback)

 

 

MythTV dev RSS feed   Index | Next | Previous | View Threaded


billymacdonald at gmail

Mar 6, 2012, 8:59 PM

Post #1 of 21 (1473 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback)

On Mar 6, 2012 5:07 PM, <noreply [at] mythtv> wrote:
>
> #10414: HDHomeRun: Bad Recordings
>
-------------------------------------------------+-------------------------
> Reporter: Billy Macdonald <billymacdonald@…> | Owner:
danielk
> Type: Bug Report - General | Status: new
> Priority: minor | Milestone:
unknown
> Component: MythTV - Recording | Version:
> Severity: medium | Unspecified
> Keywords: | Resolution:
> | Ticket locked: 0
>
-------------------------------------------------+-------------------------
>
> Comment (by danielk):
>
> Does this happen with the latest HDHomeRun firmware? I assume some if it
> happens with the prime as well; but I want to be sure you are using RTP
> and not UDP streaming.
>

Yes, I have the latest firmware. I downloaded and installed it about a
week ago as a trouble shooting step. I'm not sure how to verify it was rtp
and not udp.

I'm almost certain it was both the prime and HDhr devices, but I cannot
recall with one hundred percent accuracy right now.

Thanks,
Billy


tom at redpepperracing

Mar 7, 2012, 3:42 PM

Post #2 of 21 (1433 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Tue, Mar 6, 2012 at 11:59 PM, Billy Macdonald
<billymacdonald [at] gmail> wrote:
>
> On Mar 6, 2012 5:07 PM, <noreply [at] mythtv> wrote:
>>
>> #10414: HDHomeRun: Bad Recordings
>>
>> -------------------------------------------------+-------------------------
>>  Reporter:  Billy Macdonald <billymacdonald@…>   |          Owner:
>>  danielk
>>     Type:  Bug Report - General                 |         Status:  new
>>  Priority:  minor                                |      Milestone:
>>  unknown
>> Component:  MythTV - Recording                   |        Version:
>>  Severity:  medium                               |  Unspecified
>>  Keywords:                                       |     Resolution:
>>                                                 |  Ticket locked:  0
>>
>> -------------------------------------------------+-------------------------
>>
>> Comment (by danielk):
>>
>>  Does this happen with the latest HDHomeRun firmware? I assume some if it
>>  happens with the prime as well; but I want to be sure you are using RTP
>>  and not UDP streaming.
>>
>
> Yes,  I have the latest firmware.   I downloaded and installed it about a
> week ago as a trouble shooting step. I'm not sure how to verify it was rtp
> and not udp.
>
> I'm almost certain it was both the prime and HDhr devices, but I cannot
> recall with one hundred percent accuracy right now.

Unfortunately even with the change put in today by Daniel, I still see
the problem. The 4 concurrent HDHR streams are too much. CPU is still
around 75-80%, and all recordings in progress become choppy. It is
definitely the recordings, if I play them back after they are still
choppy.

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


warlord at MIT

Mar 9, 2012, 9:44 AM

Post #3 of 21 (1412 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

Tom Lichti <tom [at] redpepperracing> writes:

>>>  Does this happen with the latest HDHomeRun firmware? I assume some if it
>>>  happens with the prime as well; but I want to be sure you are using RTP
>>>  and not UDP streaming.
>>>
>>
>> Yes,  I have the latest firmware.   I downloaded and installed it about a
>> week ago as a trouble shooting step. I'm not sure how to verify it was rtp
>> and not udp.
>>
>> I'm almost certain it was both the prime and HDhr devices, but I cannot
>> recall with one hundred percent accuracy right now.
>
> Unfortunately even with the change put in today by Daniel, I still see
> the problem. The 4 concurrent HDHR streams are too much. CPU is still
> around 75-80%, and all recordings in progress become choppy. It is
> definitely the recordings, if I play them back after they are still
> choppy.

This leads me to believe the issue is that your backend cannot keep up
with all the data streams. Perhaps the disk is too slow, or there's a
problem with the network card, or interrupts? Can you run 'top' and
see, for example, if e.g. there is ksoftirqd spinning while all these
recordings are running?

I'm seeing similar things here, but I don't think it's a mythtv issue or
an HD Homerun issue. I think it's a hardware issue on my backend
server.

> Tom

-derek
--
Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
Member, MIT Student Information Processing Board (SIPB)
URL: http://web.mit.edu/warlord/ PP-ASEL-IA N1NWH
warlord [at] MIT PGP key available
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


tom at redpepperracing

Mar 9, 2012, 10:17 AM

Post #4 of 21 (1412 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Fri, Mar 9, 2012 at 12:44 PM, Derek Atkins <warlord [at] mit> wrote:
> Tom Lichti <tom [at] redpepperracing> writes:
>
>>>>  Does this happen with the latest HDHomeRun firmware? I assume some if it
>>>>  happens with the prime as well; but I want to be sure you are using RTP
>>>>  and not UDP streaming.
>>>>
>>>
>>> Yes,  I have the latest firmware.   I downloaded and installed it about a
>>> week ago as a trouble shooting step. I'm not sure how to verify it was rtp
>>> and not udp.
>>>
>>> I'm almost certain it was both the prime and HDhr devices, but I cannot
>>> recall with one hundred percent accuracy right now.
>>
>> Unfortunately even with the change put in today by Daniel, I still see
>> the problem. The 4 concurrent HDHR streams are too much. CPU is still
>> around 75-80%, and all recordings in progress become choppy. It is
>> definitely the recordings, if I play them back after they are still
>> choppy.
>
> This leads me to believe the issue is that your backend cannot keep up
> with all the data streams.  Perhaps the disk is too slow, or there's a
> problem with the network card, or interrupts?  Can you run 'top' and
> see, for example, if e.g. there is ksoftirqd spinning while all these
> recordings are running?
>
> I'm seeing similar things here, but I don't think it's a mythtv issue or
> an HD Homerun issue.  I think it's a hardware issue on my backend
> server.

I'm trying to rule that out. I don't think it's network or interrupts,
I have all Gig-E networking, and the backend is actually a server,
Dell Poweredge SC1435, so the only thing I can see is it being the
disk. I'll try putting a storage group on another disk and test that
and report back.

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


danielk at cuymedia

Mar 10, 2012, 8:41 AM

Post #5 of 21 (1409 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Fri, 2012-03-09 at 13:17 -0500, Tom Lichti wrote:
> On Fri, Mar 9, 2012 at 12:44 PM, Derek Atkins <warlord [at] mit> wrote:
> > Tom Lichti <tom [at] redpepperracing> writes:
>
> I'm trying to rule that out. I don't think it's network or interrupts,
> I have all Gig-E networking, and the backend is actually a server,
> Dell Poweredge SC1435, so the only thing I can see is it being the
> disk. I'll try putting a storage group on another disk and test that
> and report back.

I doubt it is the disk. 0.25 dynamically resizes it's buffer up to
about 128 MB per recording vs the user specified buffer that defaulted
to 9.6MB in 0.24.

I was able to reproduce a problem with old firmware using udp
streaming, but with newer firmware employing rtp streaming I
can not reproduce the issue. I'm also not seeing the high CPU
usage reported. I see about 15% CPU usage with two recordings.

Can you take a look at this:
http://www.cyberciti.biz/faq/linux-tcp-tuning/
And tell me if increasing the rmem_max helps?
libmythhdhomerun tries to increase the socket buffer
on startup, but perhaps the rmem_max is set low on
your system? Mine is set to 131071000.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


tom at redpepperracing

Mar 10, 2012, 1:40 PM

Post #6 of 21 (1411 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sat, Mar 10, 2012 at 11:41 AM, Daniel Kristjansson
<danielk [at] cuymedia> wrote:
> On Fri, 2012-03-09 at 13:17 -0500, Tom Lichti wrote:
>> On Fri, Mar 9, 2012 at 12:44 PM, Derek Atkins <warlord [at] mit> wrote:
>> > Tom Lichti <tom [at] redpepperracing> writes:
>>
>> I'm trying to rule that out. I don't think it's network or interrupts,
>> I have all Gig-E networking, and the backend is actually a server,
>> Dell Poweredge SC1435, so the only thing I can see is it being the
>> disk. I'll try putting a storage group on another disk and test that
>> and report back.
>
> I doubt it is the disk. 0.25 dynamically resizes it's buffer up to
> about 128 MB per recording vs the user specified buffer that defaulted
> to 9.6MB in 0.24.
>
> I was able to reproduce a problem with old firmware using udp
> streaming, but with newer firmware employing rtp streaming I
> can not reproduce the issue. I'm also not seeing the high CPU
> usage reported. I see about 15% CPU usage with two recordings.
>
> Can you take a look at this:
>  http://www.cyberciti.biz/faq/linux-tcp-tuning/
> And tell me if increasing the rmem_max helps?
> libmythhdhomerun tries to increase the socket buffer
> on startup, but perhaps the rmem_max is set low on
> your system? Mine is set to 131071000.

I did all the things recommended on that page, with very little change
in behaviour. 4 concurrent HDHR recordings, and all of them are
definitely affected. It is better, again, but still not acceptable.
CPU is about 15% per recording, give or take. No issues noticed in
atop in terms of disk or network utilization. How do we tell if it is
using rtp or not? That could be the problem. I too am on the latest
HDHR firmware, for the record. I am more than willing to test any
patches, if necessary.

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


mrand at pobox

Mar 10, 2012, 4:44 PM

Post #7 of 21 (1422 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sat, Mar 10, 2012 at 10:41 AM, Daniel Kristjansson
<danielk [at] cuymedia> wrote:
> On Fri, 2012-03-09 at 13:17 -0500, Tom Lichti wrote:
>> On Fri, Mar 9, 2012 at 12:44 PM, Derek Atkins <warlord [at] mit> wrote:
>> > Tom Lichti <tom [at] redpepperracing> writes:
>>
>> I'm trying to rule that out. I don't think it's network or interrupts,
>> I have all Gig-E networking, and the backend is actually a server,
>> Dell Poweredge SC1435, so the only thing I can see is it being the
>> disk. I'll try putting a storage group on another disk and test that
>> and report back.
>
> I doubt it is the disk. 0.25 dynamically resizes it's buffer up to
> about 128 MB per recording vs the user specified buffer that defaulted
> to 9.6MB in 0.24.
>
> I was able to reproduce a problem with old firmware using udp
> streaming, but with newer firmware employing rtp streaming I
> can not reproduce the issue. I'm also not seeing the high CPU
> usage reported. I see about 15% CPU usage with two recordings.
>
> Can you take a look at this:
>  http://www.cyberciti.biz/faq/linux-tcp-tuning/
> And tell me if increasing the rmem_max helps?
> libmythhdhomerun tries to increase the socket buffer
> on startup, but perhaps the rmem_max is set low on
> your system? Mine is set to 131071000.

Wow. http://fasterdata.es.net/fasterdata/host-tuning/linux/ seems to
suggest that values 10x smaller would be about good enough for a 10
Gbps network, and that going overboard actually hurts performance.
Wonder which site is right.

Marc
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


danielk at cuymedia

Mar 10, 2012, 5:23 PM

Post #8 of 21 (1403 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sat, 2012-03-10 at 18:44 -0600, Marc Randolph wrote:
> On Sat, Mar 10, 2012 at 10:41 AM, Daniel Kristjansson
> > Can you take a look at this:
> > http://www.cyberciti.biz/faq/linux-tcp-tuning/
> > And tell me if increasing the rmem_max helps?
> > libmythhdhomerun tries to increase the socket buffer
> > on startup, but perhaps the rmem_max is set low on
> > your system? Mine is set to 131071000.
>
> Wow. http://fasterdata.es.net/fasterdata/host-tuning/linux/ seems to
> suggest that values 10x smaller would be about good enough for a 10
> Gbps network, and that going overboard actually hurts performance.
> Wonder which site is right.

Mine is undoubtedly much higher than it needs to be. But setting the
_max values higher never has a negative impact on performance. All it
does is allow applications to set the value that high. MythTV won't
set it that high and most applications will use the default.

wmem_default is only one that can get you in trouble as a networking
end point. If you set the buffering too high it can break the
assumptions in TCP and other dynamically scaling protocols. With
udp and rtp streaming this isn't an issue though. Either you have
sufficient bandwidth or not, packets are never resent and the
HDHomeRun can't send the data at a lower rate because of the
limited bandwidth.

BTW Anyone using a recent HDHomeRun firmware is using rtp streaming.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


gary.buhrmaster at gmail

Mar 10, 2012, 5:41 PM

Post #9 of 21 (1400 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sun, Mar 11, 2012 at 00:44, Marc Randolph <mrand [at] pobox> wrote:
....
> Wow.  http://fasterdata.es.net/fasterdata/host-tuning/linux/ seems to
> suggest that values 10x smaller would be about good enough for a 10
> Gbps network, and that going overboard actually hurts performance.
> Wonder which site is right.

The ESNet site recommendations are targeted for a specific
type of application (TCP based bulk data transfers over longer
distances, with (recommended for best results) dedicated
high performance hosts) that are correct for that application.
If the buffers are "too big" or "too small" you can end up
creating some interesting issues in that targeted space.

For a local area network (on which most people are running
their MythTV systems) running UDP based applications,
some of the recommendations are going to be different.
Except for very memory limited systems, increasing the
rmem_max values modestly will not be a negative. Note
that linux since an early 2.6 kernel has had TCP autotuning,
making many of the past adjustments unnecessary
(and sometimes harmful).
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


tom at redpepperracing

Mar 11, 2012, 7:50 AM

Post #10 of 21 (1387 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
<gary.buhrmaster [at] gmail> wrote:
> On Sun, Mar 11, 2012 at 00:44, Marc Randolph <mrand [at] pobox> wrote:
> ....
>> Wow.  http://fasterdata.es.net/fasterdata/host-tuning/linux/ seems to
>> suggest that values 10x smaller would be about good enough for a 10
>> Gbps network, and that going overboard actually hurts performance.
>> Wonder which site is right.
>
> The ESNet site recommendations are targeted for a specific
> type of application (TCP based bulk data transfers over longer
> distances, with (recommended for best results) dedicated
> high performance hosts) that are correct for that application.
> If the buffers are "too big" or "too small" you can end up
> creating some interesting issues in that targeted space.
>
> For a local area network (on which most people are running
> their MythTV systems) running UDP based applications,
> some of the recommendations are going to be different.
> Except for very memory limited systems, increasing the
> rmem_max values modestly will not be a negative.  Note
> that linux since an early 2.6 kernel has had TCP autotuning,
> making many of the past adjustments unnecessary
> (and sometimes harmful).

Then may I respectfully suggest that we are going down the wrong path
on this? I've used the default network settings, the tweaked settings
on the page Daniel suggested, and Daniel's own settings, and none of
them has made a difference. The only thing that made any noticeable
difference was Daniel's commit the other day, but that only made
recordings go from completely un-watchable to highly annoying (and
still un-watchable).

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


danielk at cuymedia

Mar 11, 2012, 9:13 AM

Post #11 of 21 (1390 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
> On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
> <gary.buhrmaster [at] gmail> wrote:
> Then may I respectfully suggest that we are going down the wrong path
> on this? I've used the default network settings, the tweaked settings
> on the page Daniel suggested, and Daniel's own settings, and none of
> them has made a difference. The only thing that made any noticeable
> difference was Daniel's commit the other day, but that only made
> recordings go from completely un-watchable to highly annoying (and
> still un-watchable).

Thanks for testing the network settings though it was important
to eliminate that possibility. I've also gotten a report off-line
that logging is not the culprit. So far the only commonality
appears to be high CPU usage. My commit slightly lowered CPU usage,
but increasing that timeout further won't do much.

I plan to run an oprofile on mythbackend with a couple HDHomeRun
recordings going sometime in the coming week. Even though I'm not
seeing the issue on my i5, it should still show where CPU is being
used and hopefully show something useful.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


david at istwok

Mar 11, 2012, 9:25 AM

Post #12 of 21 (1390 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sun, Mar 11, 2012 at 12:13:21PM -0400, Daniel Kristjansson wrote:
> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
> > On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
> > <gary.buhrmaster [at] gmail> wrote:
> > Then may I respectfully suggest that we are going down the wrong path
> > on this? I've used the default network settings, the tweaked settings
> > on the page Daniel suggested, and Daniel's own settings, and none of
> > them has made a difference. The only thing that made any noticeable
> > difference was Daniel's commit the other day, but that only made
> > recordings go from completely un-watchable to highly annoying (and
> > still un-watchable).
>
> Thanks for testing the network settings though it was important
> to eliminate that possibility. I've also gotten a report off-line
> that logging is not the culprit. So far the only commonality
> appears to be high CPU usage. My commit slightly lowered CPU usage,
> but increasing that timeout further won't do much.
>
> I plan to run an oprofile on mythbackend with a couple HDHomeRun
> recordings going sometime in the coming week. Even though I'm not
> seeing the issue on my i5, it should still show where CPU is being
> used and hopefully show something useful.

I'm running a git bisect today to try to find when the higher CPU
usage started. That might not be the real cause of the problem, but I
still want to figure that out.

David
--
David Engel
david [at] istwok
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


tom at redpepperracing

Mar 11, 2012, 10:31 AM

Post #13 of 21 (1379 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sun, Mar 11, 2012 at 12:25 PM, David Engel <david [at] istwok> wrote:
> On Sun, Mar 11, 2012 at 12:13:21PM -0400, Daniel Kristjansson wrote:
>> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
>> > On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
>> > <gary.buhrmaster [at] gmail> wrote:
>> > Then may I respectfully suggest that we are going down the wrong path
>> > on this? I've used the default network settings, the tweaked settings
>> > on the page Daniel suggested, and Daniel's own settings, and none of
>> > them has made a difference. The only thing that made any noticeable
>> > difference was Daniel's commit the other day, but that only made
>> > recordings go from completely un-watchable to highly annoying (and
>> > still un-watchable).
>>
>> Thanks for testing the network settings though it was important
>> to eliminate that possibility. I've also gotten a report off-line
>> that logging is not the culprit. So far the only commonality
>> appears to be high CPU usage. My commit slightly lowered CPU usage,
>> but increasing that timeout further won't do much.
>>
>> I plan to run an oprofile on mythbackend with a couple HDHomeRun
>> recordings going sometime in the coming week. Even though I'm not
>> seeing the issue on my i5, it should still show where CPU is being
>> used and hopefully show something useful.
>
> I'm running a git bisect today to try to find when the higher CPU
> usage started.  That might not be the real cause of the problem, but I
> still want to figure that out.

Unfortunately I'm out today or I'd do the same. For doing an oprofile,
what is involved? That is something I can do later tonight, if that
would be helpful, and it's just a matter of recompiling and running a
test.

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


b.taber at comcast

Mar 11, 2012, 11:40 AM

Post #14 of 21 (1388 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On 03/11/2012 12:25 PM, David Engel wrote:
> On Sun, Mar 11, 2012 at 12:13:21PM -0400, Daniel Kristjansson wrote:
>> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
>>> On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
>>> <gary.buhrmaster [at] gmail> wrote:
>>> Then may I respectfully suggest that we are going down the wrong path
>>> on this? I've used the default network settings, the tweaked settings
>>> on the page Daniel suggested, and Daniel's own settings, and none of
>>> them has made a difference. The only thing that made any noticeable
>>> difference was Daniel's commit the other day, but that only made
>>> recordings go from completely un-watchable to highly annoying (and
>>> still un-watchable).
>> Thanks for testing the network settings though it was important
>> to eliminate that possibility. I've also gotten a report off-line
>> that logging is not the culprit. So far the only commonality
>> appears to be high CPU usage. My commit slightly lowered CPU usage,
>> but increasing that timeout further won't do much.
>>
>> I plan to run an oprofile on mythbackend with a couple HDHomeRun
>> recordings going sometime in the coming week. Even though I'm not
>> seeing the issue on my i5, it should still show where CPU is being
>> used and hopefully show something useful.
> I'm running a git bisect today to try to find when the higher CPU
> usage started. That might not be the real cause of the problem, but I
> still want to figure that out.
>
> David
I began to see these types of bad recordings about the same time as the
commit for the code that flags them as bad. Back in November? I don't
believe that was the culprit but more of a coincidence. I don't really
have a way to determine that beyond a shadow of a doubt.

I've gone round and round in testing everything I could think of to see
if I could identify the problem. The last step has been to replace the
slave backend system with one that is more powerful. That has seemed to
alleviate the problem. The same drives from the old system were moved
into the new. I think most of the bad recordings that still exist in my
setup are ones that were recorded on the previous slave. I hadn't
noticed any incidences of high CPU usage but they may have been missed.
And I haven't seen any new recordings with the same problem.

Bruce
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


adeffs.mythtv at gmail

Mar 11, 2012, 12:05 PM

Post #15 of 21 (1385 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings (was: Bad Recordings - Choppy Playback) [In reply to]

On Sun, Mar 11, 2012 at 12:13 PM, Daniel Kristjansson
<danielk [at] cuymedia> wrote:
> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
>> On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
>> <gary.buhrmaster [at] gmail> wrote:
>> Then may I respectfully suggest that we are going down the wrong path
>> on this? I've used the default network settings, the tweaked settings
>> on the page Daniel suggested, and Daniel's own settings, and none of
>> them has made a difference. The only thing that made any noticeable
>> difference was Daniel's commit the other day, but that only made
>> recordings go from completely un-watchable to highly annoying (and
>> still un-watchable).
>
> Thanks for testing the network settings though it was important
> to eliminate that possibility. I've also gotten a report off-line
> that logging is not the culprit. So far the only commonality
> appears to be high CPU usage. My commit slightly lowered CPU usage,
> but increasing that timeout further won't do much.
>
> I plan to run an oprofile on mythbackend with a couple HDHomeRun
> recordings going sometime in the coming week. Even though I'm not
> seeing the issue on my i5, it should still show where CPU is being
> used and hopefully show something useful.
>
> -- Daniel

I'm noticing high CPU usage with my setup of a Ceton and two DVB cards
as well, I don't know if it's related to this problem though? Is there
some trouble shooting I can do to see? should I start a new email?

--
Steve
http://www.mythtv.org/wiki/User:Steveadeff
Before you ask, read the FAQ!
http://www.mythtv.org/wiki/Frequently_Asked_Questions
then search the Wiki, and this list,
http://www.gossamer-threads.com/lists/mythtv/
Mailinglist etiquette - http://www.mythtv.org/wiki/Mailing_List_etiquette
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


david at istwok

Mar 11, 2012, 1:29 PM

Post #16 of 21 (1372 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On Sun, Mar 11, 2012 at 02:40:19PM -0400, Bruce Taber wrote:
> On 03/11/2012 12:25 PM, David Engel wrote:
> > On Sun, Mar 11, 2012 at 12:13:21PM -0400, Daniel Kristjansson wrote:
> >> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
> >>> On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
> >>> <gary.buhrmaster [at] gmail> wrote:
> >>> Then may I respectfully suggest that we are going down the wrong path
> >>> on this? I've used the default network settings, the tweaked settings
> >>> on the page Daniel suggested, and Daniel's own settings, and none of
> >>> them has made a difference. The only thing that made any noticeable
> >>> difference was Daniel's commit the other day, but that only made
> >>> recordings go from completely un-watchable to highly annoying (and
> >>> still un-watchable).
> >> Thanks for testing the network settings though it was important
> >> to eliminate that possibility. I've also gotten a report off-line
> >> that logging is not the culprit. So far the only commonality
> >> appears to be high CPU usage. My commit slightly lowered CPU usage,
> >> but increasing that timeout further won't do much.
> >>
> >> I plan to run an oprofile on mythbackend with a couple HDHomeRun
> >> recordings going sometime in the coming week. Even though I'm not
> >> seeing the issue on my i5, it should still show where CPU is being
> >> used and hopefully show something useful.
> > I'm running a git bisect today to try to find when the higher CPU
> > usage started. That might not be the real cause of the problem, but I
> > still want to figure that out.
> >
> > David
> I began to see these types of bad recordings about the same time as the
> commit for the code that flags them as bad. Back in November? I don't
> believe that was the culprit but more of a coincidence. I don't really
> have a way to determine that beyond a shadow of a doubt.

Your recollection is spot on except for being just a hair off on the
timing.

Daniel, I traced my high CPU usage to the following commit.

commit ca0419d8969b97bba80c590e5fed02f7c1d948c8
Author: Daniel Thor Kristjansson <danielk [at] cuymedia>
Date: Fri Dec 2 16:19:42 2011 -0500

Adds recording quality tracking to DTV recorders.

Specifically, it is the following code in DTVRecorder::BufferedWrite().

if (!timeOfFirstData.isValid() && curRecording)
{
QMutexLocker locker(&statisticsLock);
timeOfFirstData = mythCurrentDateTime();
}

uint64_t now = mythCurrentDateTime().toTime_t();
if (!timeOfLatestData.isValid() || (now - timeOfLatestData.toTime_t() >= 5))
{
QMutexLocker locker(&statisticsLock);
timeOfLatestData = mythCurrentDateTime();
}

With that code commented out and 6 tuners busy, my mythbackend CPU
usage drops from ~150% (1.5 of 2 cores busy) to ~20%. The
QMutexLocker is responsible for some of the higher usage, but the
biggest culprits are the toTime_t() and isValid() time calls.

Strangely, with my debugging reverted and all tuners busy, I didn't
see the same recording corruption I did yesterday. Perhaps my problem
yesterday was transient or I'm right on the edge and a slightly
different mix of content pushes my over.

David

> I've gone round and round in testing everything I could think of to see
> if I could identify the problem. The last step has been to replace the
> slave backend system with one that is more powerful. That has seemed to
> alleviate the problem. The same drives from the old system were moved
> into the new. I think most of the bad recordings that still exist in my
> setup are ones that were recorded on the previous slave. I hadn't
> noticed any incidences of high CPU usage but they may have been missed.
> And I haven't seen any new recordings with the same problem.
>
> Bruce
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev [at] mythtv
> http://www.mythtv.org/mailman/listinfo/mythtv-dev
>

--
David Engel
david [at] istwok
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


tom at redpepperracing

Mar 11, 2012, 5:57 PM

Post #17 of 21 (1359 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On Sun, Mar 11, 2012 at 4:29 PM, David Engel <david [at] istwok> wrote:
> On Sun, Mar 11, 2012 at 02:40:19PM -0400, Bruce Taber wrote:
>> On 03/11/2012 12:25 PM, David Engel wrote:
>> > On Sun, Mar 11, 2012 at 12:13:21PM -0400, Daniel Kristjansson wrote:
>> >> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
>> >>> On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
>> >>> <gary.buhrmaster [at] gmail>  wrote:
>> >>> Then may I respectfully suggest that we are going down the wrong path
>> >>> on this? I've used the default network settings, the tweaked settings
>> >>> on the page Daniel suggested, and Daniel's own settings, and none of
>> >>> them has made a difference. The only thing that made any noticeable
>> >>> difference was Daniel's commit the other day, but that only made
>> >>> recordings go from completely un-watchable to highly annoying (and
>> >>> still un-watchable).
>> >> Thanks for testing the network settings though it was important
>> >> to eliminate that possibility. I've also gotten a report off-line
>> >> that logging is not the culprit. So far the only commonality
>> >> appears to be high CPU usage. My commit slightly lowered CPU usage,
>> >> but increasing that timeout further won't do much.
>> >>
>> >> I plan to run an oprofile on mythbackend with a couple HDHomeRun
>> >> recordings going sometime in the coming week. Even though I'm not
>> >> seeing the issue on my i5, it should still show where CPU is being
>> >> used and hopefully show something useful.
>> > I'm running a git bisect today to try to find when the higher CPU
>> > usage started.  That might not be the real cause of the problem, but I
>> > still want to figure that out.
>> >
>> > David
>> I began to see these types of bad recordings about the same time as the
>> commit for the code that flags them as bad. Back in November? I don't
>> believe that was the culprit but more of a coincidence. I don't really
>> have a way to determine that beyond a shadow of a doubt.
>
> Your recollection is spot on except for being just a hair off on the
> timing.
>
> Daniel, I traced my high CPU usage to the following commit.
>
> commit ca0419d8969b97bba80c590e5fed02f7c1d948c8
> Author: Daniel Thor Kristjansson <danielk [at] cuymedia>
> Date:   Fri Dec 2 16:19:42 2011 -0500
>
>    Adds recording quality tracking to DTV recorders.
>
> Specifically, it is the following code in DTVRecorder::BufferedWrite().
>
>    if (!timeOfFirstData.isValid() && curRecording)
>    {
>        QMutexLocker locker(&statisticsLock);
>        timeOfFirstData = mythCurrentDateTime();
>    }
>
>    uint64_t now = mythCurrentDateTime().toTime_t();
>    if (!timeOfLatestData.isValid() || (now - timeOfLatestData.toTime_t() >= 5))
>    {
>        QMutexLocker locker(&statisticsLock);
>        timeOfLatestData = mythCurrentDateTime();
>    }
>
> With that code commented out and 6 tuners busy, my mythbackend CPU
> usage drops from ~150% (1.5 of 2 cores busy) to ~20%.  The
> QMutexLocker is responsible for some of the higher usage, but the
> biggest culprits are the toTime_t() and isValid() time calls.

Is it just a matter of commenting out those lines? If so, I can try
that tonight.

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


david at istwok

Mar 11, 2012, 6:07 PM

Post #18 of 21 (1350 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On Sun, Mar 11, 2012 at 08:57:58PM -0400, Tom Lichti wrote:
> On Sun, Mar 11, 2012 at 4:29 PM, David Engel <david [at] istwok> wrote:
> > On Sun, Mar 11, 2012 at 02:40:19PM -0400, Bruce Taber wrote:
> >> On 03/11/2012 12:25 PM, David Engel wrote:
> >> > On Sun, Mar 11, 2012 at 12:13:21PM -0400, Daniel Kristjansson wrote:
> >> >> On Sun, 2012-03-11 at 10:50 -0400, Tom Lichti wrote:
> >> >>> On Sat, Mar 10, 2012 at 8:41 PM, Gary Buhrmaster
> >> >>> <gary.buhrmaster [at] gmail>  wrote:
> >> >>> Then may I respectfully suggest that we are going down the wrong path
> >> >>> on this? I've used the default network settings, the tweaked settings
> >> >>> on the page Daniel suggested, and Daniel's own settings, and none of
> >> >>> them has made a difference. The only thing that made any noticeable
> >> >>> difference was Daniel's commit the other day, but that only made
> >> >>> recordings go from completely un-watchable to highly annoying (and
> >> >>> still un-watchable).
> >> >> Thanks for testing the network settings though it was important
> >> >> to eliminate that possibility. I've also gotten a report off-line
> >> >> that logging is not the culprit. So far the only commonality
> >> >> appears to be high CPU usage. My commit slightly lowered CPU usage,
> >> >> but increasing that timeout further won't do much.
> >> >>
> >> >> I plan to run an oprofile on mythbackend with a couple HDHomeRun
> >> >> recordings going sometime in the coming week. Even though I'm not
> >> >> seeing the issue on my i5, it should still show where CPU is being
> >> >> used and hopefully show something useful.
> >> > I'm running a git bisect today to try to find when the higher CPU
> >> > usage started.  That might not be the real cause of the problem, but I
> >> > still want to figure that out.
> >> >
> >> > David
> >> I began to see these types of bad recordings about the same time as the
> >> commit for the code that flags them as bad. Back in November? I don't
> >> believe that was the culprit but more of a coincidence. I don't really
> >> have a way to determine that beyond a shadow of a doubt.
> >
> > Your recollection is spot on except for being just a hair off on the
> > timing.
> >
> > Daniel, I traced my high CPU usage to the following commit.
> >
> > commit ca0419d8969b97bba80c590e5fed02f7c1d948c8
> > Author: Daniel Thor Kristjansson <danielk [at] cuymedia>
> > Date:   Fri Dec 2 16:19:42 2011 -0500
> >
> >    Adds recording quality tracking to DTV recorders.
> >
> > Specifically, it is the following code in DTVRecorder::BufferedWrite().
> >
> >    if (!timeOfFirstData.isValid() && curRecording)
> >    {
> >        QMutexLocker locker(&statisticsLock);
> >        timeOfFirstData = mythCurrentDateTime();
> >    }
> >
> >    uint64_t now = mythCurrentDateTime().toTime_t();
> >    if (!timeOfLatestData.isValid() || (now - timeOfLatestData.toTime_t() >= 5))
> >    {
> >        QMutexLocker locker(&statisticsLock);
> >        timeOfLatestData = mythCurrentDateTime();
> >    }
> >
> > With that code commented out and 6 tuners busy, my mythbackend CPU
> > usage drops from ~150% (1.5 of 2 cores busy) to ~20%.  The
> > QMutexLocker is responsible for some of the higher usage, but the
> > biggest culprits are the toTime_t() and isValid() time calls.
>
> Is it just a matter of commenting out those lines? If so, I can try
> that tonight.

Doing so could very likely cause the damaged recording detection to
mark everything as damaged. If you don't mind that or can have a time
window just for testing, then sure, give it a try.

David
--
David Engel
david [at] istwok
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


danielk at cuymedia

Mar 12, 2012, 5:50 AM

Post #19 of 21 (1354 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On Sun, 2012-03-11 at 15:29 -0500, David Engel wrote:

> Daniel, I traced my high CPU usage to the following commit.

> Specifically, it is the following code in DTVRecorder::BufferedWrite().
>
> if (!timeOfFirstData.isValid() && curRecording)
> {
> QMutexLocker locker(&statisticsLock);
> timeOfFirstData = mythCurrentDateTime();
> }
>
> uint64_t now = mythCurrentDateTime().toTime_t();
> if (!timeOfLatestData.isValid() || (now - timeOfLatestData.toTime_t() >= 5))
> {
> QMutexLocker locker(&statisticsLock);
> timeOfLatestData = mythCurrentDateTime();
> }

Makes sense, QDateTime can take up crazy amounts of CPU. I'll rework
this so we only need to look up the time every five seconds or so.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


raymond at wagnerrp

Mar 12, 2012, 7:22 AM

Post #20 of 21 (1350 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On 3/12/2012 08:50, Daniel Kristjansson wrote:
> On Sun, 2012-03-11 at 15:29 -0500, David Engel wrote:
>
>> Daniel, I traced my high CPU usage to the following commit.
>>
>> Specifically, it is the following code in DTVRecorder::BufferedWrite().
>>
>> if (!timeOfFirstData.isValid()&& curRecording)
>> {
>> QMutexLocker locker(&statisticsLock);
>> timeOfFirstData = mythCurrentDateTime();
>> }
>>
>> uint64_t now = mythCurrentDateTime().toTime_t();
>> if (!timeOfLatestData.isValid() || (now - timeOfLatestData.toTime_t()>= 5))
>> {
>> QMutexLocker locker(&statisticsLock);
>> timeOfLatestData = mythCurrentDateTime();
>> }
> Makes sense, QDateTime can take up crazy amounts of CPU. I'll rework
> this so we only need to look up the time every five seconds or so.

If the expense is in QDateTime's internal conversions from one type to
the next, is there any reason to not just use time() directly? Looking
through recorderbase.cpp and recordingquality.cpp, I don't see anything
that has need for better than the second precision provided by time_t,
and QDateTime is just used for convenience with its toString() method.
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


danielk at cuymedia

Mar 12, 2012, 8:34 AM

Post #21 of 21 (1344 views)
Permalink
Re: Ticket #10414: HDHomeRun: Bad Recordings [In reply to]

On Mon, 2012-03-12 at 10:22 -0400, Raymond Wagner wrote:
> If the expense is in QDateTime's internal conversions from one type to
> the next, is there any reason to not just use time() directly? Looking
> through recorderbase.cpp and recordingquality.cpp, I don't see anything
> that has need for better than the second precision provided by time_t,
> and QDateTime is just used for convenience with its toString() method.

Even just calling time() or using MythTimer::elapsed() would
cause us to enter the kernel context which we don't want to
do on every packet. Ideally we'd just use a timer that checks
if we've seen data since it was last fired. But since we're
not using a Qt event thread in the recorder we don't have
access to Qt timers. So what I've done is just introduce a
counter which get set to approximately the number of packets
seen in the every five seconds. Then we only update the time
when we've seen the appropriate number of packets.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev

MythTV dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.