Mailing List Archive: MythTV: Users

mythbackend leaking file descriptors/sockets?

Index | Next | Previous | View Threaded

mythtv at ebel

Aug 25, 2009, 11:19 PM

Post #1 of 11 (1378 views)
Permalink

mythbackend leaking file descriptors/sockets?

I just discovered that mythbackend (21-fixes) seems to be leaking file
descriptors for me. This probably only started happening recently as
the symptoms manifest within a couple days of starting it, and I've
never seen them before. The symptom is basically that it seems very
unresponsive, via mythweb or the frontend. In mythweb, the shows for
today all have no thumbnail images, just black rectangles. I straced
the backend and found that it was out of file descriptors. The file
descriptors seem to all be taken up by socket connections.

It's possible that the overused file descriptors are a symptom of
something else being broken. I'll continue to investigate. I just
wondered if anyone had seen this before and could point me in the right
direction.

For now I'll monitor the file descriptor usage over time and turn on
socket debugging on the backend to see what I come up with.

Thanks,
Joel
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

waxrat at comcast

Aug 26, 2009, 3:27 PM

Post #2 of 11 (1303 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

Joel <mythtv [at] ebel> writes:

> For now I'll monitor the file descriptor usage over time and turn on socket
> debugging on the backend to see what I come up with.

In case you're not already looking in /proc, you might find
something interesting there.

sudo ls -l /proc/$(pgrep mythback)/fd
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at ebel

Aug 27, 2009, 12:05 AM

Post #3 of 11 (1303 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

Michael Cook wrote:
> In case you're not already looking in /proc, you might find
> something interesting there.
>
> sudo ls -l /proc/$(pgrep mythback)/fd

That's where I got the information that most were sockets. I do not
know how to determine what is or isn't on the other end of such sockets.

They all look like:
455 -> socket:[2819498927]

After looking at my logs, it looks the the EIT scanner is responsible
for this. I'm getting 2 new sockets opened every 5 minutes, and these
seem to correspond to when the EIT scanner runs every 5 minutes on both
of my hdhomerun inputs. I'm guessing the two sockets are the two inputs.

In this case, I'm inclined to turn off EIT scanning and forget about it.
I have schedulesdirect, so EIT scanning seems unnecessary, though it
seems like it could be nice to get updated information from the channel.
I don't know if it winds up being any more accurate than
schedulesdirect, for instance if a show runs long, I don't know if the
EIT data gets updated.

However, I'm not usually one to leave a problem unreported, so if any
devs are interested in why the EIT scanner on my system is leaking file
descriptors, I'm happy to file a bug and/or provide more data if you
request it.

Joel

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mtdean at thirdcontact

Aug 27, 2009, 1:37 AM

Post #4 of 11 (1299 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

On 08/27/2009 03:05 AM, Joel wrote:
> In this case, I'm inclined to turn off EIT scanning and forget about
> it. I have schedulesdirect, so EIT scanning seems unnecessary, though
> it seems like it could be nice to get updated information from the
> channel. I don't know if it winds up being any more accurate than
> schedulesdirect, for instance if a show runs long, I don't know if the
> EIT data gets updated.

Mixing "Internet" (Schedules Direct or XMLTV) listings and EIT on the
same channel is not supported. If you enable EIT on a channel for which
you have Internet listings, the EIT will stomp all over the data,
breaking duplicate matching and possibly more (including scheduling).
In the future, we may have a "safe update" approach that allows mixing
them, but we don't now.

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at ebel

Aug 27, 2009, 9:12 AM

Post #5 of 11 (1272 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

Michael T. Dean wrote:
> Mixing "Internet" (Schedules Direct or XMLTV) listings and EIT on the
> same channel is not supported. If you enable EIT on a channel for which
> you have Internet listings, the EIT will stomp all over the data,
> breaking duplicate matching and possibly more (including scheduling).
> In the future, we may have a "safe update" approach that allows mixing
> them, but we don't now.

Maybe we're talking about something different then. When I select
SchedulesDirect as my listings grabber in mythtv-setup, an options
called "Perform EIT Scan" shows up. It appears as a sub-option, only
visible when SchedulesDirect is selected. If this option were somehow
incompatible with having SchedulesDirect selected, it's in entirely the
wrong place in the UI, and also completely invisible at any time it
might otherwise be useful, if it is at all useful. Under what context
should the Perform EIT Scan option be checked if SchedulesDirect is
used? If the answer is "never", then why is it there?

Also, if EIT scanning and SchedulesDirect data are incompatible, wy
doesn't the EIT scanner to a check to see if SchedulesDirect data
exists, and if so, exit? The failure mode of leaking file descriptors
at a rate that breaks the backend in two days seems an odd choice.

Joel
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mtdean at thirdcontact

Aug 27, 2009, 10:53 AM

Post #6 of 11 (1273 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

On 08/27/2009 12:12 PM, Joel wrote:
> Michael T. Dean wrote:
>> Mixing "Internet" (Schedules Direct or XMLTV) listings and EIT on the
>> same channel is not supported. If you enable EIT on a channel for
>> which you have Internet listings, the EIT will stomp all over the
>> data, breaking duplicate matching and possibly more (including
>> scheduling). In the future, we may have a "safe update" approach
>> that allows mixing them, but we don't now.
>
> Maybe we're talking about something different then. When I select
> SchedulesDirect as my listings grabber in mythtv-setup, an options
> called "Perform EIT Scan" shows up. It appears as a sub-option, only
> visible when SchedulesDirect is selected. If this option were somehow
> incompatible with having SchedulesDirect selected, it's in entirely
> the wrong place in the UI, and also completely invisible at any time
> it might otherwise be useful, if it is at all useful. Under what
> context should the Perform EIT Scan option be checked if
> SchedulesDirect is used? If the answer is "never", then why is it there?

The "on the same channel" portion of my comment is the important
distinction. Note that it's quite possible that you may have a video
source for which you wish to use Schedules Direct. However, some, for
example, small-time channel in the area may not provide its listings to
TMS/Schedules Direct; therefore, you may choose to enable EIT on the
specific channel for which you have no listings data from Schedules Direct.

The Video Sources "Perform EIT scan" setting /allows/ you to use EIT on
the video source. However, each channel within that source also has a
setting, "Use on air guide," which you enable or disable to specify
whether to use EIT for that particular channel. (Note that it defaults
to true, so you'll need to disable it for all channels for which you're
using Schedules Direct to retrieve listings.)

Therefore, since many fewer users actually have video sources where they
need to use SD/XMLTV on some channel and EIT on others, Myth is designed
to allow easy configuration for using EIT on some or none of the
channels (and takes more work for the "corner case" of using different
ones on different channels within the source). If you want to use EIT
for all channels, simply enable the "Perform EIT Scan" setting and,
since the channels all default to true for "Use on air guide," EIT will
be used on all channels. If you want to use SD/XMLTV on all channels,
simply disable the "Perform EIT Scan" setting and, even though the
channels default to true for "Use on air guide," EIT will not be used on
the channel in the video source since EIT is not allowed on the video
source.

> Also, if EIT scanning and SchedulesDirect data are incompatible, wy
> doesn't the EIT scanner to a check to see if SchedulesDirect data
> exists, and if so, exit?

Patches welcomed. :) (Though, really, the time would probably be
better spent working on a proper fix for
http://svn.mythtv.org/trac/ticket/1770 . If interested, you should
mention your proposed changes (or ask for design recommendations) on the
-dev list as a couple of the developers already have an idea of how it
should be implemented.)

> The failure mode of leaking file descriptors at a rate that breaks
> the backend in two days seems an odd choice.

That's unrelated to the mixing of EIT and SD--however, the lack of
support for mixing means you won't lose anything (and will actually gain
a better-working system) by disabling EIT.

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at ebel

Aug 27, 2009, 2:30 PM

Post #7 of 11 (1265 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

Michael T. Dean wrote:
> On 08/27/2009 12:12 PM, Joel wrote:
>> Michael T. Dean wrote:
>>> Mixing "Internet" (Schedules Direct or XMLTV) listings and EIT on the
>>> same channel is not supported. If you enable EIT on a channel for
>>> which you have Internet listings, the EIT will stomp all over the
>>> data, breaking duplicate matching and possibly more (including
>>> scheduling). In the future, we may have a "safe update" approach
>>> that allows mixing them, but we don't now.
>>
>> Maybe we're talking about something different then. When I select
>> SchedulesDirect as my listings grabber in mythtv-setup, an options
>> called "Perform EIT Scan" shows up. It appears as a sub-option, only
>> visible when SchedulesDirect is selected. If this option were somehow
>> incompatible with having SchedulesDirect selected, it's in entirely
>> the wrong place in the UI, and also completely invisible at any time
>> it might otherwise be useful, if it is at all useful. Under what
>> context should the Perform EIT Scan option be checked if
>> SchedulesDirect is used? If the answer is "never", then why is it there?
>
> The "on the same channel" portion of my comment is the important
> distinction. Note that it's quite possible that you may have a video
> source for which you wish to use Schedules Direct. However, some, for
> example, small-time channel in the area may not provide its listings to
> TMS/Schedules Direct; therefore, you may choose to enable EIT on the
> specific channel for which you have no listings data from Schedules Direct.
>
> The Video Sources "Perform EIT scan" setting /allows/ you to use EIT on
> the video source. However, each channel within that source also has a
> setting, "Use on air guide," which you enable or disable to specify
> whether to use EIT for that particular channel. (Note that it defaults
> to true, so you'll need to disable it for all channels for which you're
> using Schedules Direct to retrieve listings.)
>
> Therefore, since many fewer users actually have video sources where they
> need to use SD/XMLTV on some channel and EIT on others, Myth is designed
> to allow easy configuration for using EIT on some or none of the
> channels (and takes more work for the "corner case" of using different
> ones on different channels within the source). If you want to use EIT
> for all channels, simply enable the "Perform EIT Scan" setting and,
> since the channels all default to true for "Use on air guide," EIT will
> be used on all channels. If you want to use SD/XMLTV on all channels,
> simply disable the "Perform EIT Scan" setting and, even though the
> channels default to true for "Use on air guide," EIT will not be used on
> the channel in the video source since EIT is not allowed on the video
> source.

That makes sense. Thanks for the explanation. I do have a few local
channels with no SD data. I may try enabling EIT out of curiosity to
see if the fd leaks still happen only on those channels.

>
>> Also, if EIT scanning and SchedulesDirect data are incompatible, wy
>> doesn't the EIT scanner to a check to see if SchedulesDirect data
>> exists, and if so, exit?
>
> Patches welcomed. :) (Though, really, the time would probably be
> better spent working on a proper fix for
> http://svn.mythtv.org/trac/ticket/1770 . If interested, you should
> mention your proposed changes (or ask for design recommendations) on the
> -dev list as a couple of the developers already have an idea of how it
> should be implemented.)

We'll see if I get the time and determination to look into it any
further. I'm rather swamped at the moment, and was only forced to look
into this problem as far as I did because mythbackend was needing to be
restarted every 42 hours, which got annoying. That bug is interesting
though, I remember seeing behavior like that at some point in the past
as well. I probably disabled EIT scanning at the time to fix it and
forgot about it.

>
>> The failure mode of leaking file descriptors at a rate that breaks
>> the backend in two days seems an odd choice.
>
> That's unrelated to the mixing of EIT and SD--however, the lack of
> support for mixing means you won't lose anything (and will actually gain
> a better-working system) by disabling EIT.

I'll start by re-enabling EIT just on channels without SD data. If I
still see it leaking fd's I'll try to dig a little deeper.

Thanks,
Joel
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

udovdh at xs4all

Aug 29, 2009, 1:27 AM

Post #8 of 11 (1254 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

On 2009-08-26 08:19, Joel wrote:
> For now I'll monitor the file descriptor usage over time and turn on
> socket debugging on the backend to see what I come up with.

Did you also monitor the memory consumption of the machine as a whole
and of the backend specifically?
If your backend's memory consumption is not perpetually increasing and
causing these slowdowns you may really have another 'leakage' on your hands.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at ebel

Aug 30, 2009, 12:19 AM

Post #9 of 11 (1215 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

Joel wrote:
> I'll start by re-enabling EIT just on channels without SD data. If I
> still see it leaking fd's I'll try to dig a little deeper.

It turns out that even if I only scan channels with no SD data, I still
get a fd leak. Here's an example of an strace of a thread responsible
for the leak:
http://ebel.mybox.org/fdleak.trace

Any ideas what might be causing that? I'm not familiar enough with the
code to trace this back to what's causing it. It's obvious that the EIT
scanner is related, and that this thread is communicating with the
hdhomerun on fd 7. But I can't figure what it opens fd 24 for. And
apparently neither can mythbackend.

Thanks,
Joel
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at ebel

Aug 30, 2009, 12:21 AM

Post #10 of 11 (1214 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

Udo van den Heuvel wrote:
> On 2009-08-26 08:19, Joel wrote:
>> For now I'll monitor the file descriptor usage over time and turn on
>> socket debugging on the backend to see what I come up with.
>
> Did you also monitor the memory consumption of the machine as a whole
> and of the backend specifically?
> If your backend's memory consumption is not perpetually increasing and
> causing these slowdowns you may really have another 'leakage' on your hands.

I do not see a significant increase in memory usage. Just in file
descriptors. mythbackend runs just fine for 42 hours until it hits the
1024 fd limit. Then it fails on anything that tries to open a new one.

Joel
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

udovdh at xs4all

Aug 30, 2009, 12:48 AM

Post #11 of 11 (1216 views)
Permalink

Re: mythbackend leaking file descriptors/sockets? [In reply to]

On 2009-08-30 09:21, Joel wrote:
>> If your backend's memory consumption is not perpetually increasing and
>> causing these slowdowns you may really have another 'leakage' on your
>> hands.
>
> I do not see a significant increase in memory usage. Just in file
> descriptors. mythbackend runs just fine for 42 hours until it hits the
> 1024 fd limit. Then it fails on anything that tries to open a new one.

Wow!
That's even worse than the memory leak I see.
Luckily you can run debugging tools. Would valgrind help in your case?

In my case the machine is too slow to run mythbackend *and* valgrind so
the bug I see is not being fixed. (yes, go figure)

%CPU %MEM VSZ RSS TTY STAT START TIME
8.8 7.5 406932 73388 ? Ssl Jul12 66:16
9.5 8.9 419968 86440 ? Ssl Jul12 208:24
9.7 10.0 430568 97152 ? Ssl Jul12 352:41
9.8 11.2 442716 109300 ? Ssl Jul12 498:47
9.8 12.4 454348 120588 ? Ssl Jul12 642:41
9.9 13.6 465720 132804 ? Ssl Jul12 787:34
9.8 14.4 474128 140428 ? Ssl Jul12 929:27
9.8 15.5 483596 150588 ? Ssl Jul12 1067:41
9.8 16.6 496132 162028 ? Ssl Jul12 1213:01
9.9 18.0 508792 174968 ? Ssl Jul12 1363:09
9.9 19.1 519904 185872 ? Ssl Jul12 1507:43
9.9 20.2 531156 196576 ? Ssl Jul12 1654:26
9.9 21.4 543420 208636 ? Ssl Jul12 1801:54
9.9 22.4 552212 217884 ? Ssl Jul12 1939:52
9.9 23.2 561016 225672 ? Ssl Jul12 2075:35
9.9 24.5 573292 238180 ? Ssl Jul12 2221:10
9.9 25.9 586980 251756 ? Ssl Jul12 2370:42
9.9 27.2 598616 264180 ? Ssl Jul12 2517:40
10.0 28.5 611928 276940 ? Ssl Jul12 2668:58
10.0 29.3 595772 284880 ? Ssl Jul12 2817:10
9.9 28.7 633624 278868 ? Ssl Jul12 2955:12
9.9 28.8 642948 279992 ? Ssl Jul12 3093:19
9.9 29.5 647676 286624 ? Ssl Jul12 3237:22
10.0 29.9 669528 290960 ? Ssl Jul12 3390:36
10.0 30.0 682020 291436 ? Ssl Jul12 3538:38
10.0 27.9 693508 271320 ? Ssl Jul12 3691:24
10.0 28.2 706940 274180 ? Ssl Jul12 3846:07
10.0 28.5 715216 276652 ? Ssl Jul12 3984:52
10.0 28.8 723620 279748 ? Ssl Jul12 4120:59
10.0 29.4 737340 285600 ? Ssl Jul12 4268:13
10.0 30.1 750652 292900 ? Ssl Jul12 4422:32
10.0 30.6 763824 297348 ? Ssl Jul12 4576:37
10.0 31.1 776140 302624 ? Ssl Jul12 4729:27
10.1 32.0 788544 310632 ? Ssl Jul12 4883:11
10.1 32.2 798108 312792 ? Ssl Jul12 5023:18
10.0 32.2 806388 312944 ? Ssl Jul12 5162:54
10.0 32.8 820472 318984 ? Ssl Jul12 5311:43
10.1 31.1 833268 301840 ? Ssl Jul12 5462:41
10.1 31.4 836716 305508 ? Ssl Jul12 5614:35
10.1 32.2 857712 313332 ? Ssl Jul12 5764:40
10.1 28.5 873696 276856 ? Ssl Jul12 5921:53
10.1 28.0 882124 271832 ? Ssl Jul12 6064:38
10.1 28.0 890392 272616 ? Ssl Jul12 6205:41
10.1 28.3 902788 274720 ? Ssl Jul12 6355:56
10.1 28.8 916376 279948 ? Ssl Jul12 6512:21
10.1 29.2 929284 284140 ? Ssl Jul12 6662:59

The typical graph looks like this:
http://img151.imageshack.us/img151/4247/mythbackendeatsmemory.png
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads