Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Dev

Expiry problem

 

 

MythTV dev RSS feed   Index | Next | Previous | View Threaded


ian at beware

Apr 17, 2012, 4:10 PM

Post #1 of 14 (824 views)
Permalink
Expiry problem

Hi,

I have been running a recent 0.25 pre-release(*) and have run into
problems with expiry. What happens is that the disk fills up, the expiry
thread runs, deletes the entry from the database, but does not delete
the actual files containing the recording. Since the disk is still full,
it goes ahead and deletes the next show and in a short space of time
there is a full disk, and all programs which allow auto expire have been
deleted from the DB. I recover the DB from backup and manually delete
files which have been recorded since the backup, but obviously it is
annoying to have to do this.

Possibly relevant is that a) I have a separate master and slave backend
and b) am using NFS for the storage. The logs don't show anything
interesting. Mythtv *thinks* it is doing the right thing. There are
messages about expiring recordings, but nothing to do with failing to
delete a file.

I'd appreciate any clues about what could cause this or how to debug it.


* I've upgraded again but it is too early to tell if the problem is
still there.

--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


michael at thewatsonfamily

Apr 17, 2012, 5:37 PM

Post #2 of 14 (805 views)
Permalink
Re: Expiry problem [In reply to]

On 18/04/2012 9:10 AM, Ian Dall wrote:
> Hi,
>
> I have been running a recent 0.25 pre-release(*) and have run into
> problems with expiry. What happens is that the disk fills up, the expiry
> thread runs, deletes the entry from the database, but does not delete
> the actual files containing the recording. Since the disk is still full,
> it goes ahead and deletes the next show and in a short space of time
> there is a full disk, and all programs which allow auto expire have been
> deleted from the DB. I recover the DB from backup and manually delete
> files which have been recorded since the backup, but obviously it is
> annoying to have to do this.
>
> Possibly relevant is that a) I have a separate master and slave backend
> and b) am using NFS for the storage. The logs don't show anything
> interesting. Mythtv *thinks* it is doing the right thing. There are
> messages about expiring recordings, but nothing to do with failing to
> delete a file.
>
> I'd appreciate any clues about what could cause this or how to debug it.
>
>
> * I've upgraded again but it is too early to tell if the problem is
> still there.
>
More a question for the mythtv-users list.
But sounds like a permission problem.

Michael

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 18, 2012, 1:34 AM

Post #3 of 14 (802 views)
Permalink
Re: Expiry problem [In reply to]

On Wed, 2012-04-18 at 10:37 +1000, Michael Watson wrote:
> On 18/04/2012 9:10 AM, Ian Dall wrote:
> > Hi,
> >
> > I have been running a recent 0.25 pre-release(*) and have run into
> > problems with expiry. What happens is that the disk fills up, the expiry
> > thread runs, deletes the entry from the database, but does not delete
> > the actual files containing the recording. Since the disk is still full,
> > it goes ahead and deletes the next show and in a short space of time
> > there is a full disk, and all programs which allow auto expire have been
> > deleted from the DB. I recover the DB from backup and manually delete
> > files which have been recorded since the backup, but obviously it is
> > annoying to have to do this.
> >
> > Possibly relevant is that a) I have a separate master and slave backend
> > and b) am using NFS for the storage. The logs don't show anything
> > interesting. Mythtv *thinks* it is doing the right thing. There are
> > messages about expiring recordings, but nothing to do with failing to
> > delete a file.
> >
> > I'd appreciate any clues about what could cause this or how to debug it.
> >
> >
> > * I've upgraded again but it is too early to tell if the problem is
> > still there.
> >
> More a question for the mythtv-users list.
> But sounds like a permission problem.

Thanks, but I don't think so. Firstly I have checked, secondly this
started happening after an upgrade with no change in configuration and
thirdly I know of no unix or nfs permissions which allow create but not
delete privilege and clearly files are being created or my disk wouldn't
fill up ;-(

As for users vs dev I was expecting to debug this myself. I'd assume
that if others were affected, it would have been noticed!

--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


michael at thewatsonfamily

Apr 18, 2012, 1:44 AM

Post #4 of 14 (801 views)
Permalink
Re: Expiry problem [In reply to]

On 18/04/2012 6:34 PM, Ian Dall wrote:
> On Wed, 2012-04-18 at 10:37 +1000, Michael Watson wrote:
>> On 18/04/2012 9:10 AM, Ian Dall wrote:
>>> Hi,
>>>
>>> I have been running a recent 0.25 pre-release(*) and have run into
>>> problems with expiry. What happens is that the disk fills up, the expiry
>>> thread runs, deletes the entry from the database, but does not delete
>>> the actual files containing the recording. Since the disk is still full,
>>> it goes ahead and deletes the next show and in a short space of time
>>> there is a full disk, and all programs which allow auto expire have been
>>> deleted from the DB. I recover the DB from backup and manually delete
>>> files which have been recorded since the backup, but obviously it is
>>> annoying to have to do this.
>>>
>>> Possibly relevant is that a) I have a separate master and slave backend
>>> and b) am using NFS for the storage. The logs don't show anything
>>> interesting. Mythtv *thinks* it is doing the right thing. There are
>>> messages about expiring recordings, but nothing to do with failing to
>>> delete a file.
>>>
>>> I'd appreciate any clues about what could cause this or how to debug it.
>>>
>>>
>>> * I've upgraded again but it is too early to tell if the problem is
>>> still there.
>>>
>> More a question for the mythtv-users list.
>> But sounds like a permission problem.
> Thanks, but I don't think so. Firstly I have checked, secondly this
> started happening after an upgrade with no change in configuration and
> thirdly I know of no unix or nfs permissions which allow create but not
> delete privilege and clearly files are being created or my disk wouldn't
> fill up ;-(
That depends on your NFS configuration, and which machine writes the
file, and which machine is trying to delete the file.
The permissions of a file that has not been deleted, are they the say
when viewed from each machine?

>
> As for users vs dev I was expecting to debug this myself. I'd assume
> that if others were affected, it would have been noticed!
>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 18, 2012, 4:06 PM

Post #5 of 14 (797 views)
Permalink
Re: Expiry problem [In reply to]

On Wed, 2012-04-18 at 18:44 +1000, Michael Watson wrote:
> On 18/04/2012 6:34 PM, Ian Dall wrote:
> > On Wed, 2012-04-18 at 10:37 +1000, Michael Watson wrote:
> >> On 18/04/2012 9:10 AM, Ian Dall wrote:
> >>> Hi,
> >>>
> >>> I have been running a recent 0.25 pre-release(*) and have run into
> >>> problems with expiry. What happens is that the disk fills up, the expiry
> >>> thread runs, deletes the entry from the database, but does not delete
> >>> the actual files containing the recording. Since the disk is still full,
> >>> it goes ahead and deletes the next show and in a short space of time
> >>> there is a full disk, and all programs which allow auto expire have been
> >>> deleted from the DB. I recover the DB from backup and manually delete
> >>> files which have been recorded since the backup, but obviously it is
> >>> annoying to have to do this.
> >>>
> >>> Possibly relevant is that a) I have a separate master and slave backend
> >>> and b) am using NFS for the storage. The logs don't show anything
> >>> interesting. Mythtv *thinks* it is doing the right thing. There are
> >>> messages about expiring recordings, but nothing to do with failing to
> >>> delete a file.
> >>>
> >>> I'd appreciate any clues about what could cause this or how to debug it.
> >>>
> >>>
> >>> * I've upgraded again but it is too early to tell if the problem is
> >>> still there.
> >>>
> >> More a question for the mythtv-users list.
> >> But sounds like a permission problem.
> > Thanks, but I don't think so. Firstly I have checked, secondly this
> > started happening after an upgrade with no change in configuration and
> > thirdly I know of no unix or nfs permissions which allow create but not
> > delete privilege and clearly files are being created or my disk wouldn't
> > fill up ;-(
> That depends on your NFS configuration, and which machine writes the
> file, and which machine is trying to delete the file.
> The permissions of a file that has not been deleted, are they the say
> when viewed from each machine?

Good point, but no. On each machine I did
su mythtv
touch /var/media/myth/mythtv/foo
rm /var/media/myth/mythtv/foo
ps -fp $(cat /var/run/mythbackend.pid)

and everything checks out.

As I said, I've upgraded mythtv to v0.26-pre-25-something and turned on
"file" logging. I've also rebooted the virtual machine which was running
mythmaster. It had been up for 59 days and was low on free memory and
swap (I don't recall why). So I'll now wait and see if it happens again.

Thanks for your thoughts.


--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Apr 18, 2012, 6:32 PM

Post #6 of 14 (790 views)
Permalink
Re: Expiry problem [In reply to]

On 04/17/2012 07:10 PM, Ian Dall wrote:
> I have been running a recent 0.25 pre-release(*) and have run into
> problems with expiry. What happens is that the disk fills up, the expiry
> thread runs, deletes the entry from the database, but does not delete
> the actual files containing the recording. Since the disk is still full,
> it goes ahead and deletes the next show and in a short space of time
> there is a full disk, and all programs which allow auto expire have been
> deleted from the DB. I recover the DB from backup and manually delete
> files which have been recorded since the backup, but obviously it is
> annoying to have to do this.
>
> Possibly relevant is that a) I have a separate master and slave backend
> and b) am using NFS for the storage. The logs don't show anything
> interesting. Mythtv *thinks* it is doing the right thing. There are
> messages about expiring recordings, but nothing to do with failing to
> delete a file.
>
> I'd appreciate any clues about what could cause this or how to debug it.
>

Are the expired shows in the Deleted recording group? Watch Recordings,
MENU|Change Group Filter, then select "Deleted" and see if any that the
backend logs say were expired are there.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 22, 2012, 8:14 AM

Post #7 of 14 (754 views)
Permalink
Re: Expiry problem [In reply to]

On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
> On 04/17/2012 07:10 PM, Ian Dall wrote:
> > I have been running a recent 0.25 pre-release(*) and have run into
> > problems with expiry. What happens is that the disk fills up, the expiry
> > thread runs, deletes the entry from the database, but does not delete
> > the actual files containing the recording. Since the disk is still full,
> > it goes ahead and deletes the next show and in a short space of time
> > there is a full disk, and all programs which allow auto expire have been
> > deleted from the DB. I recover the DB from backup and manually delete
> > files which have been recorded since the backup, but obviously it is
> > annoying to have to do this.
> >
> > Possibly relevant is that a) I have a separate master and slave backend
> > and b) am using NFS for the storage. The logs don't show anything
> > interesting. Mythtv *thinks* it is doing the right thing. There are
> > messages about expiring recordings, but nothing to do with failing to
> > delete a file.
> >
> > I'd appreciate any clues about what could cause this or how to debug it.
> >
>
> Are the expired shows in the Deleted recording group? Watch Recordings,
> MENU|Change Group Filter, then select "Deleted" and see if any that the
> backend logs say were expired are there.

Ah! It took a while, but I think I have caught the misbehaviour in the
act. I had a recording from an iptv "tuner". This should have been about
1 hour long, but the file was still open (as reported by lsof) and
growing 2 hours later, even though the show showed as finished in "Watch
Recordings". The entire file was watchable.

So, I deleted the recording through "Watch Recordings",Menu,Delete. It
then disappeared from the "Watch Recordings" list, but the file was
still on disk and still growing! Checking the "Deleted" group showed it
there. Restarting the master backend (which also has the iptv tuner on
it) at least stopped the file growing, but it was still there in the
Deleted group.

It did eventually get deleted, presumably when the delete thread was
run.

So it all makes sense except for the recording which never terminated
problem. If this happens it would explain my original symptoms. It would
also explain another strangeness. After my recovering from the db from
restore and cleaning up, I'd expect only about 10% free space (only one
day of shows), but was getting more like 50% free space. Presumably
there was one or a few bloated files taking up most of the space.

--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 22, 2012, 8:26 AM

Post #8 of 14 (762 views)
Permalink
Re: Expiry problem [In reply to]

On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
> > On 04/17/2012 07:10 PM, Ian Dall wrote:
> > > I have been running a recent 0.25 pre-release(*) and have run into
> > > problems with expiry. What happens is that the disk fills up, the expiry
> > > thread runs, deletes the entry from the database, but does not delete
> > > the actual files containing the recording. Since the disk is still full,
> > > it goes ahead and deletes the next show and in a short space of time
> > > there is a full disk, and all programs which allow auto expire have been
> > > deleted from the DB. I recover the DB from backup and manually delete
> > > files which have been recorded since the backup, but obviously it is
> > > annoying to have to do this.
> > >
> > > Possibly relevant is that a) I have a separate master and slave backend
> > > and b) am using NFS for the storage. The logs don't show anything
> > > interesting. Mythtv *thinks* it is doing the right thing. There are
> > > messages about expiring recordings, but nothing to do with failing to
> > > delete a file.
> > >
> > > I'd appreciate any clues about what could cause this or how to debug it.
> > >
> >
> > Are the expired shows in the Deleted recording group? Watch Recordings,
> > MENU|Change Group Filter, then select "Deleted" and see if any that the
> > backend logs say were expired are there.
>
> Ah! It took a while, but I think I have caught the misbehaviour in the
> act. I had a recording from an iptv "tuner". This should have been about
> 1 hour long, but the file was still open (as reported by lsof) and
> growing 2 hours later, even though the show showed as finished in "Watch
> Recordings". The entire file was watchable.

And I should add, in the system status information the ipv "tuner" was
shown as NOT recording...


--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Apr 22, 2012, 3:37 PM

Post #9 of 14 (750 views)
Permalink
Re: Expiry problem [In reply to]

On 04/22/2012 11:26 AM, Ian Dall wrote:
> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
>>> On 04/17/2012 07:10 PM, Ian Dall wrote:
>>>> I have been running a recent 0.25 pre-release(*) and have run into
>>>> problems with expiry. What happens is that the disk fills up, the expiry
>>>> thread runs, deletes the entry from the database, but does not delete
>>>> the actual files containing the recording. Since the disk is still full,
>>>> it goes ahead and deletes the next show and in a short space of time
>>>> there is a full disk, and all programs which allow auto expire have been
>>>> deleted from the DB. I recover the DB from backup and manually delete
>>>> files which have been recorded since the backup, but obviously it is
>>>> annoying to have to do this.
>>>>
>>>> Possibly relevant is that a) I have a separate master and slave backend
>>>> and b) am using NFS for the storage. The logs don't show anything
>>>> interesting. Mythtv *thinks* it is doing the right thing. There are
>>>> messages about expiring recordings, but nothing to do with failing to
>>>> delete a file.
>>>>
>>>> I'd appreciate any clues about what could cause this or how to debug it.
>>>>
>>> Are the expired shows in the Deleted recording group? Watch Recordings,
>>> MENU|Change Group Filter, then select "Deleted" and see if any that the
>>> backend logs say were expired are there.
>> Ah! It took a while, but I think I have caught the misbehaviour in the
>> act. I had a recording from an iptv "tuner". This should have been about
>> 1 hour long, but the file was still open (as reported by lsof) and
>> growing 2 hours later, even though the show showed as finished in "Watch
>> Recordings". The entire file was watchable.
> And I should add, in the system status information the ipv "tuner" was
> shown as NOT recording...

http://code.mythtv.org/trac/ticket/9670 (specifically
http://code.mythtv.org/trac/ticket/9670#comment:31 )
and
http://code.mythtv.org/trac/ticket/10620

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 23, 2012, 3:55 PM

Post #10 of 14 (740 views)
Permalink
Re: Expiry problem [In reply to]

On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote:
> On 04/22/2012 11:26 AM, Ian Dall wrote:
> > On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
> >> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
> >>> On 04/17/2012 07:10 PM, Ian Dall wrote:
> >>>> I have been running a recent 0.25 pre-release(*) and have run into
> >>>> problems with expiry. What happens is that the disk fills up, the expiry
> >>>> thread runs, deletes the entry from the database, but does not delete
> >>>> the actual files containing the recording. Since the disk is still full,
> >>>> it goes ahead and deletes the next show and in a short space of time
> >>>> there is a full disk, and all programs which allow auto expire have been
> >>>> deleted from the DB. I recover the DB from backup and manually delete
> >>>> files which have been recorded since the backup, but obviously it is
> >>>> annoying to have to do this.
> >>>>
> >>>> Possibly relevant is that a) I have a separate master and slave backend
> >>>> and b) am using NFS for the storage. The logs don't show anything
> >>>> interesting. Mythtv *thinks* it is doing the right thing. There are
> >>>> messages about expiring recordings, but nothing to do with failing to
> >>>> delete a file.
> >>>>
> >>>> I'd appreciate any clues about what could cause this or how to debug it.
> >>>>
> >>> Are the expired shows in the Deleted recording group? Watch Recordings,
> >>> MENU|Change Group Filter, then select "Deleted" and see if any that the
> >>> backend logs say were expired are there.
> >> [iptv problem]

Unfortunately it seems that the iptv problem is not the cause of the
fail to delete option.

So, the disk filled up again, and yes, it ended up with about 90
recordings in the "Deleted" recording group. Restarting the master
backend causes all the files to be deleted. I notice in the logs that
there are successful regular deletions, (signalled by "About to delete
file" messages) up until about 2:30pm then nothing until after the
backend has been restarted at 11pm (after the disk space is exhausted),
when all the files in the "Deleted" recording group get deleted.

I notice that the delete thread does check that the file has gone (and
would produce an error message if it isn't) but I don't see any of these
error messages.

It looks like the DeleteThread is simply not running. A check with gdb
doesn't seem to show any deadlocked threads, but it is not always easy
to easy to identify which thread is which.

One other point. I notice error messages to do with the "recordedfile"
table. The recordedfile table exists but is empty. A search of the
sources shows SELECT and DELETE actions on this table, but no INSERT. I
assume that this is a legacy table and these messages can be ignored?


Regards,
Ian

--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Apr 23, 2012, 6:46 PM

Post #11 of 14 (743 views)
Permalink
Re: Expiry problem [In reply to]

On 04/22/2012 06:37 PM, Michael T. Dean wrote:
> On 04/22/2012 11:26 AM, Ian Dall wrote:
>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
>>>> On 04/17/2012 07:10 PM, Ian Dall wrote:
>>>>> I have been running a recent 0.25 pre-release(*) and have run into
>>>>> problems with expiry. What happens is that the disk fills up, the
>>>>> expiry
>>>>> thread runs, deletes the entry from the database, but does not delete
>>>>> the actual files containing the recording. Since the disk is still
>>>>> full,
>>>>> it goes ahead and deletes the next show and in a short space of time
>>>>> there is a full disk, and all programs which allow auto expire
>>>>> have been
>>>>> deleted from the DB. I recover the DB from backup and manually delete
>>>>> files which have been recorded since the backup, but obviously it is
>>>>> annoying to have to do this.
>>>>>
>>>>> Possibly relevant is that a) I have a separate master and slave
>>>>> backend
>>>>> and b) am using NFS for the storage. The logs don't show anything
>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are
>>>>> messages about expiring recordings, but nothing to do with failing to
>>>>> delete a file.
>>>>>
>>>>> I'd appreciate any clues about what could cause this or how to
>>>>> debug it.
>>>>>
>>>> Are the expired shows in the Deleted recording group? Watch
>>>> Recordings,
>>>> MENU|Change Group Filter, then select "Deleted" and see if any that
>>>> the
>>>> backend logs say were expired are there.
>>> Ah! It took a while, but I think I have caught the misbehaviour in the
>>> act. I had a recording from an iptv "tuner". This should have been
>>> about
>>> 1 hour long, but the file was still open (as reported by lsof) and
>>> growing 2 hours later, even though the show showed as finished in
>>> "Watch
>>> Recordings". The entire file was watchable.
>> And I should add, in the system status information the ipv "tuner" was
>> shown as NOT recording...
>
> http://code.mythtv.org/trac/ticket/9670 (specifically
> http://code.mythtv.org/trac/ticket/9670#comment:31 )
> and
> http://code.mythtv.org/trac/ticket/10620

And, http://code.mythtv.org/trac/ticket/10493 , which is the one I was
really looking for.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Apr 23, 2012, 6:50 PM

Post #12 of 14 (738 views)
Permalink
Re: Expiry problem [In reply to]

On 04/23/2012 06:55 PM, Ian Dall wrote:
> On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote:
>> On 04/22/2012 11:26 AM, Ian Dall wrote:
>>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
>>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
>>>>> On 04/17/2012 07:10 PM, Ian Dall wrote:
>>>>>> I have been running a recent 0.25 pre-release(*) and have run into
>>>>>> problems with expiry. What happens is that the disk fills up, the expiry
>>>>>> thread runs, deletes the entry from the database, but does not delete
>>>>>> the actual files containing the recording. Since the disk is still full,
>>>>>> it goes ahead and deletes the next show and in a short space of time
>>>>>> there is a full disk, and all programs which allow auto expire have been
>>>>>> deleted from the DB. I recover the DB from backup and manually delete
>>>>>> files which have been recorded since the backup, but obviously it is
>>>>>> annoying to have to do this.
>>>>>>
>>>>>> Possibly relevant is that a) I have a separate master and slave backend
>>>>>> and b) am using NFS for the storage. The logs don't show anything
>>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are
>>>>>> messages about expiring recordings, but nothing to do with failing to
>>>>>> delete a file.
>>>>>>
>>>>>> I'd appreciate any clues about what could cause this or how to debug it.
>>>>>>
>>>>> Are the expired shows in the Deleted recording group? Watch Recordings,
>>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the
>>>>> backend logs say were expired are there.
>>>> [iptv problem]
> Unfortunately it seems that the iptv problem is not the cause of the
> fail to delete option.
>
> So, the disk filled up again, and yes, it ended up with about 90
> recordings in the "Deleted" recording group. Restarting the master
> backend causes all the files to be deleted. I notice in the logs that
> there are successful regular deletions, (signalled by "About to delete
> file" messages) up until about 2:30pm then nothing until after the
> backend has been restarted at 11pm (after the disk space is exhausted),
> when all the files in the "Deleted" recording group get deleted.
>
> I notice that the delete thread does check that the file has gone (and
> would produce an error message if it isn't) but I don't see any of these
> error messages.
>
> It looks like the DeleteThread is simply not running. A check with gdb
> doesn't seem to show any deadlocked threads, but it is not always easy
> to easy to identify which thread is which.

It sounds to me like MythTV starts an iptv recording and "stops" it, but
the recording keeps going without MythTV's "conscious" knowledge. These
recordings fill up the file system, and MythTV begins to expire
recordings, but because they're still open/in use, they don't actually
get removed and, perhaps, cause the delete thread to hang. Restarting
mythbackend closes all open files and means that next time you restart
it will resume deleting the files it had in the queue, which are no
longer open, and will get removed.

> One other point. I notice error messages to do with the "recordedfile"
> table. The recordedfile table exists but is empty. A search of the
> sources shows SELECT and DELETE actions on this table, but no INSERT. I
> assume that this is a legacy table and these messages can be ignored?
>

It's nothing--and about to be replaced. We changed the plan for it, and
it has never actually been in use for anything.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 23, 2012, 9:05 PM

Post #13 of 14 (736 views)
Permalink
Re: Expiry problem [In reply to]

On Mon, 2012-04-23 at 21:50 -0400, Michael T. Dean wrote:
> On 04/23/2012 06:55 PM, Ian Dall wrote:
> > On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote:
> >> On 04/22/2012 11:26 AM, Ian Dall wrote:
> >>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
> >>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
> >>>>> On 04/17/2012 07:10 PM, Ian Dall wrote:
> >>>>>> I have been running a recent 0.25 pre-release(*) and have run into
> >>>>>> problems with expiry. What happens is that the disk fills up, the expiry
> >>>>>> thread runs, deletes the entry from the database, but does not delete
> >>>>>> the actual files containing the recording. Since the disk is still full,
> >>>>>> it goes ahead and deletes the next show and in a short space of time
> >>>>>> there is a full disk, and all programs which allow auto expire have been
> >>>>>> deleted from the DB. I recover the DB from backup and manually delete
> >>>>>> files which have been recorded since the backup, but obviously it is
> >>>>>> annoying to have to do this.
> >>>>>>
> >>>>>> Possibly relevant is that a) I have a separate master and slave backend
> >>>>>> and b) am using NFS for the storage. The logs don't show anything
> >>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are
> >>>>>> messages about expiring recordings, but nothing to do with failing to
> >>>>>> delete a file.
> >>>>>>
> >>>>>> I'd appreciate any clues about what could cause this or how to debug it.
> >>>>>>
> >>>>> Are the expired shows in the Deleted recording group? Watch Recordings,
> >>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the
> >>>>> backend logs say were expired are there.
> >>>> [iptv problem]
> > Unfortunately it seems that the iptv problem is not the cause of the
> > fail to delete option.
> >
> > So, the disk filled up again, and yes, it ended up with about 90
> > recordings in the "Deleted" recording group. Restarting the master
> > backend causes all the files to be deleted. I notice in the logs that
> > there are successful regular deletions, (signalled by "About to delete
> > file" messages) up until about 2:30pm then nothing until after the
> > backend has been restarted at 11pm (after the disk space is exhausted),
> > when all the files in the "Deleted" recording group get deleted.
> >
> > I notice that the delete thread does check that the file has gone (and
> > would produce an error message if it isn't) but I don't see any of these
> > error messages.
> >
> > It looks like the DeleteThread is simply not running. A check with gdb
> > doesn't seem to show any deadlocked threads, but it is not always easy
> > to easy to identify which thread is which.
>
> It sounds to me like MythTV starts an iptv recording and "stops" it, but
> the recording keeps going without MythTV's "conscious" knowledge. These
> recordings fill up the file system, and MythTV begins to expire
> recordings, but because they're still open/in use, they don't actually
> get removed and, perhaps, cause the delete thread to hang.

That's what I was thinking too, but I deleted the iptv "tuner" before
this latest round of trouble, which is why I concluded this is a
separate problem.

--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


ian at beware

Apr 27, 2012, 8:30 PM

Post #14 of 14 (703 views)
Permalink
Re: Expiry problem [In reply to]

On Mon, 2012-04-23 at 21:50 -0400, Michael T. Dean wrote:
> On 04/23/2012 06:55 PM, Ian Dall wrote:
> > On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote:
> >> On 04/22/2012 11:26 AM, Ian Dall wrote:
> >>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
> >>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
> >>>>> On 04/17/2012 07:10 PM, Ian Dall wrote:
> >>>>>> I have been running a recent 0.25 pre-release(*) and have run into
> >>>>>> problems with expiry. What happens is that the disk fills up, the expiry
> >>>>>> thread runs, deletes the entry from the database, but does not delete
> >>>>>> the actual files containing the recording. Since the disk is still full,
> >>>>>> it goes ahead and deletes the next show and in a short space of time
> >>>>>> there is a full disk, and all programs which allow auto expire have been
> >>>>>> deleted from the DB. I recover the DB from backup and manually delete
> >>>>>> files which have been recorded since the backup, but obviously it is
> >>>>>> annoying to have to do this.
> >>>>>>
> >>>>>> Possibly relevant is that a) I have a separate master and slave backend
> >>>>>> and b) am using NFS for the storage. The logs don't show anything
> >>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are
> >>>>>> messages about expiring recordings, but nothing to do with failing to
> >>>>>> delete a file.
> >>>>>>
> >>>>>> I'd appreciate any clues about what could cause this or how to debug it.
> >>>>>>
> >>>>> Are the expired shows in the Deleted recording group? Watch Recordings,
> >>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the
> >>>>> backend logs say were expired are there.
> >>>> [iptv problem]
> > Unfortunately it seems that the iptv problem is not the cause of the
> > fail to delete option.
> >
> > So, the disk filled up again, and yes, it ended up with about 90
> > recordings in the "Deleted" recording group. Restarting the master
> > backend causes all the files to be deleted. I notice in the logs that
> > there are successful regular deletions, (signalled by "About to delete
> > file" messages) up until about 2:30pm then nothing until after the
> > backend has been restarted at 11pm (after the disk space is exhausted),
> > when all the files in the "Deleted" recording group get deleted.
> >
> > I notice that the delete thread does check that the file has gone (and
> > would produce an error message if it isn't) but I don't see any of these
> > error messages.
> >
> > It looks like the DeleteThread is simply not running. A check with gdb
> > doesn't seem to show any deadlocked threads, but it is not always easy
> > to easy to identify which thread is which.
>
> It sounds to me like MythTV starts an iptv recording and "stops" it, but
> the recording keeps going without MythTV's "conscious" knowledge. These
> recordings fill up the file system, and MythTV begins to expire
> recordings, but because they're still open/in use, they don't actually
> get removed and, perhaps, cause the delete thread to hang. Restarting
> mythbackend closes all open files and means that next time you restart
> it will resume deleting the files it had in the queue, which are no
> longer open, and will get removed.

As noted, I have ruled out iptv by deleting the tuner.

I have also been capturing the network traffic between master and slave.
I have also set gdb breakpoints in the slave, and /still/ I don't
understand what is going on ;-(

What I can say for sure is that the slave gets a DELETE_RECORDING
command. The recording is, at this stage in the "Default" recording
group so MainServer::DoHandleDeleteRecording (mainserver.cpp:2468)
figures that "justexpire" is true, changes the recording group to
"Deleted" and returns. The code to start the DeleteThread is after the
return and is never executed (on the slave at least).

I'm having trouble figuring out which threads are supposed to persist,
which get started on demand and which get started from timers.
Particularly in the case of the DeleteThread, how is it supposed to be
started? I also am having trouble figuring out why delete involves the
slave at all. The recording is available to the master so why doesn't
the master just delete it?

Regards,
Ian


--
Ian Dall <ian [at] beware>

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev

MythTV dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.