Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Dev

Storage Groups functionality

 

 

First page Previous page 1 2 Next page Last page  View All MythTV dev RSS feed   Index | Next | Previous | View Threaded


cpinkham at bc2va

Feb 5, 2006, 10:16 AM

Post #1 of 28 (8158 views)
Permalink
Storage Groups functionality

I wanted to start a thread to talk about the Storage Groups functionality
that has been mentioned for post-0.19 work.

This is mainly to Kevin, but I wanted to post to the list to see if anyone
elase had any other ideas.

Over time, there has been 2 different ways mentioned to spread out
recordings over multiple directories. The first is to just have multiple
recording directories listed where files are either round-robin balanced
between the three or where new recordings just go to the filesystem with
the most free space. The second is the concept of storage groups where
the user can specify multiple storage groups each one with it's
own directory. Then each scheduled recording can be assigned to one
of the storage groups or the default.

I'd like to see a merger of both those ideas. I know there are lots of
people who would use the functionality to do things like separate their
'A-Team' recordings onto a Raid-10 filesystem while keeping their
wife's 'Desparate Housewifes' on a non-redundant filesystem, but there
are also a lot of us who just want to load-balance storage between
multiple filesystems. I like the idea of Storage Groups because I
can separate off my A-Team recordings, but for the other 95% of
recordings, I may just want to spread them out over my three video
filesystems so the filesystems are always pretty much equal in
the amount of free space they have. The Storage Group concept
doesn't have to even be multiple filesystems, it could be just different
directories on the same filesystem as others have mentioned they would
like to be able to do.

As far as implementation goes, I think it would be a lot easier to
just handle the Storage Group at the record table level and not at
the recorded table level. I think we should go one step farther
than we are now with the 'basename' column and add a 'dirname' (or
whatever it is called) column. This would hold the directory of
the file (obviously). This way we don't have to deal with storage
group changes, etc. like when a user decides to change the directory
on one of their storage groups. I think the ability to move
recordings from one Storage Group to another is a nice-to-have, but
not a requirement for the initial implementation. The
directory-determining logic could look at the record.storagegroup field
and if it was a normal group name, it would use the directory specified
for that group, but if it was one of the special virtual groups such
as 'Most Free Disk Space' or possibly 'Least Recordings' or others,
then it would perform a few calculations to figure out which directory
out of all the real Storage Groups should be used. Perhaps the default
would even be to just use the storage group with the most amount of
disk space free, or there could be a special 'Default' storage group
where the group name was not user-editable, only the directory.

Of course we'll have to handle this in lots of places in the code, but
those can be talked about later, I just wanted to see if I could start
some kind of discussion on the topic.

--
Chris

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Feb 5, 2006, 10:36 AM

Post #2 of 28 (8034 views)
Permalink
Re: Storage Groups functionality [In reply to]

On 02/05/2006 01:16 PM, Chris Pinkham wrote:
> As far as implementation goes, I think it would be a lot easier to
> just handle the Storage Group at the record table level and not at
> the recorded table level. I think we should go one step farther
> than we are now with the 'basename' column and add a 'dirname' (or
> whatever it is called) column. This would hold the directory of
> the file (obviously). This way we don't have to deal with storage
> group changes, etc. like when a user decides to change the directory
> on one of their storage groups. I think the ability to move
> recordings from one Storage Group to another is a nice-to-have, but
> not a requirement for the initial implementation. The
> directory-determining logic could look at the record.storagegroup field
> and if it was a normal group name, it would use the directory specified
> for that group, but if it was one of the special virtual groups such
> as 'Most Free Disk Space' or possibly 'Least Recordings' or others,
> then it would perform a few calculations to figure out which directory
> out of all the real Storage Groups should be used. Perhaps the default
> would even be to just use the storage group with the most amount of
> disk space free, or there could be a special 'Default' storage group
> where the group name was not user-editable, only the directory.
>

Good ideas.

Regarding the Most Free Disk Space/Least Recordings, we should also
consider the effect multiple concurrent recordings will have. Say 2 or
3 recordings start simultaneously, the same directory will have the most
free space/least recordings at the start, but may not after the
recordings finish. This may result in recordings being auto-expired
from this directory (if near the end of available storage) when choosing
different directories may have allowed all three to finish successfully
without auto-expiring other recordings. Also, choosing the same
filesystem for simultaneous starts may cause an I/O bottleneck.

The round robin is nice for spreading I/O across multiple spindles, but
may actually cause problems with multiple network filesystems (i.e. if 3
NFS mounts are chosen for 3 concurrent HDTV recordings while the system
is doing near-real-time commflagging or... instead of choosing some
local and some remote filesystems).

In the initial implementation, at least, we could ignore these issues as
long as we give the user the ability to explicitly choose a storage
group at the record table level, as you mentioned.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Feb 5, 2006, 11:15 AM

Post #3 of 28 (8023 views)
Permalink
Re: Storage Groups functionality [In reply to]

On 02/05/2006 01:36 PM, Michael T. Dean wrote:
> The round robin is nice for spreading I/O across multiple spindles, but
> may actually cause problems with multiple network filesystems (i.e. if 3
> NFS mounts are chosen for 3 concurrent HDTV recordings while the system
> is doing near-real-time commflagging or... instead of choosing some
> local and some remote filesystems).
>
> In the initial implementation, at least, we could ignore these issues as
> long as we give the user the ability to explicitly choose a storage
> group at the record table level, as you mentioned.
>

Although, we should probably allow the user to specify whether a
filesystem is remote to the current backend, though.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


jra at baylink

Feb 5, 2006, 12:05 PM

Post #4 of 28 (8008 views)
Permalink
Re: Storage Groups functionality [In reply to]

On Sun, Feb 05, 2006 at 02:15:50PM -0500, Michael T. Dean wrote:
> On 02/05/2006 01:36 PM, Michael T. Dean wrote:
> > The round robin is nice for spreading I/O across multiple spindles, but
> > may actually cause problems with multiple network filesystems (i.e. if 3
> > NFS mounts are chosen for 3 concurrent HDTV recordings while the system
> > is doing near-real-time commflagging or... instead of choosing some
> > local and some remote filesystems).
> >
> > In the initial implementation, at least, we could ignore these issues as
> > long as we give the user the ability to explicitly choose a storage
> > group at the record table level, as you mentioned.
>
> Although, we should probably allow the user to specify whether a
> filesystem is remote to the current backend, though.

I'd also like to see it deal well with off-line storage, both for the
"removable FireWire drive" case, and for future expansion into, say,
DVD-jukebox storage of archival programs which are still in internal
format (keeping them in internal format has many advantages to burning
the out to DVD-Video).

Cheers,
-- jra
--
Jay R. Ashworth jra [at] baylink
Designer Baylink RFC 2100
Ashworth & Associates The Things I Think '87 e24
St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274

A: No.
Q: Should I include quotations after my message body?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 5, 2006, 8:19 PM

Post #5 of 28 (8014 views)
Permalink
Re: Storage Groups functionality [In reply to]

> Regarding the Most Free Disk Space/Least Recordings, we should also
> consider the effect multiple concurrent recordings will have. Say 2 or

Well, anything we could do to spread recordings out would be a benefit,
but even if we don't do anything to handle situations like this, there is
no negative effect since right now there is only one recordings directory.
I agree it would be nice if we could handle this, but doing this correctly
is going to involve coordination between all backends since a lot of
people (including myself) use shared NFS recording directories.

> 3 recordings start simultaneously, the same directory will have the most
> free space/least recordings at the start, but may not after the
> recordings finish. This may result in recordings being auto-expired

If we wanted to handle this, we could perform a similar check to the one
that the status page in mythfrontend does. We can look at the average
bitrate and use that combined with knowledge of what is currently recording
in order to determine where to put the next recording. Again, this will
require coordination between all backends though since we may have shared
NFS storage between one or more backends.

> The round robin is nice for spreading I/O across multiple spindles, but
> may actually cause problems with multiple network filesystems (i.e. if 3
> NFS mounts are chosen for 3 concurrent HDTV recordings while the system
> is doing near-real-time commflagging or... instead of choosing some
> local and some remote filesystems).

So far, I hadn't even considered the fact of multiple backends. I was
assuming the Storage Groups would be global settings, but this may not
work with multiple backends. If a user does not use NFS but instead has
multiple local recording directories on each backend, they would want
host-specific Storage Group values. The load-balancing logic could be
coded to prefer local storage over NAS (yes, I mean Network Attached
Storage, just in case someone uses Samba, etc..) I personally do not have
any Myth storage in my backends, I use a dedicated NFS server for my Myth
and other storage needs at home. Myth has 3 dedicated filesystems on
the NFS server currently.

> In the initial implementation, at least, we could ignore these issues as
> long as we give the user the ability to explicitly choose a storage
> group at the record table level, as you mentioned.

I agree.

--
Chris

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 5, 2006, 8:33 PM

Post #6 of 28 (8014 views)
Permalink
Re: Storage Groups functionality [In reply to]

> I'd also like to see it deal well with off-line storage, both for the
> "removable FireWire drive" case, and for future expansion into, say,
> DVD-jukebox storage of archival programs which are still in internal
> format (keeping them in internal format has many advantages to burning
> the out to DVD-Video).

I would like to see the ability for some form of off-line storage as well.
I don't know how much the Storage Group concept would have with
archiving recordings though. If we store the directory name in the
recorded table, then you can archive some recordings off to dvd and change
their recorded.dirname to /mnt/dvd and you just have to pop in the dvd,
mount it, and proceed to play. This could even work with frontends since
the frontend will play a file if it detects the file local rather than
trying to stream the recording from the backend. As part of an archiving
solution though, I would like to see something that highlights what
recordings have been archived on the watch recordings screen. Maybe this
is just an 'Archived' recording group or something, but it would be nice
to distinguish the difference from normal recordings. We don't want to
check for existance of a file to determine whether it has been archived
or not because this can get rather slow for large Myth setups. We used
to query the filesize of each recording when querying the recordings list
from the backend and this was getting quite slow for some people, that is
the reason we started storing the filesize in the recorded table. There
could even be a descriptor somewhere to allow you to indicate where the
recording was archived to (ie, DVD #132) so it is easy to find and insert.
This could be displayed by the theme. But I'm starting to drift off-topic
for this thread. Do you have any specific ideas on how archiving would
integrate with Storage Groups specifically?

--
Chris

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


f-myth-users at media

Feb 6, 2006, 1:38 AM

Post #7 of 28 (8017 views)
Permalink
Storage Groups functionality [In reply to]

> Date: Sun, 5 Feb 2006 23:33:24 -0500 (EST)
> From: "Chris Pinkham" <cpinkham [at] bc2va>

> Do you have any specific ideas on how archiving would
> integrate with Storage Groups specifically?

I do, since this is something I'm looking at right this second; maybe
my solution will give people some ideas. I'm (going to be) archiving
large amounts of data off Myth for later use in what amounts to a tape
robot implemented with DVDs. The data's just being written as data
files, not mastered in some way that a standalone DVD player could
play, since we're not really "watching" the data in a conventional way.

One unsolved problem: It'd be really, really nice to be able to take
an archived program (which has presumably been deleted from "recorded"
because it's been deleted from the UI) and reinsert all the relevant
table data again once the video bits have been restored to disk. Is
it sufficient to simply save the relevant row from the "recorded"
table and reinsert it? (I'm assuming that mythcommflag --rebuild will
also have to be rerun on the video to make seeking work again.) Is
there anywhere else (e.g., other tables) that the info would need to
be reinserted? I'm poking around trying to establish that right now,
but if somebody knows, it could save me some time. [.The idea is to be
able to run the commflagger & refind all the commercials, and
otherwise treat it as if it had just been recorded, rather than
relegating the file to second-class status in the "external videos"
part of Myth.]

Anyway:

My strategy is to add a column to the oldrecorded table, which stores
a sequence of DVD numbers corresponding to which DVD(s) each recording
got written to. (Generally, this is only one, but large recordings
might get split across DVDs, and in general it might make sense to
split across DVDs so as not to waste some amount of space at the end
of each one.) Since this isn't part of Myth's actual code, I'm
calling the column "_archived" in the assumption that no actual column
name will ever begin with an underscore. The column's field type is
VARCHAR(128) NOT NULL and consists of a set of small integers,
corresponding to unique DVD numbers. (Right now, that sequence of
numbers is generated externally from Myth, and the data is inserted
into the table by the script that burns the DVDs; if at some point I
might need to backtranslate from DVD number to what's supposed to be
on it, I'll make another table indexed by DVD number instead [.but then
I'd have to have some unambiguous way of referring to individual
programs, hence my earlier question of whether Myth had such a way;
I'll probably wind up using channum + starttime to do so]. Such a
table would use autoincrement so that the mysql DB is responsible for
ensuring that we don't duplicate a DVD sequence number, but for the
moment I can just treat the contents of _archived as a set of ints
that will be stored and parsed by a very simple external script. [.If
there was some plausible way to tell mysql "set of ints" I'd do that
instead of storing a bunch of ints as a single character string in the
varchar, but this'll work for now.])

People who actually -author- DVDs and try to make sure content
actually fits on a single DVD might use such a column with a real
int (probably smallint) datatype instead, but OTOH such people might
be calling their DVDs by titles ("My Vacation") instead, so a varchar
might work for them, too.

(It'd also be nice if "oldrecorded" had the same "syndicatedepisodenumber"
column that "recordedprogram" did! Right now I'm having to kluge that sort
of thing after the fact...)

P.S. A random detail:

I've also added a second column called "_par". Every 5 DVDs we burn
has an accompanying PAR2 archive consisting of enough recovery blocks
to completely recover any one DVD's contents. This gives us some
resiliency against a scratch taking out a file or some worse insult
taking out an entire DVD (including, of course, simply -losing- the
entire DVD... :) Note that calculating a PAR2 archive across the
~23gig of data involved is fairly slow; the sweet spot on filesystem
buffering vs PAR2 memory-handling turns out to be "par2 -m64" on this
hardware (AMD 2800+) and generating the data takes about six hours.
This inflates the number of burned DVD's by 20%, but eliminates a
large number of possible problems with archiving this data for perhaps
years, and the cost of using DVDs this way is around 10-20% of what
using spinning hard disks would be for this enormous amount of data.

[.Soon there will be another column called "_cc" (type text) which will
hold the closed-captioning data for the program, if any, to enable
searching it for programs mentioning certain terms. Obviously, this
entire scheme depends on -nothing- in Myth itself ever deleting
something from "oldrecorded"; if such a thing can happen, I'll have
to make an entirely separate table so Myth keeps its mitts off it.]
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


chris at chrisdos

Feb 6, 2006, 5:31 AM

Post #8 of 28 (8010 views)
Permalink
Re: Storage Groups functionality [In reply to]

Chris Pinkham wrote:
> This is mainly to Kevin, but I wanted to post to the list to see if anyone
> elase had any other ideas.


There have been a lot of good ideas put forth for this. I'd like to add my
thoughts.

If there are multiple storage units, MythTV should run a little one time
performance test for each storage volume when the drives are idle. Maybe
bonnie++ or some such utility. Preference could be given to the fastest storage
volumes for HD recordings.

Maybe instead of straight round robin, a recording will go to the storage
volumes that has the most available free space.

It would also be nice to tell it to balance the recordings amoung the volumes
when MythTV is idle. For maintenence purposes it would be good to tell Myth to
move all the recordings from one volume to the rest in case you need to take
that volume off line.

Chris
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mythtv at dsl

Feb 6, 2006, 12:09 PM

Post #9 of 28 (8005 views)
Permalink
Re: Storage Groups functionality [In reply to]

----- Original Message -----
From: <f-myth-users [at] media>
To: <mythtv-dev [at] mythtv>
Cc: <f-myth-users [at] media>
Sent: Monday, February 06, 2006 9:38 AM
Subject: [mythtv] Storage Groups functionality


>I do, since this is something I'm looking at right this second; maybe
>my solution will give people some ideas. I'm (going to be) archiving
>large amounts of data off Myth for later use in what amounts to a tape
>robot implemented with DVDs. The data's just being written as data
>files, not mastered in some way that a standalone DVD player could
>play, since we're not really "watching" the data in a conventional way.

>One unsolved problem: It'd be really, really nice to be able to take
>an archived program (which has presumably been deleted from "recorded"
>because it's been deleted from the UI) and reinsert all the relevant
>table data again once the video bits have been restored to disk. Is
>it sufficient to simply save the relevant row from the "recorded"
>table and reinsert it? (I'm assuming that mythcommflag --rebuild will
>also have to be rerun on the video to make seeking work again.) Is
>there anywhere else (e.g., other tables) that the info would need to
>be reinserted? I'm poking around trying to establish that right now,
>but if somebody knows, it could save me some time. [.The idea is to be
>able to run the commflagger & refind all the commercials, and
>otherwise treat it as if it had just been recorded, rather than
>relegating the file to second-class status in the "external videos"
>part of Myth.]

>Anyway:

>My strategy is to add a column to the oldrecorded table, which stores
>a sequence of DVD numbers corresponding to which DVD(s) each recording
>got written to. (Generally, this is only one, but large recordings
>might get split across DVDs, and in general it might make sense to
>split across DVDs so as not to waste some amount of space at the end
>of each one.) Since this isn't part of Myth's actual code, I'm
>calling the column "_archived" in the assumption that no actual column
>name will ever begin with an underscore. The column's field type is
>VARCHAR(128) NOT NULL and consists of a set of small integers,
>corresponding to unique DVD numbers. (Right now, that sequence of
>numbers is generated externally from Myth, and the data is inserted
>into the table by the script that burns the DVDs; if at some point I
>might need to backtranslate from DVD number to what's supposed to be
>on it, I'll make another table indexed by DVD number instead [.but then
>I'd have to have some unambiguous way of referring to individual
>programs, hence my earlier question of whether Myth had such a way;
>I'll probably wind up using channum + starttime to do so]. Such a
>table would use autoincrement so that the mysql DB is responsible for
>ensuring that we don't duplicate a DVD sequence number, but for the
>moment I can just treat the contents of _archived as a set of ints
>that will be stored and parsed by a very simple external script. [.If
>there was some plausible way to tell mysql "set of ints" I'd do that
>instead of storing a bunch of ints as a single character string in the
>varchar, but this'll work for now.])

>People who actually -author- DVDs and try to make sure content
>actually fits on a single DVD might use such a column with a real
>int (probably smallint) datatype instead, but OTOH such people might
>be calling their DVDs by titles ("My Vacation") instead, so a varchar
>might work for them, too.

>(It'd also be nice if "oldrecorded" had the same "syndicatedepisodenumber"
>column that "recordedprogram" did! Right now I'm having to kluge that sort
>of thing after the fact...)

>P.S. A random detail:

>I've also added a second column called "_par". Every 5 DVDs we burn
>has an accompanying PAR2 archive consisting of enough recovery blocks
>to completely recover any one DVD's contents. This gives us some
>resiliency against a scratch taking out a file or some worse insult
>taking out an entire DVD (including, of course, simply -losing- the
>entire DVD... :) Note that calculating a PAR2 archive across the
>~23gig of data involved is fairly slow; the sweet spot on filesystem
>buffering vs PAR2 memory-handling turns out to be "par2 -m64" on this
>hardware (AMD 2800+) and generating the data takes about six hours.
>This inflates the number of burned DVD's by 20%, but eliminates a
>large number of possible problems with archiving this data for perhaps
>years, and the cost of using DVDs this way is around 10-20% of what
>using spinning hard disks would be for this enormous amount of data.

>[.Soon there will be another column called "_cc" (type text) which will
>hold the closed-captioning data for the program, if any, to enable
>searching it for programs mentioning certain terms. Obviously, this
>entire scheme depends on -nothing- in Myth itself ever deleting
>something from "oldrecorded"; if such a thing can happen, I'll have
>to make an entirely separate table so Myth keeps its mitts off it.]

Do you plan to make the scripts to do this public? The reason I ask is
I'm thinking of writing a generic import/export plugin for mythtv that eventually
will allow you to archive tv recording and video files as well as music files to
various formats (raw files, dvd format, ipod, psp, mp3 cd etc.). Alot of this can already
be done using various scripts/tools scattered around the web. I'd just like to create
some sort of generic frontend that people can extend to suit there needs.

You would be able to choose to archive a recording/file from various points in the myth
interface, say the watch recording screen, or from within MythVideo or MythMusic
which would probably just add some details into a db table for later use. Later you
could open the archive plugin which would show you the files you have chosen
to archive and allow you choose what format you would like to archive the files
to and to enter any options required etc. Finally the plugin would just run the
tool/script that does all the hard work passing any options chosen etc.

One of the first import/exporters I was going to include was a way to save myth
recordings to DVD along with all the metadata to recreate the recording in myth.
It sound like you have already done that =) at least the exporting part.

Out of interest why don't you save all the recordedmarkup info for a recording so
you don't have to recreate it later?

One other point. I guess you value your recordings more that me but I really don't want
to have a machine chugging away for 6 hours just so I can create a backup. For what blank
DVD's cost these days if a recording is that important I would rather just burn a second
copy. Just a thought.

Paul H.


_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


jra at baylink

Feb 6, 2006, 12:44 PM

Post #10 of 28 (8000 views)
Permalink
Re: Storage Groups functionality [In reply to]

On Mon, Feb 06, 2006 at 08:09:16PM -0000, Paul Harrison wrote:
> One other point. I guess you value your recordings more that me but
> I really don't want to have a machine chugging away for 6 hours just
> so I can create a backup. For what blank DVD's cost these days if a
> recording is that important I would rather just burn a second copy.

You're a watcher, not a keeper.

We keep *seasons* of shows around on-line, because it's convenient to
refer back to them. My personal goal would be to be able to buy a used
jukebox off someone on eBay, and have it filled with shows which were
archived off the MythBox (preferably as-transcoded to MPEG-4), and have
all the metadata live in the system, so that if we wanted to look at
something, we could just hit play, and the box would tell the juke to
either mount said disc, or copy the file in to staging for playing from
HDD. Or, at the very least, to prompt us to manually drop a DVD-R in
the drive for said purposes, as a first cut.

You understand what I mean, though? My personal wish is for Myth to
*natively* support archiving of programs which it still considers under
it's control.

This would generalize not only to DVD-R's as above, but to multiple
disconnected hard-drives, identifiable by VOLID or mount point.

Cheers,
-- jra
--
Jay R. Ashworth jra [at] baylink
Designer Baylink RFC 2100
Ashworth & Associates The Things I Think '87 e24
St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274

A: No.
Q: Should I include quotations after my message body?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


f-myth-users at media

Feb 6, 2006, 1:31 PM

Post #11 of 28 (8006 views)
Permalink
Storage Groups functionality [In reply to]

> Date: Mon, 6 Feb 2006 20:09:16 -0000
> From: "Paul Harrison" <mythtv [at] dsl>

> Do you plan to make the scripts to do this public?

Sure, I'd be happy to make them available once they're reasonably
stable. Some of them might be a bit too specialized for anyone else
to use, but you never know...

> The reason I ask is
> I'm thinking of writing a generic import/export plugin for mythtv that eventually
> will allow you to archive tv recording and video files as well as music files to
> various formats (raw files, dvd format, ipod, psp, mp3 cd etc.). Alot of this can already
> be done using various scripts/tools scattered around the web. I'd just like to create
> some sort of generic frontend that people can extend to suit there needs.

That sounds like a really nice utility to have.

> You would be able to choose to archive a recording/file from various points in the myth
> interface, say the watch recording screen, or from within MythVideo or MythMusic
> which would probably just add some details into a db table for later use. Later you
> could open the archive plugin which would show you the files you have chosen
> to archive and allow you choose what format you would like to archive the files
> to and to enter any options required etc. Finally the plugin would just run the
> tool/script that does all the hard work passing any options chosen etc.

Sounds good to me!

> One of the first import/exporters I was going to include was a way to save myth
> recordings to DVD along with all the metadata to recreate the recording in myth.
> It sound like you have already done that =) at least the exporting part.

Well, sort of. I'm not sure yet that I really -do- have the exporting
part, at least when it comes to which DB information needs to be
saved. I'm in the process of seeing what's really required there, and
furthermore this might change once 0.19 comes out (my work so far is
based on 0.18.1 because I need stability).

Out of interest why don't you save all the recordedmarkup info for a recording so
you don't have to recreate it later?

My impression was that it's quite a bit of data per hour just to save
the info needed to making skipping work. Granted, it's probably not
all that much, and -could- just get written to the DVD along with the
actual mpeg and not consume much space. I haven't actually verified
whether that supposition is correct, though, and my impression was
that just doing the --rebuild was almost as fast as simply reading the
raw data from the disk. Commflagging is another story, though, so it
might be worth just saving everything so as not to have to wait for
all that work to be redone if a recording is re-imported. (Can I save
commflagging data without saving all the data required to make
skipping work? Haven't checked yet.)

Btw, in case anyone's wondering why I'm bothering to save (or
regenerate) commflagging data instead of just saving the video
without the ads:
(a) I can't (yet) do mpeg2->mpeg2 cutting (though in theory this
should work in .19, yes?), which means I'd have to transcode.
[.Even if I -could-, unless it was "minimally invasive" along
the lines of dvbcut, it might lose the closed-captioning data.]
(b) I haven't been able to make transcoding to megp4 work (at least,
using whatever Myth 0.18.1 does by default if enabled):
(1) Audio level goes down many dB.
(2) The transcoded mpeg4 blows up my mplayer.
(3) There's no agreed-upon way to keep the closed-captioning data
in the resulting video stream.
[.I could cope with (3) if (1) & (2) looked solvable, but I've
asked on the -users list and haven't gotten any solutions. It
sure would be nice to halve my data requirements, though---given
that I'm already stripping the CC data into ASCII text, I could
stand to toss it from the video stream (meaning no on-screen
captioning) if the transcoding step worked for me.]
(c) A lot of this data may get written directly to DVD in a fairly
automated process, having been checked only superficially to make
sure the video isn't blank or messed up (as I said, this isn't
the typical "I'm watching TV" sort of application :). Since
commflagging isn't 100% accurate, we can't just chuck those parts
of the video that the commflagger suspected were ads, but since
nobody's necessarily going to spend the time making sure the
cutpoints are accurate, the whole thing gets archived and dealt
with on-demand later if somebody actually needs a comm-free video.
So given how cheap disk (and especially DVD) storage is vs human
time, it's easier to just save everything.

> One other point. I guess you value your recordings more that me but I really don't want
> to have a machine chugging away for 6 hours just so I can create a backup. For what blank
> DVD's cost these days if a recording is that important I would rather just burn a second
> copy. Just a thought.

The problem with a second copy is that it uses twice as many DVDs, and
I don't expect every original DVD to fail. Properly done, a PAR2
archive with 20% redundancy means that any single DVD in a group of
five can be recovered no matter what happens to it, while only
increasing the number of DVDs required by 20%. That's probably still
too much redundancy; 10% with groups of 10 DVDs would work fine, too,
but then either generating PAR or recovering a single DVD would take 12
hours, and would require over 50GB of temporary disk space for either
operation. That's still feasible (and reduces the total number of
DVDs burned by 10%), but anything bigger would start to get pretty
irritating. And since the machine doing the computation is otherwise
pretty much unloaded, the CPU time is just going to waste anyway.

(How many DVDs -do- I expect to fail? I dunno. I've heard wild
lifetimes that are all over the map, from months to centuries. I'm
using Taiyo Yuden DVDs, which are generally highly regarded, but even
if reliability per-disk is quite high, if we potentially might have
thousands of them, then the chances of a single-disk failure are
actually non-negligible if the failures are independent. And media
longevity might not be the dominant failure mode, anyway---it doesn't
include the possibility that something will go wrong with the robot
that will shatter a disk! :)
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


adeffs.mythtv at gmail

Feb 6, 2006, 1:39 PM

Post #12 of 28 (8000 views)
Permalink
Re: Storage Groups functionality [In reply to]

On 2/6/06, Jay R. Ashworth <jra [at] baylink> wrote:
> On Mon, Feb 06, 2006 at 08:09:16PM -0000, Paul Harrison wrote:
> > One other point. I guess you value your recordings more that me but
> > I really don't want to have a machine chugging away for 6 hours just
> > so I can create a backup. For what blank DVD's cost these days if a
> > recording is that important I would rather just burn a second copy.
>
> You're a watcher, not a keeper.
>
> We keep *seasons* of shows around on-line, because it's convenient to
> refer back to them. My personal goal would be to be able to buy a used
> jukebox off someone on eBay, and have it filled with shows which were
> archived off the MythBox (preferably as-transcoded to MPEG-4), and have
> all the metadata live in the system, so that if we wanted to look at
> something, we could just hit play, and the box would tell the juke to
> either mount said disc, or copy the file in to staging for playing from
> HDD. Or, at the very least, to prompt us to manually drop a DVD-R in
> the drive for said purposes, as a first cut.
>
> You understand what I mean, though? My personal wish is for Myth to
> *natively* support archiving of programs which it still considers under
> it's control.
>
> This would generalize not only to DVD-R's as above, but to multiple
> disconnected hard-drives, identifiable by VOLID or mount point.
>

so basically you want to merge the Watch Recordings screen and MythVideo?

Which is similar to what I think would be cool.

Right now I add my archived shows manually to the db to list them in
my Watch Recordings screen, they're symlinked from nfs mounted drives.
I wrote a perl program to scrape tv.com for the show information and
add the show to the recorded table in the db. I archive in xvid avi's
so no mythcommflag --rebuild is required, but I've included in my perl
program a check so that if the file being added is an mpeg it will run
mythcommflag after db insertion.

I like the idea of recording groups being able to specify a recording
subdirectory. It would also be nice to have a menu option to move a
recording to a subdirectory, have it move the file and redirect the db
information to the new location.

I also like the idea of being able to "hot-mount" drives under the
recordings drive and have Myth import the shows automatically. Perhaps
an xml file with the same base name as the recorded file that contains
the information Myth needs (the shows guide information, etc). This
way Myth just has to keep watch for newly mounted subdirectories and
then scan the xml files.

--
Steve
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


jra at baylink

Feb 6, 2006, 2:26 PM

Post #13 of 28 (7996 views)
Permalink
Re: Storage Groups functionality [In reply to]

On Mon, Feb 06, 2006 at 04:39:40PM -0500, Steven Adeff wrote:
> On 2/6/06, Jay R. Ashworth <jra [at] baylink> wrote:
> > We keep *seasons* of shows around on-line, because it's convenient to
> > refer back to them. My personal goal would be to be able to buy a used
> > jukebox off someone on eBay, and have it filled with shows which were
> > archived off the MythBox (preferably as-transcoded to MPEG-4), and have
> > all the metadata live in the system, so that if we wanted to look at
> > something, we could just hit play, and the box would tell the juke to
> > either mount said disc, or copy the file in to staging for playing from
> > HDD. Or, at the very least, to prompt us to manually drop a DVD-R in
> > the drive for said purposes, as a first cut.
> >
> > You understand what I mean, though? My personal wish is for Myth to
> > *natively* support archiving of programs which it still considers under
> > it's control.
> >
> > This would generalize not only to DVD-R's as above, but to multiple
> > disconnected hard-drives, identifiable by VOLID or mount point.
>
> so basically you want to merge the Watch Recordings screen and MythVideo?

Explicitly, no.

MythVideo a) is out-of-band to the normal use of the box, and b) ...
well, I dunno; let me think on that further; perhaps my knee was
jerking. I don't *think* so, though; MythVideo is sort of a second
class citizen.

For example, programs moved from Myth to MV would no longer be skipped
in recording schedules, I don't think.

> Right now I add my archived shows manually to the db to list them in
> my Watch Recordings screen, they're symlinked from nfs mounted drives.
> I wrote a perl program to scrape tv.com for the show information and
> add the show to the recorded table in the db. I archive in xvid avi's
> so no mythcommflag --rebuild is required, but I've included in my perl
> program a check so that if the file being added is an mpeg it will run
> mythcommflag after db insertion.

But you already *recorded them* though myth in the first place, no? So
why *dump* all the metadata in the first place?

> I also like the idea of being able to "hot-mount" drives under the
> recordings drive and have Myth import the shows automatically. Perhaps
> an xml file with the same base name as the recorded file that contains
> the information Myth needs (the shows guide information, etc). This
> way Myth just has to keep watch for newly mounted subdirectories and
> then scan the xml files.

I was merely looking for Myth to a) permit them to be recorded on/moved
to that removable drive, and b) not have apoplexy if they're
temporarily off-line -- while still otherwise treating them as
first-class citizens.

Cheers,
-- jra
--
Jay R. Ashworth jra [at] baylink
Designer Baylink RFC 2100
Ashworth & Associates The Things I Think '87 e24
St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274

A: No.
Q: Should I include quotations after my message body?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


adeffs.mythtv at gmail

Feb 6, 2006, 3:24 PM

Post #14 of 28 (8015 views)
Permalink
Re: Storage Groups functionality [In reply to]

On 2/6/06, Jay R. Ashworth <jra [at] baylink> wrote:
> On Mon, Feb 06, 2006 at 04:39:40PM -0500, Steven Adeff wrote:
> > On 2/6/06, Jay R. Ashworth <jra [at] baylink> wrote:
> > > We keep *seasons* of shows around on-line, because it's convenient to
> > > refer back to them. My personal goal would be to be able to buy a used
> > > jukebox off someone on eBay, and have it filled with shows which were
> > > archived off the MythBox (preferably as-transcoded to MPEG-4), and have
> > > all the metadata live in the system, so that if we wanted to look at
> > > something, we could just hit play, and the box would tell the juke to
> > > either mount said disc, or copy the file in to staging for playing from
> > > HDD. Or, at the very least, to prompt us to manually drop a DVD-R in
> > > the drive for said purposes, as a first cut.
> > >
> > > You understand what I mean, though? My personal wish is for Myth to
> > > *natively* support archiving of programs which it still considers under
> > > it's control.
> > >
> > > This would generalize not only to DVD-R's as above, but to multiple
> > > disconnected hard-drives, identifiable by VOLID or mount point.
> >
> > so basically you want to merge the Watch Recordings screen and MythVideo?
>
> Explicitly, no.
>
> MythVideo a) is out-of-band to the normal use of the box, and b) ...
> well, I dunno; let me think on that further; perhaps my knee was
> jerking. I don't *think* so, though; MythVideo is sort of a second
> class citizen.
>
> For example, programs moved from Myth to MV would no longer be skipped
> in recording schedules, I don't think.
>
> > Right now I add my archived shows manually to the db to list them in
> > my Watch Recordings screen, they're symlinked from nfs mounted drives.
> > I wrote a perl program to scrape tv.com for the show information and
> > add the show to the recorded table in the db. I archive in xvid avi's
> > so no mythcommflag --rebuild is required, but I've included in my perl
> > program a check so that if the file being added is an mpeg it will run
> > mythcommflag after db insertion.
>
> But you already *recorded them* though myth in the first place, no? So
> why *dump* all the metadata in the first place?

yes and no. Some are shows from BBC/Sky One shows that I don't get
here. Some are shows I downloaded before I had a MythTV machine, some
are shows I ripped from my TiVo, etc. Most of which are xvid
transcodes done outside of Myth. So now I'll take my Myth recordings,
transcode them on my other machine, save the output to one of my
archive drives and use my script to import them into the Watch
Recordings list.

I do this because I don't like the built in transcoding to mpeg-4, I
prefer more control over the encode. I could just relink them in the
db but this method lets me more easily import more than what I've
recorded in Myth without having to have two seperate processes as
well.

Eventually I'll buy them on HD-DVD, and hopefully hdd costs will come
down enough that I won't have to transcode I can just commercial cut
and have a huge RAID5/RAID6 array for my recordings so I can keep them
like this until I can purchase them on HD-DVD (or regular DVD
depending on the show).


> > I also like the idea of being able to "hot-mount" drives under the
> > recordings drive and have Myth import the shows automatically. Perhaps
> > an xml file with the same base name as the recorded file that contains
> > the information Myth needs (the shows guide information, etc). This
> > way Myth just has to keep watch for newly mounted subdirectories and
> > then scan the xml files.
>
> I was merely looking for Myth to a) permit them to be recorded on/moved
> to that removable drive, and b) not have apoplexy if they're
> temporarily off-line -- while still otherwise treating them as
> first-class citizens.

something I'd live to see as well, just trying to think out some ideas
for how to implement.

--
Steve
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 6, 2006, 8:33 PM

Post #15 of 28 (7996 views)
Permalink
Re: Storage Groups functionality [In reply to]

> > Date: Sun, 5 Feb 2006 23:33:24 -0500 (EST)
> > From: "Chris Pinkham" <cpinkham [at] bc2va>
>
> > Do you have any specific ideas on how archiving would
> > integrate with Storage Groups specifically?
>
> I do, since this is something I'm looking at right this second; maybe
> my solution will give people some ideas. I'm (going to be) archiving
> large amounts of data off Myth for later use in what amounts to a tape

<long reply deleted>

I still don't see what any of this has to do with the directory that
the recording was originally in other than the fact that the location
of the archived file is not the original location and hence could be
considered to be in another Storage Group.

> One unsolved problem: It'd be really, really nice to be able to take
> an archived program (which has presumably been deleted from "recorded"
> because it's been deleted from the UI) and reinsert all the relevant
> table data again once the video bits have been restored to disk. Is
> it sufficient to simply save the relevant row from the "recorded"
> table and reinsert it? (I'm assuming that mythcommflag --rebuild will

IMHO, the archiving discussion is drifting off-topic, but why even delete
the data from the database at all? All you need to do is flag the
recording so that it is not shown in the recordings list if it is
archived. When an 'archive' was restored (by either mounting a disk
or external drive, etc., then you just have to mark the archived
recordings on that media as available and they show up in the Watch
Recordings screen.

You're making this way too complicated, a simple 'archived' flag in
the recorded table along with an archive location would handle it.

Again, I think the archive discussion is drifting off-topic, it has
virtually nothing to do with Storage Groups that I can see.

--
Chris

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 6, 2006, 8:38 PM

Post #16 of 28 (7995 views)
Permalink
Re: Storage Groups functionality [In reply to]

> If there are multiple storage units, MythTV should run a little one time
> performance test for each storage volume when the drives are idle. Maybe
> bonnie++ or some such utility. Preference could be given to the fastest storage
> volumes for HD recordings.

Speaking for myself, this is way too indepth. There are too many things
that could affect this and too much logic required in Myth. I doubt you'd
see any of the core developers implementing this.

> Maybe instead of straight round robin, a recording will go to the storage
> volumes that has the most available free space.

This method was listed in my original email.

> It would also be nice to tell it to balance the recordings amoung the volumes
> when MythTV is idle. For maintenence purposes it would be good to tell Myth to
> move all the recordings from one volume to the rest in case you need to take
> that volume off line.

Again, this is quite a bit of logic and code for very little gain, I doubt you'll
see anything like this anytime soon. Maybe as a user-contributed script that was
run from cron, but not in Myth itself. If you use most-available-free-space as
your load-balancing method, then there is no need to waste time shuffling files
around at night because new recordings will fill in the "gaps" left by deleting
recordings, so your file systems should stay fairly equal over time.

--
Chris

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 6, 2006, 8:44 PM

Post #17 of 28 (7996 views)
Permalink
Re: Storage Groups functionality [In reply to]

> archived off the MythBox (preferably as-transcoded to MPEG-4), and have
> all the metadata live in the system, so that if we wanted to look at
> something, we could just hit play, and the box would tell the juke to
> either mount said disc, or copy the file in to staging for playing from
> HDD. Or, at the very least, to prompt us to manually drop a DVD-R in
> the drive for said purposes, as a first cut.

I agree with this idea. If there was an 'archived' flag and an 'archive
location' for each recording, then mythfrontend could prompt for the
proper media to be inserted or could even run a script to autoload the
media indicated.

> You understand what I mean, though? My personal wish is for Myth to
> *natively* support archiving of programs which it still considers under
> it's control.

I agree. It would be nice to be able to burn a few shows off to DVD,
tell Myth I 'archived' them to DVD #109, and potentially have those
recordings show up as archived or not even show up unless I went to
the 'archived recordings' group/filter/whatever. The autoload script
mentioned above could work like the userjobs and be passed things like
the archive location, chanid, starttime, filename, dirname, etc..

--
Chris (watching this to go further off-topic)

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 6, 2006, 8:52 PM

Post #18 of 28 (7982 views)
Permalink
Re: Storage Groups functionality [In reply to]

> so basically you want to merge the Watch Recordings screen and MythVideo?
>
> Which is similar to what I think would be cool.

I have a patch that I wrote at least a year ago that allows you to view
the MythVideo files in the normal Watch Recordings screen. It allows
you to play the files just like they were normal Myth recordings, but
it uses the MythVideo player code so it uses the internal Myth player,
mplayer, xine, etc., whatever you have configured for that file. It
has the ability to list the files under their own title or under a
virtual 'MythVideo' title. It's crude, but made it so I could use
the same interface for my normal Myth stuff and some .avi recordings I
have.

--
Chris (even more off-topic now)

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


jra at baylink

Feb 6, 2006, 8:54 PM

Post #19 of 28 (7996 views)
Permalink
Re: Storage Groups functionality [In reply to]

On Mon, Feb 06, 2006 at 11:44:51PM -0500, Chris Pinkham wrote:
> > archived off the MythBox (preferably as-transcoded to MPEG-4), and have
> > all the metadata live in the system, so that if we wanted to look at
> > something, we could just hit play, and the box would tell the juke to
> > either mount said disc, or copy the file in to staging for playing from
> > HDD. Or, at the very least, to prompt us to manually drop a DVD-R in
> > the drive for said purposes, as a first cut.
>
> I agree with this idea. If there was an 'archived' flag and an 'archive
> location' for each recording, then mythfrontend could prompt for the
> proper media to be inserted or could even run a script to autoload the
> media indicated.

He's correct that this is OT for Storage Groups, and my apologies for
dragging it off there. But yes, that's what I'm after.

> > You understand what I mean, though? My personal wish is for Myth to
> > *natively* support archiving of programs which it still considers under
> > it's control.
>
> I agree. It would be nice to be able to burn a few shows off to DVD,
> tell Myth I 'archived' them to DVD #109, and potentially have those
> recordings show up as archived or not even show up unless I went to
> the 'archived recordings' group/filter/whatever. The autoload script
> mentioned above could work like the userjobs and be passed things like
> the archive location, chanid, starttime, filename, dirname, etc..

Exactly.

Cheers,
-- jra
--
Jay R. Ashworth jra [at] baylink
Designer Baylink RFC 2100
Ashworth & Associates The Things I Think '87 e24
St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274

A: No.
Q: Should I include quotations after my message body?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Feb 6, 2006, 9:04 PM

Post #20 of 28 (7996 views)
Permalink
Re: Storage Groups functionality [In reply to]

> > I agree with this idea. If there was an 'archived' flag and an 'archive
> > location' for each recording, then mythfrontend could prompt for the
> > proper media to be inserted or could even run a script to autoload the
> > media indicated.
>
> He's correct that this is OT for Storage Groups, and my apologies for
> dragging it off there. But yes, that's what I'm after.

:) I saved a copy in my mythtv-ideas mailbox for future reference in
case I get to that part on my TODO list before someone else does.

--
Chris

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


jra at baylink

Feb 7, 2006, 7:17 AM

Post #21 of 28 (7985 views)
Permalink
Re: Storage Groups functionality [In reply to]

On Tue, Feb 07, 2006 at 12:04:01AM -0500, Chris Pinkham wrote:
> > He's correct that this is OT for Storage Groups, and my apologies for
> > dragging it off there. But yes, that's what I'm after.
>
> :) I saved a copy in my mythtv-ideas mailbox for future reference in
> case I get to that part on my TODO list before someone else does.

I will attempt, some time this week, to write up in more detail exactly
what I think I mean, and put it up on [[Feature Wishlist]].

Cheers,
-- jra
--
Jay R. Ashworth jra [at] baylink
Designer Baylink RFC 2100
Ashworth & Associates The Things I Think '87 e24
St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274

A: No.
Q: Should I include quotations after my message body?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mythtv-dev at hyperbole-software

Feb 7, 2006, 11:17 AM

Post #22 of 28 (7986 views)
Permalink
Re: Storage Groups functionality [In reply to]

f-myth-users [at] media wrote:

><snip...>
>
>.... Obviously, this entire scheme depends on -nothing- in Myth itself ever deleting something from "oldrecorded"; if such a thing can happen, I'll have
>to make an entirely separate table so Myth keeps its mitts off it....
>
><...snip...>
>
>
>
If you look at mythtv/programs/mythfilldatabase/filldata.cpp:2805 of the
latest svn (8893) you'll see that mythfilldatabase deletes records older
than 320 days from oldrecorded.



Carl.




_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Feb 7, 2006, 3:05 PM

Post #23 of 28 (7978 views)
Permalink
Re: Storage Groups functionality [In reply to]

Carl Reynolds wrote:

>f-myth-users [at] media wrote:
>
>><snip...>
>>
>>.... Obviously, this entire scheme depends on -nothing- in Myth itself ever deleting something from "oldrecorded"; if such a thing can happen, I'll have
>>to make an entirely separate table so Myth keeps its mitts off it....
>>
>><...snip...>
>>
>If you look at mythtv/programs/mythfilldatabase/filldata.cpp:2805 of the
>latest svn (8893) you'll see that mythfilldatabase deletes records older
>than 320 days from oldrecorded.
>
>
oldprogram != oldrecorded (just do a desc on each to see that they're
_very_ differrent tables)

The only records mythfilldatabase ever deletes from oldrecorded are
those that are older than the user-specified setting "CleanOldRecorded"
and that were not recorded. (SVN stores programs that match recording
rules in oldrecorded even if they're not recorded so you can check why
they weren't recorded--within CleanOldRecorded days, at least. In
0.18.1 and below, mfdb should not ever delete records from oldrecorded.)

Deleting records that show what you recorded from oldrecorded would be A
Bad Thing considering they take very little space and provide
information (so that in you don't start re-recording season 1 when
season 3 starts--which it would do if the records were removed every 320
days (~1 year)).

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mythtv-dev at hyperbole-software

Feb 7, 2006, 6:27 PM

Post #24 of 28 (7964 views)
Permalink
Re: Storage Groups functionality [In reply to]

Michael T. Dean wrote:

>Carl Reynolds wrote:
>
>
>
>>f-myth-users [at] media wrote:
>>
>>
>>
>>><snip...>
>>>
>>>.... Obviously, this entire scheme depends on -nothing- in Myth itself ever deleting something from "oldrecorded"; if such a thing can happen, I'll have
>>>to make an entirely separate table so Myth keeps its mitts off it....
>>>
>>><...snip...>
>>>
>>>
>>>
>>If you look at mythtv/programs/mythfilldatabase/filldata.cpp:2805 of the
>>latest svn (8893) you'll see that mythfilldatabase deletes records older
>>than 320 days from oldrecorded.
>>
>>
>>
>>
>oldprogram != oldrecorded (just do a desc on each to see that they're
>_very_ differrent tables)
>
>The only records mythfilldatabase ever deletes from oldrecorded are
>those that are older than the user-specified setting "CleanOldRecorded"
>and that were not recorded. (SVN stores programs that match recording
>rules in oldrecorded even if they're not recorded so you can check why
>they weren't recorded--within CleanOldRecorded days, at least. In
>0.18.1 and below, mfdb should not ever delete records from oldrecorded.)
>
>Deleting records that show what you recorded from oldrecorded would be A
>Bad Thing considering they take very little space and provide
>information (so that in you don't start re-recording season 1 when
>season 3 starts--which it would do if the records were removed every 320
>days (~1 year)).
>
>Mike
>_______________________________________________
>mythtv-dev mailing list
>mythtv-dev [at] mythtv
>http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
>
>
>
>
>
You're right Mike,

I was in a hurry when I read the original question and remembered having
looked at that section of filldata.cpp a couple of days ago. I didn't
check the table name carefully enough. Sorry. :-[



Carl.



_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


f-myth-users at media

Feb 14, 2006, 1:59 AM

Post #25 of 28 (7918 views)
Permalink
Storage Groups functionality [In reply to]

> Date: Mon, 6 Feb 2006 23:33:46 -0500 (EST)
> From: "Chris Pinkham" <cpinkham [at] bc2va>

[.Prolonging this one more message since it might give somebody an idea,
and because "Paul Harrison" <mythtv [at] dsl> and---I think---at
least one other person in this thread have also said they're talking
about making some sort of offlining tool.]

> > One unsolved problem: It'd be really, really nice to be able to take
> > an archived program (which has presumably been deleted from "recorded"
> > because it's been deleted from the UI) and reinsert all the relevant
> > table data again once the video bits have been restored to disk. Is
> > it sufficient to simply save the relevant row from the "recorded"
> > table and reinsert it? (I'm assuming that mythcommflag --rebuild will

> IMHO, the archiving discussion is drifting off-topic, but why even delete
> the data from the database at all? All you need to do is flag the
> recording so that it is not shown in the recordings list if it is
> archived. When an 'archive' was restored (by either mounting a disk
> or external drive, etc., then you just have to mark the archived
> recordings on that media as available and they show up in the Watch
> Recordings screen.

If you're moving recordings offline, but will later be hauling them
back online, the one table that it might make sense to move offline
with them seems to be recordedmarkup. That table currently consumes
75% of all the space my DB consumes, and seems to correspond to about
1 megabyte per hour of recorded video. (If you export the data and
compress it, it's about 60K or so.) A meg per hour doesn't seem like
much, but if the whole idea of moving things offline is because you
save a lot of things, it can bloat the database fairly effectively---
a terabyte of online storage means a gigabyte of recordedmarkup in the
DB, and lots of people -have- a terabyte (or sizeable fraction thereof)
online, so it seems safe to assume that offline storers are probably
storing 10x that or more---and now we're talking 10GB of database and
a fairly substantial figure.

(I used to think, "just regenerate recordedmarkup with mythcommflag
--rebuild when the recording is put back online", but recent
discussions indicate that this only works for ivtv cards until 19, and
worse, I think you'd have to rerun commercial-flagging and get all the
cutpoints accurate again, which would use a lot of runtime and human
time. Easier to just store a 60K compressed file that's associated
with the video when you migrate it offline and stick it back in the
table when it comes back. Or maybe just store the -compressed-
version in the DB? Then, 10TB of recordings is 600meg of space in
mysql, which isn't too horrible, though still, even that space is
wasted w/o the original video it refers to. I'm a little surprised
that mysql isn't already using a much more compact format for this
data, but my naive analysis of this (export whole table, check its
size, and compare to the size of everything under /var/lib/mysql---and
they're about equal!) seems to indicate that this might not be the
case.) [.Hmm---though I suppose, if you write out this table w/a
recording, you'd better also write out DBSchemaVer with it in case the
schema changes in the future in an incompatible way, so you can know
how to fix up what you read in... ick.]

And until the UI -does- have an "archived" flag (and probably a way of
saying "and don't show them to me, either" so you don't drown in them),
I'm going to have to actually delete offlined recordings, which means
I'm back to my original question of, "And what tables must I dump to
make later reinsertion work?" Since nobody answered that one, I'll
try to figure out all the DB calls made when a recording is deleted,
which should tell me which tables to look at more closely. (Or I'll
export mythconverg, delete a recording, export again, and diff---that
could be a lot faster, easier, and less error-prone... :)

P.S. My question about the "recorded" table above might be moot, since
I reported earlier that only shows with no pre/postroll were winding up
in there (at least in 18; nobody said anything about whether this was
a bug or a feature, but it sure seems strange to me), and yet recordings
not being in there doesn't -seem- to have affected anything, so it's
not clear to me why this table is there until I take a closer look at
the sources. And clearly I'm not going to touch oldrecorded; that's
keeping track of where any offline files went, via some added columns.
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

First page Previous page 1 2 Next page Last page  View All MythTV dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.