Mailing List Archive: MythTV: Users

Mooting architecture for a DataDirect replacement

1 2 3 4 5 6

View All

Index | Next | Previous | View Threaded

jra at baylink

Jun 20, 2007, 1:54 PM

Post #1 of 140 (2351 views)
Permalink

Mooting architecture for a DataDirect replacement

On Wed, Jun 20, 2007 at 11:58:09AM -0700, Chris Petersen wrote:
> We are also looking at other providers, as well as the option of working
> with other projects (both open and closed source) to band together on a
> similar service.

That seems like a launching point for this discussion. Since
architecture is the only thing I'm really good at, I'll start. :-)

(If you're not interested, Kill the thread. If you're offended, relax.)

If the time comes when we have to create a service to replace Z2L's
DataDirect, there *have* to be better ways to accomplish it, leveragint
technologies and protocols which have already been invented in the last
30 years on the internet.

To decide which ones might be best, I think we need to start by
enumerating the goals and describing the data and it's sources.

The fundamental item -- and yes, I know, DD's schema is floating around
in (I think) a PDF -- is an AIRING, which is a combination of PROGRAM,
NETWORK, and TIME.

NETWORK maps to CHANNEL through SERVICE, TIME includes a TIMEZONE and
is exact to the second, and PROGRAM is, where possible, primary keyed
by PRODUCTIONCO and PRODUCTIONNUM, which I believe to be globally
unique.

The PROGRAM records originate from the production company -- for
Scrubs, frex, that's Touchstone TV. The AIRING records originate from
the network, and filter through the local station, for network
affiliated broadcast stations. For cable, they mostly don't filter.

One of the *most* important characteristics of AIRING data for the PVR
market is that you want to propagate changes to the audience just as
*late* as you possibly can -- 5 minutes before airtime is not
unreasonable, nor impossible, if you pick the right transmission
architecture.

This will occasionally screw with MythTV's schedule architecture, but
that's the price you pay for predictive scheduling.

The second most important issue is *accurate* start and end timing. If
you're gonna start ER at 9:59, damnit, *say so*. There's no reason to
get that wrong except passive aggressive attempts to piss off PVR
owners enough to watch live, IMO.

*My* personal approach to the transfer problem would be to package up
whatever updates a given source has, and post them to a Usenet
Newsgroup on some server, appropriate signed with a public key so it's
clear that they're valid, and possibly encrypted with another key, if
that's necessary commercially. A system similar to CSS could be used,
where a random symmetric key is PKC encrypted with multiple public
keys and appended to the packet.

Each box could then subscribe to that newsgroup, and ping the server
for new news every ... 5 minutes? The infrastructure is *designed* for
that sort of thing, and very well understood; proper server coverage --
even in these days of Supernews et al -- would be trivial, and flood
quite quickly... and you could push out "The president's on! He's on
*every channel*!!" updates in jig time.

Thoughts?

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

ylee at pobox

Jun 20, 2007, 3:42 PM

Post #2 of 140 (2301 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Jay R. Ashworth <jra [at] baylink> says:
> One of the *most* important characteristics of AIRING data for the
> PVR market is that you want to [continue to] propagate changes to
> the audience just as *late* as you possibly can -- 5 minutes before
> airtime is not unreasonable, nor impossible, if you pick the right
> transmission architecture.

I inserted a key omitted phrase in the above because, of course,
lineup changes ought to be propagated ASAP *and on an ongoing basis*.

Otherwise, I heartily agree with your larger message about how this
event could spark an overdue revolution in how these things work in
the first place.

--
Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythlist at michaelandholly

Jun 20, 2007, 6:19 PM

Post #3 of 140 (2284 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Ok.

How about this..

Rather than creating a downloadable, encrypted file which must be
downloaded in its entirety why not create an SQL database which
tracks the programs on in a specific market. Have the Mythtv Log
into this database query for that system's market and only update
records which need updating?

This should reduce downloads significantly since the only information
that needs to be downloaded is new information or information that
needs to change since the last download.

- Michael
On Jun 20, 2007, at 4:54 PM, Jay R. Ashworth wrote:

> On Wed, Jun 20, 2007 at 11:58:09AM -0700, Chris Petersen wrote:
>> We are also looking at other providers, as well as the option of
>> working
>> with other projects (both open and closed source) to band together
>> on a
>> similar service.
>
> That seems like a launching point for this discussion. Since
> architecture is the only thing I'm really good at, I'll start. :-)
>
> (If you're not interested, Kill the thread. If you're offended,
> relax.)
>
> If the time comes when we have to create a service to replace Z2L's
> DataDirect, there *have* to be better ways to accomplish it,
> leveragint
> technologies and protocols which have already been invented in the
> last
> 30 years on the internet.
>
> To decide which ones might be best, I think we need to start by
> enumerating the goals and describing the data and it's sources.
>
> The fundamental item -- and yes, I know, DD's schema is floating
> around
> in (I think) a PDF -- is an AIRING, which is a combination of PROGRAM,
> NETWORK, and TIME.
>
> NETWORK maps to CHANNEL through SERVICE, TIME includes a TIMEZONE and
> is exact to the second, and PROGRAM is, where possible, primary keyed
> by PRODUCTIONCO and PRODUCTIONNUM, which I believe to be globally
> unique.
>
> The PROGRAM records originate from the production company -- for
> Scrubs, frex, that's Touchstone TV. The AIRING records originate from
> the network, and filter through the local station, for network
> affiliated broadcast stations. For cable, they mostly don't filter.
>
> One of the *most* important characteristics of AIRING data for the PVR
> market is that you want to propagate changes to the audience just as
> *late* as you possibly can -- 5 minutes before airtime is not
> unreasonable, nor impossible, if you pick the right transmission
> architecture.
>
> This will occasionally screw with MythTV's schedule architecture, but
> that's the price you pay for predictive scheduling.
>
> The second most important issue is *accurate* start and end
> timing. If
> you're gonna start ER at 9:59, damnit, *say so*. There's no reason to
> get that wrong except passive aggressive attempts to piss off PVR
> owners enough to watch live, IMO.
>
> *My* personal approach to the transfer problem would be to package up
> whatever updates a given source has, and post them to a Usenet
> Newsgroup on some server, appropriate signed with a public key so it's
> clear that they're valid, and possibly encrypted with another key, if
> that's necessary commercially. A system similar to CSS could be used,
> where a random symmetric key is PKC encrypted with multiple public
> keys and appended to the packet.
>
> Each box could then subscribe to that newsgroup, and ping the server
> for new news every ... 5 minutes? The infrastructure is *designed*
> for
> that sort of thing, and very well understood; proper server
> coverage --
> even in these days of Supernews et al -- would be trivial, and flood
> quite quickly... and you could push out "The president's on! He's on
> *every channel*!!" updates in jig time.
>
> Thoughts?
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink
> jra [at] baylink
> Designer The Things I
> Think RFC 2100
> Ashworth & Associates http://
> baylink.pitas.com '87 e24
> St Petersburg FL USA http://photo.imageinc.us +1
> 727 647 1274
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

lists at forevermore

Jun 20, 2007, 6:31 PM

Post #4 of 140 (2281 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Michael Jones wrote:
> Rather than creating a downloadable, encrypted file which must be
> downloaded in its entirety why not create an SQL database which
> tracks the programs on in a specific market. Have the Mythtv Log
> into this database query for that system's market and only update
> records which need updating?

That would imply redesigning the way that MythTV keeps track of shows.
Works great in a market where you have episode serial numbers (for lack
of a better term), but not when you're limited to EIT data or other
sources where information is of mixed accuracy (e.g. show/episode stay
the same but description changes).

Personally, I'd love to see something like this, since it'd make for a
relatively cleaner MythTV database, too. So, I guess, don't worry too
much that any new system done by the devs will at least have input from
me, and I'd like to aim for something the community can update/maintain
to enhance/correct show descriptions, etc. I'm fed up with inaccuracies
in TMS data, tv.com info, etc., and would love to see something at least
partially driven by the community.

-Chris

Attachments:

signature.asc (0.18 KB)

schachte at csse

Jun 20, 2007, 7:52 PM

Post #5 of 140 (2269 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Michael Jones wrote:

> Rather than creating a downloadable, encrypted file which must be
> downloaded in its entirety why not create an SQL database which
> tracks the programs on in a specific market. Have the Mythtv Log
> into this database query for that system's market and only update
> records which need updating?
>
> This should reduce downloads significantly since the only information
> that needs to be downloaded is new information or information that
> needs to change since the last download.

That sounds like a good way to do it, but it's rather myth-specific and would
require a fair bit of work. You could get most of the benefit in a more
application-neutral way with much less effort by working with XMLTV files. You
could just always keep the last XMLTV file you downloaded, and then use
something like rsync to update it before reloading it into myth.

--
Peter Schachte We hang the petty thieves and appoint the great
schachte [at] cs ones to public office.
www.cs.mu.oz.au/~schachte/ -- Aesop
Phone: +61 3 8344 1338
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythlist at michaelandholly

Jun 20, 2007, 8:21 PM

Post #6 of 140 (2275 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Ok.. if the SQL only is too "myth specific" try this on for size... :-)

If the data were in an SQL database, a frontend on the service could
EASILY export it to a file, if that's what the user/subscription/
preference whatever decided. The point is not the format that is
used.. since the contents of the SQL database can be output into
almost any format imaginable at the drop of a digital hat (usually
measured in milliseconds). Keep the format in most general,
manageable format and distribute it as needed.

Senario:

A) My system runs MythTV, I have an SQL grabber of some kind that
makes a request from the parent server every X minutes, this request
contains the market I need and the last time I did an update. The
parent server retreives all changes since that time for that market
and sends it to me in some sort of compressed stream, my SQL database
updates itself from this data and all is good.

B) The guy down the street has X system.. his needs are Y.. he has
some sort of grabber that communicates with the front end every X
minutes, the query contains his market and the last time he made a
query and the parent server retrieves the information and sends it in
a compressed text stream that his system parses and imports into it's
proprietary format.

C) The guy further down the street has Z system, his likes XML
data. So his server talks to the parent.. sends the proper codes..
blah blah blah and magically in his email inbox arrives an XML data
file that he manually parses and does whatever his little heart
desires with it..

D) Another user loads up a PHP web front end, types in his market,
zip, whatever and up pops the full channel lineup for his region that
he is free to peruse on his web browser.

Ok.. Here's another one..

E) Another system takes RSS feeds.. the back end SQL brain exports
the pertinent data into the proper format via PHP, Java or somesuch
and the RSS based client system sucks up the data as it wants it..

Here's the unique part of the whole thing. The data is in a format
that can be repurposed and exported, published, pushed, pulled,
sucked and .. well you get the idea... to whatever the need is.
The point is keep the data in a predictable, stable, accessible
format on the back end have the front end (or multiple front ends)
manage the publish action with the data. The data is system-
independent. If the user needs only a small portion of data, that
data is provided and it reduces the number of bits transmitted and
therefore the load on the parent system.

Ideas?

Hey.. I know Chris likes it already ;-) If the system were designed
correctly it could even publish the data the correct format for
DataDirect if that's what Myth sticks with.

- Michael

On Jun 20, 2007, at 10:52 PM, Peter Schachte wrote:

> Michael Jones wrote:
>
>> Rather than creating a downloadable, encrypted file which must be
>> downloaded in its entirety why not create an SQL database which
>> tracks the programs on in a specific market. Have the Mythtv Log
>> into this database query for that system's market and only update
>> records which need updating?
>>
>> This should reduce downloads significantly since the only information
>> that needs to be downloaded is new information or information that
>> needs to change since the last download.
>
> That sounds like a good way to do it, but it's rather myth-specific
> and would
> require a fair bit of work. You could get most of the benefit in a
> more
> application-neutral way with much less effort by working with XMLTV
> files. You
> could just always keep the last XMLTV file you downloaded, and then
> use
> something like rsync to update it before reloading it into myth.
>
> --
> Peter Schachte We hang the petty thieves and appoint
> the great
> schachte [at] cs ones to public office.
> www.cs.mu.oz.au/~schachte/ -- Aesop
> Phone: +61 3 8344 1338
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

myth at dermanouelian

Jun 20, 2007, 8:33 PM

Post #7 of 140 (2281 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Jun 20, 2007, at 8:21 PM, Michael Jones wrote:

> Ok.. if the SQL only is too "myth specific" try this on for
> size... :-)
>
> If the data were in an SQL database, a frontend on the service could
> EASILY export it to a file, if that's what the user/subscription/
> preference whatever decided. The point is not the format that is
> used.. since the contents of the SQL database can be output into
> almost any format imaginable at the drop of a digital hat (usually
> measured in milliseconds). Keep the format in most general,
> manageable format and distribute it as needed.

Wasn't this the promise of XML and XSLT?

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythlist at michaelandholly

Jun 20, 2007, 8:43 PM

Post #8 of 140 (2274 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Does XML or XSLT have the ability to control the *amount* of data
that is published? In my limited experience, no.

A system with an "intelligent" back end could publish the data
quickly, accurately, and in virtually any format imaginable.

- Michael

On Jun 20, 2007, at 11:33 PM, Brad DerManouelian wrote:

> On Jun 20, 2007, at 8:21 PM, Michael Jones wrote:
>
>> Ok.. if the SQL only is too "myth specific" try this on for
>> size... :-)
>>
>> If the data were in an SQL database, a frontend on the service could
>> EASILY export it to a file, if that's what the user/subscription/
>> preference whatever decided. The point is not the format that is
>> used.. since the contents of the SQL database can be output into
>> almost any format imaginable at the drop of a digital hat (usually
>> measured in milliseconds). Keep the format in most general,
>> manageable format and distribute it as needed.
>
> Wasn't this the promise of XML and XSLT?
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

schachte at csse

Jun 20, 2007, 9:33 PM

Post #9 of 140 (2283 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Michael Jones wrote:
> Ok.. if the SQL only is too "myth specific" try this on for size... :-)
>
> If the data were in an SQL database, a frontend on the service could
> EASILY export it to a file, if that's what the user/subscription/
> preference whatever decided. The point is not the format that is
> used.. since the contents of the SQL database can be output into
> almost any format imaginable at the drop of a digital hat (usually
> measured in milliseconds). Keep the format in most general,
> manageable format and distribute it as needed.

Sure, there's nothing wrong with using a database, but I think an XMLTV file
would work just as well, and could be much cheaper on server resources to
handle. The scheme you're suggesting requires the server to figure out what's
changed since a certain date for every transaction. If the number of requests
gets high, that could be expensive. It also requires a fair bit of coding. I
would think rsync would be more efficient for the server.

And if rsync is too expensive, a simple alternative requiring only a little
coding and just an anonymous ftp or http server could lower the costs. This
would be for the server to store for each market a set of difference files
sufficient to bring the client up to date from whenever it last updated. Every
5 minutes, it could compute and store the changes in the last 5 minutes. On
the hour, it could delete all the 5-minute files and compute the changes over
the last hour, and at midnight it could delete all the 1-hour files and compute
the differences since midnight yesterday. This only requires the server to
compute one diff every 5 minutes for each TV market it services, plus a service
a bunch of (mostly small) ftp/http file requests.

Then the client just needs to keep around the last midnight file and the last
on-the-hour file in addition to the latest file. When it connects to the host,
it just downloads all the files generated since it last connected. If it
downloaded any on-the-hour diffs, it deletes its local 5-minute file; if it
downoaded any midnight files, it also deletes its local on-the-hour file. Then
it applies the diffs it downloaded in order to its latest file, bringing it up
to date. This shouldn't require too much CPU resources. And the coding should
just require a couple of reasonably simple shell or python scripts.

The whole thing could be made more efficient, too, by using a binary difference
format. It's not like anyone is going to be editing these XMLTV files by hand,
or they're going to be changed by different parties independently, so the
redundancy of diff isn't needed. A binary diff format could contain a
cryptographic hash of the final file, so after application the result could be
verified to be byte-correct. The file itself could just be a sequence of
simple commands to move forward a number of bytes, delete a number of bytes,
and insert a string. That would all take a bit more coding, but it's optional.

--
Peter Schachte We hang the petty thieves and appoint the great
schachte [at] cs ones to public office.
www.cs.mu.oz.au/~schachte/ -- Aesop
Phone: +61 3 8344 1338
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

seven at seven

Jun 20, 2007, 9:41 PM

Post #10 of 140 (2279 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Silly Idea here why not use legal P2P or a modified version of Bittorrent
too legally distribute this information? this should scale indefinably if
everyone shares their information.

I'm sure there are flaws in my idea but just putting it out there too see
what everyone thinks
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythlist at michaelandholly

Jun 20, 2007, 9:49 PM

Post #11 of 140 (2285 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Hmm.. good point.. MySQL with the potential number of hits, and
trying to generate the data every time a request was made could get
"expensive".

On Jun 21, 2007, at 12:33 AM, Peter Schachte wrote:

> Michael Jones wrote:
>> Ok.. if the SQL only is too "myth specific" try this on for
>> size... :-)
>>
>> If the data were in an SQL database, a frontend on the service could
>> EASILY export it to a file, if that's what the user/subscription/
>> preference whatever decided. The point is not the format that is
>> used.. since the contents of the SQL database can be output into
>> almost any format imaginable at the drop of a digital hat (usually
>> measured in milliseconds). Keep the format in most general,
>> manageable format and distribute it as needed.
>
> Sure, there's nothing wrong with using a database, but I think an
> XMLTV file
> would work just as well, and could be much cheaper on server
> resources to
> handle. The scheme you're suggesting requires the server to figure
> out what's
> changed since a certain date for every transaction. If the number
> of requests
> gets high, that could be expensive. It also requires a fair bit of
> coding. I
> would think rsync would be more efficient for the server.
>
> And if rsync is too expensive, a simple alternative requiring only
> a little
> coding and just an anonymous ftp or http server could lower the
> costs. This
> would be for the server to store for each market a set of
> difference files
> sufficient to bring the client up to date from whenever it last
> updated. Every
> 5 minutes, it could compute and store the changes in the last 5
> minutes. On
> the hour, it could delete all the 5-minute files and compute the
> changes over
> the last hour, and at midnight it could delete all the 1-hour files
> and compute
> the differences since midnight yesterday. This only requires the
> server to
> compute one diff every 5 minutes for each TV market it services,
> plus a service
> a bunch of (mostly small) ftp/http file requests.
>
> Then the client just needs to keep around the last midnight file
> and the last
> on-the-hour file in addition to the latest file. When it connects
> to the host,
> it just downloads all the files generated since it last connected.
> If it
> downloaded any on-the-hour diffs, it deletes its local 5-minute
> file; if it
> downoaded any midnight files, it also deletes its local on-the-hour
> file. Then
> it applies the diffs it downloaded in order to its latest file,
> bringing it up
> to date. This shouldn't require too much CPU resources. And the
> coding should
> just require a couple of reasonably simple shell or python scripts.
>
> The whole thing could be made more efficient, too, by using a
> binary difference
> format. It's not like anyone is going to be editing these XMLTV
> files by hand,
> or they're going to be changed by different parties independently,
> so the
> redundancy of diff isn't needed. A binary diff format could contain a
> cryptographic hash of the final file, so after application the
> result could be
> verified to be byte-correct. The file itself could just be a
> sequence of
> simple commands to move forward a number of bytes, delete a number
> of bytes,
> and insert a string. That would all take a bit more coding, but
> it's optional.
>
> --
> Peter Schachte We hang the petty thieves and appoint
> the great
> schachte [at] cs ones to public office.
> www.cs.mu.oz.au/~schachte/ -- Aesop
> Phone: +61 3 8344 1338
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

lists at forevermore

Jun 20, 2007, 10:52 PM

Post #12 of 140 (2278 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

Anthony Giggins wrote:
> Silly Idea here why not use legal P2P or a modified version of Bittorrent
> too legally distribute this information? this should scale indefinably if
> everyone shares their information.

The main reason would be that it would prevent users from customizing
their listings.

Bandwidth isn't really an issue, anyway.

-Chris

Attachments:

signature.asc (0.18 KB)

jra at baylink

Jun 21, 2007, 6:40 AM

Post #13 of 140 (2245 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Wed, Jun 20, 2007 at 03:42:47PM -0700, Yeechang Lee wrote:
> Jay R. Ashworth <jra [at] baylink> says:
> > One of the *most* important characteristics of AIRING data for the
> > PVR market is that you want to [continue to] propagate changes to
> > the audience just as *late* as you possibly can -- 5 minutes before
> > airtime is not unreasonable, nor impossible, if you pick the right
> > transmission architecture.
>
> I inserted a key omitted phrase in the above because, of course,
> lineup changes ought to be propagated ASAP *and on an ongoing basis*.

True, and an even more important pair of observations occurred to me
this morning.

Well, one this morning, and one last night, but I neglected to mention
it.

Given NNTP to moved the schedule data packets around, you can then have
individual Mythboxen participate in the forest, if their users want to,
with a little infrastructural glue; NNTP's real good at that.

Secondly: it becomes possible for *users* to spot late schedule
changes, and disseminate them to whomever has approved that user's
public key as a valid schedule update source -- possibly with
"auto-schedule" as an option. (If Janie sends out a schedule update
about a Scott Bakula appearance, add it to my schedule and record it
automatically.)

> Otherwise, I heartily agree with your larger message about how this
> event could spark an overdue revolution in how these things work in
> the first place.

I, personally, am heartily convinced that USAdian progress on many
fronts has waned since the end of the Cold War. Enemies are good for
you. They keep you on your toes.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 6:42 AM

Post #14 of 140 (2248 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Wed, Jun 20, 2007 at 09:19:53PM -0400, Michael Jones wrote:
> How about this..
>
> Rather than creating a downloadable, encrypted file which must be
> downloaded in its entirety why not create an SQL database which
> tracks the programs on in a specific market. Have the Mythtv Log
> into this database query for that system's market and only update
> records which need updating?

I'm trying as hard as possible to avoid "smart centralized server *that
everyone has to log in to*". IME, that's one of the things that's made
Z2L hard to support.

> This should reduce downloads significantly since the only information
> that needs to be downloaded is new information or information that
> needs to change since the last download.

I don't want any downloads at all. I'm after "just sit around and
catch."

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 6:45 AM

Post #15 of 140 (2248 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Wed, Jun 20, 2007 at 06:31:27PM -0700, Chris Petersen wrote:
> Personally, I'd love to see something like this, since it'd make for a
> relatively cleaner MythTV database, too. So, I guess, don't worry too
> much that any new system done by the devs will at least have input from
> me, and I'd like to aim for something the community can update/maintain
> to enhance/correct show descriptions, etc. I'm fed up with inaccuracies
> in TMS data, tv.com info, etc., and would love to see something at least
> partially driven by the community.

You know, I've just had a blinding flash of the obvious.

The target for a Project to replace TMS is *broadcast automation system
vendors*. Sundance Digital, the makers of FastBreak, and their ilk.

Those systems are the ones that contain the (local station) data we
care about, anyway.

Writing a trigger to extract, translate, and push schedule change data
when it happens should be one programmer-day for each of them. Writing
a standardized spec for how to ship it around, another day.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 6:46 AM

Post #16 of 140 (2243 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Thu, Jun 21, 2007 at 12:52:53PM +1000, Peter Schachte wrote:
> That sounds like a good way to do it, but it's rather myth-specific
> and would require a fair bit of work. You could get most of the
> benefit in a more application-neutral way with much less effort by
> working with XMLTV files. You could just always keep the last XMLTV
> file you downloaded, and then use something like rsync to update it
> before reloading it into myth.

I have to look at XMLTV's schema. Who's that XMLTV guy who just
prariedogged here?

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 6:47 AM

Post #17 of 140 (2241 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Thu, Jun 21, 2007 at 02:33:38PM +1000, Peter Schachte wrote:
> Sure, there's nothing wrong with using a database, but I think an XMLTV file
> would work just as well, and could be much cheaper on server resources to
> handle. The scheme you're suggesting requires the server to figure out what's
> changed since a certain date for every transaction. If the number of requests
> gets high, that could be expensive. It also requires a fair bit of coding. I
> would think rsync would be more efficient for the server.

Sure. But NNTP and "believe the last signed entry you got for the
timeslot/channel" is even lower load. On both ends.

Cheers,
-- jr '<record condition="broken">' a
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jedi at mishnet

Jun 21, 2007, 7:02 AM

Post #18 of 140 (2244 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

> Ok.. if the SQL only is too "myth specific" try this on for size... :-)
>
> If the data were in an SQL database, a frontend on the service could
> EASILY export it to a file, if that's what the user/subscription/
> preference whatever decided. The point is not the format that is
> used.. since the contents of the SQL database can be output into
> almost any format imaginable at the drop of a digital hat (usually
> measured in milliseconds). Keep the format in most general,
> manageable format and distribute it as needed.
>
> Senario:

No. Just export in a consistent well defined interface and let
the individual users sort out their preferences. Possibly split the
data into smaller pieces so that individual channels only get dumped
from the reference copy when their own data changes.

Provide a mysql import utility if needed.

Let the rest fend for themelves.

[deletia]

The solution needs to be scalable in terms of money and manpower.
Simpler and fewer options are better.

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythlist at michaelandholly

Jun 21, 2007, 7:23 AM

Post #19 of 140 (2239 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

True.. consistent and well defined MUST be a given no matter what
happens.

On Jun 21, 2007, at 10:02 AM, jedi [at] mishnet wrote:

>> Ok.. if the SQL only is too "myth specific" try this on for
>> size... :-)
>>
>> If the data were in an SQL database, a frontend on the service could
>> EASILY export it to a file, if that's what the user/subscription/
>> preference whatever decided. The point is not the format that is
>> used.. since the contents of the SQL database can be output into
>> almost any format imaginable at the drop of a digital hat (usually
>> measured in milliseconds). Keep the format in most general,
>> manageable format and distribute it as needed.
>>
>> Senario:
>
> No. Just export in a consistent well defined interface and let
> the individual users sort out their preferences. Possibly split the
> data into smaller pieces so that individual channels only get dumped
> from the reference copy when their own data changes.
>
> Provide a mysql import utility if needed.
>
> Let the rest fend for themelves.
>
> [deletia]
>
> The solution needs to be scalable in terms of money and manpower.
> Simpler and fewer options are better.
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jedi at mishnet

Jun 21, 2007, 9:23 AM

Post #20 of 140 (2246 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

> On Wed, Jun 20, 2007 at 03:42:47PM -0700, Yeechang Lee wrote:
>> Jay R. Ashworth <jra [at] baylink> says:
>> > One of the *most* important characteristics of AIRING data for the
>> > PVR market is that you want to [continue to] propagate changes to
>> > the audience just as *late* as you possibly can -- 5 minutes before
>> > airtime is not unreasonable, nor impossible, if you pick the right
>> > transmission architecture.
>>
>> I inserted a key omitted phrase in the above because, of course,
>> lineup changes ought to be propagated ASAP *and on an ongoing basis*.

Changes can be presented by themselves and picked up by any interested
parties. It could just be a "differential" from the last full dump of
the schedule data.

Take the master copy plus the latest changes.

[deletia]

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 9:34 AM

Post #21 of 140 (2245 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Thu, Jun 21, 2007 at 11:23:42AM -0500, jedi [at] mishnet wrote:
> > On Wed, Jun 20, 2007 at 03:42:47PM -0700, Yeechang Lee wrote:
> >> Jay R. Ashworth <jra [at] baylink> says:
> >> > One of the *most* important characteristics of AIRING data for the
> >> > PVR market is that you want to [continue to] propagate changes to
> >> > the audience just as *late* as you possibly can -- 5 minutes before
> >> > airtime is not unreasonable, nor impossible, if you pick the right
> >> > transmission architecture.
> >>
> >> I inserted a key omitted phrase in the above because, of course,
> >> lineup changes ought to be propagated ASAP *and on an ongoing basis*.
>
> Changes can be presented by themselves and picked up by any interested
> parties. It could just be a "differential" from the last full dump of
> the schedule data.
>
> Take the master copy plus the latest changes.

Well, the *real* question is "how far back/forwards" do you need to
go... but remember how Usenet servers *work*. You can pull, from the
server, as much cached older sked data as you need, in posting order,
and process it. The subject lines can be machineable, so you can
figure out which data you don't already have, too.

That way there *isn't* really a "full dump" or a "master" copy. All
there is is "new items"... which could expire 12 hours after their
airdate, as well.

Leveraging RFC1036 and NNTP to move blocks of airing data seems to have
quite a number of advantages, operationally.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jedi at mishnet

Jun 21, 2007, 11:40 AM

Post #22 of 140 (2235 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Wed, Jun 20, 2007 at 10:52:32PM -0700, Chris Petersen wrote:
> Anthony Giggins wrote:
> > Silly Idea here why not use legal P2P or a modified version of Bittorrent
> > too legally distribute this information? this should scale indefinably if
> > everyone shares their information.
>
> The main reason would be that it would prevent users from customizing
> their listings.

Not really. People could download the listings for the channels
they actually use. A P2P or BitTorrent doesn't change this. Although the
bulk of data may be small enough that being picky might not matter.

For a lot of people, just having access to guide data for the
national channels would be enough. It would not be ideal in my own case
but at least workable.

>
> Bandwidth isn't really an issue, anyway.
>
> -Chris
>

> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jedi at mishnet

Jun 21, 2007, 11:47 AM

Post #23 of 140 (2238 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Thu, Jun 21, 2007 at 12:34:15PM -0400, Jay R. Ashworth wrote:
> On Thu, Jun 21, 2007 at 11:23:42AM -0500, jedi [at] mishnet wrote:
> > > On Wed, Jun 20, 2007 at 03:42:47PM -0700, Yeechang Lee wrote:
> > >> Jay R. Ashworth <jra [at] baylink> says:
> > >> > One of the *most* important characteristics of AIRING data for the
> > >> > PVR market is that you want to [continue to] propagate changes to
> > >> > the audience just as *late* as you possibly can -- 5 minutes before
> > >> > airtime is not unreasonable, nor impossible, if you pick the right
> > >> > transmission architecture.
> > >>
> > >> I inserted a key omitted phrase in the above because, of course,
> > >> lineup changes ought to be propagated ASAP *and on an ongoing basis*.
> >
> > Changes can be presented by themselves and picked up by any interested
> > parties. It could just be a "differential" from the last full dump of
> > the schedule data.
> >
> > Take the master copy plus the latest changes.
>
> Well, the *real* question is "how far back/forwards" do you need to
> go... but remember how Usenet servers *work*. You can pull, from the

Go as far ahead as the data will allow.

Go as far back as storage and processing make prudent.

Applying n+1 differentials to a base copy is just annoying.
There has to be some sane smaller limit that will be set by human
convenience more than technical considerations.

In either case, chunking the data in one week increments would probably
make some sense. "full backups" need to be frequent enough that people don't
want to throw sharp objects at you.

> server, as much cached older sked data as you need, in posting order,
> and process it. The subject lines can be machineable, so you can
> figure out which data you don't already have, too.
>
> That way there *isn't* really a "full dump" or a "master" copy. All
> there is is "new items"... which could expire 12 hours after their
> airdate, as well.
>
> Leveraging RFC1036 and NNTP to move blocks of airing data seems to have
> quite a number of advantages, operationally.

Polices regarding "initial" copies versus "deltas" should be independent
of the transport mechanism.

>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink jra [at] baylink
> Designer The Things I Think RFC 2100
> Ashworth & Associates http://baylink.pitas.com '87 e24
> St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 12:21 PM

Post #24 of 140 (2246 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Thu, Jun 21, 2007 at 01:40:11PM -0500, Kevin Hulse wrote:
> On Wed, Jun 20, 2007 at 10:52:32PM -0700, Chris Petersen wrote:
> > Anthony Giggins wrote:
> > > Silly Idea here why not use legal P2P or a modified version of Bittorrent
> > > too legally distribute this information? this should scale indefinably if
> > > everyone shares their information.
> >
> > The main reason would be that it would prevent users from customizing
> > their listings.
>
> Not really. People could download the listings for the channels
> they actually use. A P2P or BitTorrent doesn't change this. Although the
> bulk of data may be small enough that being picky might not matter.
>
> For a lot of people, just having access to guide data for the
> national channels would be enough. It would not be ideal in my own case
> but at least workable.

Indeed.

Subject: Svc: ABC; Date: 2007-JUN-27; Time: ALL
Subject: Svc: TLC; Date: 2007-JUN-22; Time: 1900-2259

... which raises another good point. The only reasonable way to handle
time is to do it in local, which means you need to know which coast's
feed you're watching for some channels and locations ("8pm, 7 Central")

I'm not *fond* of things tagged with local time, but I suspect the
contortions necessary to do anything else are prohibitive, especially
since the program source services are scheduling in that.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jra at baylink

Jun 21, 2007, 12:23 PM

Post #25 of 140 (2252 views)
Permalink

Re: Mooting architecture for a DataDirect replacement [In reply to]

On Thu, Jun 21, 2007 at 01:47:25PM -0500, Kevin Hulse wrote:
> > Well, the *real* question is "how far back/forwards" do you need to
> > go... but remember how Usenet servers *work*. You can pull, from the
>
> Go as far ahead as the data will allow.
>
> Go as far back as storage and processing make prudent.
>
> Applying n+1 differentials to a base copy is just annoying.
> There has to be some sane smaller limit that will be set by human
> convenience more than technical considerations.

Sure. But I wasn't talking about "differentials", because they imply a
"full", and I don't propose that, either.

Doesn't *anyone* remember Usenet? :-)

> In either case, chunking the data in one week increments would probably
> make some sense. "full backups" need to be frequent enough that people don't
> want to throw sharp objects at you.

Most people will have their machines on and slurping all the time
anyway, so I don't see them *needing* "a full dump".

> > Leveraging RFC1036 and NNTP to move blocks of airing data seems to have
> > quite a number of advantages, operationally.
>
> Polices regarding "initial" copies versus "deltas" should be independent
> of the transport mechanism.

I don't see that 1036 imposes any policies at all, which the competing
proposal seem *to*.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra [at] baylink
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

1 2 3 4 5 6

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads