Mailing List Archive: MythTV: Users

Planning Tuners and Disk I/O for a MythTV Backend System

Index | Next | Previous | View Threaded

beckett.jim at gmail

Feb 5, 2010, 11:46 PM

Post #1 of 10 (1066 views)
Permalink

Planning Tuners and Disk I/O for a MythTV Backend System

Hey all,

Since I started using MythTV a few years ago, I have wondered how you
would be able to determine how many tuners, and disks, would be optimal
for a MythTV setup.

I'm going to build a bigger-badder system some day, and I wanted to
think it through, and get some opinions about it.

Please feel free to set me straight on anything that I have wrong in my
thought process.

I came up with the following (as I understand it):

SATA1 systems can perform a sustained disk write at ~72 MB/second.
SATA2 systems can perform a sustained disk write at ~144 MB/second.

A full transport stream (multiplex, mux) is limited to 38.8 MB/sec data
rate.
A full transport stream makes up one entire tuning frequency as tuned by
a physical ATSC tuner.

2 full HD Channels can be carried by a transport stream (19.4 MB/sec
each) maximum.
1 full HD Channel, and 1 (or more) SD channels can be on a single
transport stream (total amount of data never exceeding the 38.8 MB/sec
limit)
Several SD channels can be transmitted on a single transport stream
(again, total amount of data not exceeding the 38.8 MB/sec limit)

That leads me to believe that the practical number of physical tuners
for a PVR system can be limited by disk IO throughput, and should be an
important consideration when designing, and using the system.

Each physical tuner is assigned a number of virtual tuners so that it
may capture some, or all, of the data transmitted within a given
transport stream, but the total is limited to the 38.8 MB/s maximum of
the transport stream, so virtual tuners don't figure into the equation
for disk IO.

I imagine it would be rare in real world situations to be recording the
entire contents of multiple data streams, and that my concern is
probably not an issue for most people, but could it happen?

The data above suggests to me that some configurations could have some
issues, so I charted it out:

Single Disk Storage

Number of disks: Tuners Maximum Concurrent Recording Capablility

Single-Disk (SATA1) 1 Can record 1 transport stream (38.8 MB/s
Maximum data transfer possible. )

Single-Disk (SATA1) 2 Can record 2 transport streams (77.6
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Single-Disk (SATA1) 3 Can record 3 transport streams (116.4
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Single-Disk (SATA1) 4 Can record 4 transport streams (155.2
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Single-Disk (SATA2) 1 Can record 1 transport stream (38.8 MB/s
Maximum data transfer possible. )

Single-Disk (SATA2) 2 Can record 2 transport streams (77.6
MB/s Maximum data transfer possible. )

Single-Disk (SATA2) 3 Can record 3 transport streams (116.4
MB/s Maximum data transfer possible. )

Single-Disk (SATA2) 4 Can record 4 transport streams (155.2
MB/s Maximum data transfer possible. )
^** This has potential to over-run the disk IO write capability **

Multiple Disk Storage (Balancing the Recording Activity Based on IO)

Number of disks: Tuners Maximum Concurrent Recording Capablility

Double-Disk (SATA1) 1 Can record 1 transport stream (38.8 MB/s
Maximum data transfer possible. )

Double-Disk (SATA1) 2 Can record 2 transport streams (77.6
MB/s Maximum data transfer possible.)

Double-Disk (SATA1) 3 Can record 3 transport streams (116.4
MB/s Maximum data transfer possible.)

Double-Disk (SATA1) 4 Can record 4 transport streams (155.2
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Double-Disk (SATA1) 5 Can record 5 transport streams (194 MB/s
Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Double-Disk (SATA1) 6 Can record 6 transport streams (232.8
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Double-Disk (SATA1) 7 Can record 7 transport streams (271.6
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Double-Disk (SATA1) 8 Can record 8 transport streams (310.4
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Double-Disk (SATA2) 1 Can record 1 transport stream (38.8 MB/s
Maximum data transfer possible. )

Double-Disk (SATA2) 2 Can record 2 transport streams (77.6
MB/s Maximum data transfer possible. )

Double-Disk (SATA2) 3 Can record 3 transport streams (116.4
MB/s Maximum data transfer possible. )

Double-Disk (SATA2) 4 Can record 4 transport streams (155.2
MB/s Maximum data transfer possible. )

Double-Disk (SATA2) 5 Can record 5 transport streams (194 MB/s
Maximum data transfer possible.)

Double-Disk (SATA2) 6 Can record 6 transport streams (232.8
MB/s Maximum data transfer possible.)

Double-Disk (SATA2) 7 Can record 7 transport streams (271.6
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

Double-Disk (SATA2) 8 Can record 8 transport streams (310.4
MB/s Maximum data transfer possible.)
^** This has potential to over-run the disk IO write capability **

I am ignoring the ring buffer, because I am not sure it would be
reliable if the amount of incoming data exceeds the amount of data the
disk(s) can record for a sustained period of time.

I did not take this any further because it seemed pointless to do so,
but I am serious about the decisions for when I build the next backend...

Taking all this into consideration, is there a mechanism in MythTV that
recognizes when disk IO limits are be likely to exceeded? If so, does it
adjust the recording schedule, or perhaps, show a conflict?

If not, then I suppose it would be something that might interest one of
the developers. (I'm not up to speed in C/C++) My thought is that it
could make the scheduler even smarter if there is a way to
determine/predict the likely data recording rate, based on the number of
recordings scheduled, and how much system IO resources would be likely
to be available at any given time.

- Jim
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

raymond at wagnerrp

Feb 6, 2010, 12:04 AM

Post #2 of 10 (1033 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

On 2/6/2010 02:46, Jim Beckett wrote:
> SATA1 systems can perform a sustained disk write at ~72 MB/second.
> SATA2 systems can perform a sustained disk write at ~144 MB/second.

No. SATA1 and 2 are 150MB/s and 300MB/s, respectively. However that is
the interface bandwidth. It has no bearing on the physical speed of the
disk. A modern high density disk will average in excess of 100MB/s.
http://www.tomshardware.com/charts/2009-3.5-desktop-hard-drive-charts/h2benchw-3.12-Avg-Write-Throughput,1013.html

> A full transport stream (multiplex, mux) is limited to 38.8 MB/sec
> data rate.

A full QAM256 multiplex will be ~38mbps, while an ATSC multiplex will be
~19mbps. Note the big 'B' and small 'b'. 38mbps means 4.75MB/s.

> That leads me to believe that the practical number of physical tuners
> for a PVR system can be limited by disk IO throughput, and should be
> an important consideration when designing, and using the system.

No. You can probably put a dozen digital tuners in a system before you
might have to worry about exceeding the throughput of a single modern
hard drive.

> Taking all this into consideration, is there a mechanism in MythTV
> that recognizes when disk IO limits are be likely to exceeded? If so,
> does it adjust the recording schedule, or perhaps, show a conflict?

No. You're expected to provide a system capable of the necessary
throughput for all supplied tuners. As long as you're not using
framegrabbers, this should not be a problem.

> If not, then I suppose it would be something that might interest one
> of the developers. (I'm not up to speed in C/C++) My thought is that
> it could make the scheduler even smarter if there is a way to
> determine/predict the likely data recording rate, based on the number
> of recordings scheduled, and how much system IO resources would be
> likely to be available at any given time.

MythTV will estimate the storage space a recording will take, and
auto-expire shows accordingly. It will also perform load balancing
across multiple disks if you add them to the storage group as
independent drives.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

beckett.jim at gmail

Feb 6, 2010, 1:45 AM

Post #3 of 10 (1026 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

Raymond,

Wow, I sure struck out on that post... It's comical that so much work
went into chasing erroneous data... Doh!

Thanks for straightening me out.

- Jim

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

dennis.cartier at gmail

Feb 6, 2010, 5:34 AM

Post #4 of 10 (1015 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

On Sat, Feb 6, 2010 at 3:04 AM, Raymond Wagner <raymond [at] wagnerrp> wrote:

> On 2/6/2010 02:46, Jim Beckett wrote:
>
>> SATA1 systems can perform a sustained disk write at ~72 MB/second.
>> SATA2 systems can perform a sustained disk write at ~144 MB/second.
>>
>
> No. SATA1 and 2 are 150MB/s and 300MB/s, respectively. However that is the
> interface bandwidth. It has no bearing on the physical speed of the disk. A
> modern high density disk will average in excess of 100MB/s.
>
> http://www.tomshardware.com/charts/2009-3.5-desktop-hard-drive-charts/h2benchw-3.12-Avg-Write-Throughput,1013.html
>
>
> A full transport stream (multiplex, mux) is limited to 38.8 MB/sec data
>> rate.
>>
>
> A full QAM256 multiplex will be ~38mbps, while an ATSC multiplex will be
> ~19mbps. Note the big 'B' and small 'b'. 38mbps means 4.75MB/s.
>
>
> That leads me to believe that the practical number of physical tuners for
>> a PVR system can be limited by disk IO throughput, and should be an
>> important consideration when designing, and using the system.
>>
>
> No. You can probably put a dozen digital tuners in a system before you
> might have to worry about exceeding the throughput of a single modern hard
> drive.
>
>
> Taking all this into consideration, is there a mechanism in MythTV that
>> recognizes when disk IO limits are be likely to exceeded? If so, does it
>> adjust the recording schedule, or perhaps, show a conflict?
>>
>
> No. You're expected to provide a system capable of the necessary throughput
> for all supplied tuners. As long as you're not using framegrabbers, this
> should not be a problem.
>
>
> If not, then I suppose it would be something that might interest one of
>> the developers. (I'm not up to speed in C/C++) My thought is that it could
>> make the scheduler even smarter if there is a way to determine/predict the
>> likely data recording rate, based on the number of recordings scheduled, and
>> how much system IO resources would be likely to be available at any given
>> time.
>>
>
> MythTV will estimate the storage space a recording will take, and
> auto-expire shows accordingly. It will also perform load balancing across
> multiple disks if you add them to the storage group as independent drives.
>
>
Just to provide a real world example to Raymond's post, my master backend
uses 3 HD Homeruns (6 tuners) and on occasion records 6 OTA HD streams
simultaneously. While the backend is recording, there may be up to 2 streams
being watched (1 on the master backend itself and 1 on a frontend) and 2
streams being commercial flagged on a remote backend (my desktop PC). The
disk I/O is spread across 2 drives on the master.

Amazingly, my master backend is an Atom 330 dual core @1.6Ghz, so not a
speedy system by any definition.

What I have found to be key, is as Raymond says, use a capture method that
only has to record the stream (E.g HD Homerun) and use a file system that
can handle huge files adeptly and can be defragged easily (XFS with 500MB
preallocation and nightly online defrag). Also placing the recordings on
dedicated drives (not the OS drive) and sending the commercial flagging off
to a brawnier system also helped keep things running smooth.

I tried a smaller version of the above originally (4 tuners and 2 drives -
including the OS drive) running EXT4 with the master doing the commercial
flagging but it ended badly.

Dennis

ke-myth at retriever

Feb 7, 2010, 8:08 AM

Post #5 of 10 (948 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

On Sat, Feb 06, 2010 at 01:45:31AM -0800, Jim Beckett wrote:
> Raymond,
>
> Wow, I sure struck out on that post... It's comical that so much work
> went into chasing erroneous data... Doh!
>
> Thanks for straightening me out.

Sounds like you've got the bandwidth thing straightened out.
I would encourage you to get lots of space. I *thought*
a terabyte would be sufficient. Technically, it is but
I hate having to worry about going over.

Also, not sure about other peoples' opinions on the subject,
but Linux' software RAID has really worked well for me.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

beckett.jim at gmail

Feb 7, 2010, 6:00 PM

Post #6 of 10 (925 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

Kevin Bailey wrote:
> I would encourage you to get lots of space. I *thought*
> a terabyte would be sufficient. Technically, it is but
> I hate having to worry about going over.
I have 3- 500GB drives, and I have to say that although I am getting
close to filling that up, it's taken a couple of months.

I find it's way more TV than I can personally keep up with. That being
said, I am thinking about setting up some sort of user transcode job for
the recordings that I consider keepers, and reduce them down to DVD
quality to reclaim some of my currently used space.

But then again, a lot of what is stored on the disks are things I'll
probably never watch again, and should auto-expire (hasn't happened
yet), so it hasn't been a priority for me.

-Jim

nick.rout at gmail

Feb 7, 2010, 6:30 PM

Post #7 of 10 (912 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

On Mon, Feb 8, 2010 at 3:00 PM, Jim Beckett <beckett.jim [at] gmail> wrote:
> Kevin Bailey wrote:
>
> I would encourage you to get lots of space. I *thought*
> a terabyte would be sufficient. Technically, it is but
> I hate having to worry about going over.
>
> I have 3- 500GB drives, and I have to say that although I am getting close
> to filling that up, it's taken a couple of months.
>
> I find it's way more TV than I can personally keep up with. That being said,
> I am thinking about setting up some sort of user transcode job for the
> recordings that I consider keepers, and reduce them down to DVD quality to
> reclaim some of my currently used space.
>
> But then again, a lot of what is stored on the disks are things I'll
> probably never watch again, and should auto-expire (hasn't happened yet), so
> it hasn't been a priority for me.

Yeah I have 1 TB for recordings and about 1.4 for 'video' files that
I've acquired in various places. Its more than I can ever watch, but
the kids seem to like watching endless southpark, family guy, simpsons
and futurama, all of which seem to be filling it up...

Annoyingly the videos files are scattered over three hard drives in 3
machines, plus a couple of externals. I really need to get a decent
NAS and simplify it all...(and find the power supply for that external
that is lying dead in the server room)
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

lists at foxhill

Feb 9, 2010, 2:42 AM

Post #8 of 10 (839 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

In article <8e8b35e41002060534w5ae22710l75450e1d5c92640c [at] mail>,
Dennis Cartier wrote:
> What I have found to be key, is as Raymond says, use a capture method that
> only has to record the stream (E.g HD Homerun) and use a file system that
> can handle huge files adeptly and can be defragged easily (XFS with 500MB
> preallocation and nightly online defrag). Also placing the recordings on
> dedicated drives (not the OS drive) and sending the commercial flagging off
> to a brawnier system also helped keep things running smooth.

I'm currently recording UK DVB-T to my OS drive, but want to attach a 1TB
external drive via SATA ASAP. Thanks for the tip regards XFS.

Am I better just copying existing stuff to the drive and mounting it where
the recordings currently are, or is trying to figure out storage groups
better?

Ian

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

blammo.doh at gmail

Feb 9, 2010, 11:03 AM

Post #9 of 10 (817 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

On Sat, Feb 6, 2010 at 1:45 AM, Jim Beckett <beckett.jim [at] gmail> wrote:
> Wow, I sure struck out on that post... It's comical that so much work went
> into chasing erroneous data... Doh!

You're not the first person to get bit by the little/big B/M/K/etc..
I've had people with far more acronyms after their name than I, make
very expensive mistakes by getting that wrong.

There are no stupid questions, just unasked ones... You've shown an
amazing amount of dedication and research to better understand why
pieces of the MythTV design should be done in one way or the other.

In addition, you've now helped create hundreds of copies (thanks to
unknown quantity of archives of this mailing list) of useful
information that other users will benefit from. Not that you needed
my "Atta-boy", but just wanted to burn part of some virtual paper to
say... thanks.. :)
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

beckett.jim at gmail

Feb 9, 2010, 12:44 PM

Post #10 of 10 (810 views)
Permalink

Re: Planning Tuners and Disk I/O for a MythTV Backend System [In reply to]

Blammo wrote:
> In addition, you've now helped create hundreds of copies (thanks to
> unknown quantity of archives of this mailing list) of useful
> information that other users will benefit from. Not that you needed
> my "Atta-boy", but just wanted to burn part of some virtual paper to
> say... thanks.. :)

Thanks for the 'Atta-boy', Blammo! I appreciate it.

Hopefully, I won't be sent any invoices for all the space, and bandwidth
this this thread has consumed! ;)

There has been some great discussion on the topic(s).

I have to say, that in coming from the Microsoft way of doing things, I
was spoiled when I installed Mythbuntu (8.04, I think, maybe an earlier
version) on my system a couple of years ago.

(I replaced XP, and GBPVR, and was pretty happy with it.)

IMO, Mythbuntu did way too good of a job of automating the process for
me to get any useful troubleshooting experience from it.

The only thing I remember having to figure out was that I needed to
download and build the firmware for my HD cards, and then dump it in
/lib/firmware.

Everything else, as I recall, just worked.

I did a fresh install of Mythbuntu 9.10 a couple of months ago, which
has been a lot more difficult to get up, and running, at an acceptable
F.A.F.

The good part about this is that it has forced me to learn a lot more
about how GNU/Linux and MythTV work, and that is VERY valuable
experience in my eyes.

Just to prove what a glutton for punishment I am, I'm running Debian
Squeeze/Sid with unstable/testing packages on my workstation!

There seems to be a weekly gotchas that I have to try to work through!
(It's kind of fun.)

All, in all, it is great experience. Who knows, someday I might just get
good at it!

- Jim

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads