Mailing List Archive: MythTV: Dev

[mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling

Index | Next | Previous | View Threaded

f-myth-users at media

Dec 8, 2006, 7:02 AM

Post #1 of 10 (1190 views)
Permalink

[mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling

> Date: Fri, 8 Dec 2006 16:42:05 +0200
> From: "osma ahvenlampi" <oa [at] iki>

> On 12/8/06, Daniel Kristjansson <danielk [at] cuymedia> wrote:
> > Simple math, lets say you have 4 ATSC recorders which are
> > recording, and 3 frontends are watching 3 other streams off
> > your disk, and your disk read/write speed is 21 MB/s. Since
> > each stream is about 3 MB/s, adding another frontend process
> > or starting a commercial flagging or transcode process would
> > bring us over the edge.

> This is why I referred to CFQ scheduling. With CFQ enabled and the
> jobs being niced, they won't be getting a whole lot of I/O bandwidth
> in this situation, so I really doubt that will be the thing taking you
> over the edge -- it'll be the too-many-recordings-and-playbacks alone
> that would be the real contributors, as you said (in the part I
> clipped).

No I/O scheduling setting did a damn at preventing trashed recordings
when I was debugging issues of MySQL being too disk-intensive a few
months ago. I tried 'em all. This was in Ubunto Breezy; later
releases' kernels might implement this better (though I -was- seeing
changes, just not good-enough changes). But since you can't specify
"thou shalt run a kernel where I/O scheduling must work in
such-and-such a way", you can't depend on it.

What -did- work (sort of) was changing some MySQL parameters (I'll
send followup mail at some point for the archives), and even so, I
have to be cautious not to have too many processes flailing on the
MBE's disk (e.g., via NFS mounts from the SBE/FE, via other programs
moving things around as part of my automation, via commflagging, etc.)
I can force a failure (e.g., dropped video from encoders if the
scheduler decides to run) by loading up the I/O system enough, and
while I've got it so my normal usage won't do it, one part of that is
strictly limiting commflagging to one local-to-the-MBE job and one
local-to-the-SBE job (whose 100baseT network connection acts as a
natural throttle on the demands it can make of the MBE's disk).

I've had enough issues with glitched recordings due to other things
using the disk (even though I'm recording on fairly beefy machines
with modern disks, etc) that, if your patch didn't have an OFF switch,
I wouldn't upgrade. There are too many variables in how people set up
their Myths (e.g., hardware, system architecture, versions of every
single piece of software under the sun) for you to be assured that
your patch will always do the right thing. It might, and for those
for whom it does, fine. But you can't claim "it works so well it
won't need an OFF switch" because you haven't seen everyone's
configurations, and there will always be new ones in the future.
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 8, 2006, 2:08 PM

Post #2 of 10 (1120 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/8/06, f-myth-users [at] media <f-myth-users [at] media> wrote:
> No I/O scheduling setting did a damn at preventing trashed recordings
> when I was debugging issues of MySQL being too disk-intensive a few
> months ago. I tried 'em all.

Did you try niceing MySQL? See my earlier message.

> But since you can't specify
> "thou shalt run a kernel where I/O scheduling must work in
> such-and-such a way", you can't depend on it.

In the same vein, one might also argue that since you can't specify
users must have hard disks faster than the combined bandwidth of their
recorders, you can't depend on that either -- but if you don't,
recordings will break. So you HAVE to make assumptions, it's just a
question what assumptions are good ones.

> What -did- work (sort of) was changing some MySQL parameters (I'll
> send followup mail at some point for the archives), and even so, I
> have to be cautious not to have too many processes flailing on the
> MBE's disk (e.g., via NFS mounts from the SBE/FE, via other programs
> moving things around as part of my automation, via commflagging, etc.)

All of this would make a good argument for using an embedded libsqlite
instead of MySQL, since then the parameters would be in MythTV's
control.. I don't want to pick a fight by arguing that, though :)

> for whom it does, fine. But you can't claim "it works so well it
> won't need an OFF switch" because you haven't seen everyone's
> configurations, and there will always be new ones in the future.

Sure. So is that "off switch" a ./configure param, a mythtv-setup
param, or what? Does the old settings need to stay too? Is everyone
really happy with the ever-increasing amount of config params and how
they create an exponentially more complicated universe of different
combinations?

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

f-myth-users at media

Dec 8, 2006, 2:42 PM

Post #3 of 10 (1111 views)
Permalink

[mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

> Date: Sat, 9 Dec 2006 00:08:07 +0200
> From: "osma ahvenlampi" <oa [at] iki>

> On 12/8/06, f-myth-users [at] media <f-myth-users [at] media> wrote:
> > No I/O scheduling setting did a damn at preventing trashed recordings
> > when I was debugging issues of MySQL being too disk-intensive a few
> > months ago. I tried 'em all.

> Did you try niceing MySQL? See my earlier message.

If you're talking CPU niceness, it doesn't matter. MySQL uses a tiny
amount of CPU while scheduling, but absolutely thrashes the disk
unless it has huge caches. (I don't know how much of that is writing,
but if any substantial amount of it is, then ionice won't help since
ionice [according to a recent post] doesn't nice writes at all, and
the encoders are writing too. I don't recall if Breezy fully
supported ionice or only the I/O scheduling stuff, but I wasn't
about to go to the pain of revving the kernel [.hence -everything- on
-both- boxes, e.g., ivtv versions, lirc, etc etc] just to test it.)

> > But since you can't specify
> > "thou shalt run a kernel where I/O scheduling must work in
> > such-and-such a way", you can't depend on it.

> In the same vein, one might also argue that since you can't specify
> users must have hard disks faster than the combined bandwidth of their
> recorders, you can't depend on that either -- but if you don't,
> recordings will break. So you HAVE to make assumptions, it's just a
> question what assumptions are good ones.

Don't create strawman arguments. The disks -aren't- slower than the
recorders (not even 6 SD recorders), but they -are- slower if the
heads are being thrashed by another process, and even a -single-
recording will be trashed if the heads are thrashed enough. I saw
that behavior when the scheduler ran when using MySQL's default
settings for this distribution (Ubuntu Breezy), and the scheduler
runs -every- time a recording ends. That means that a recording that
ended while another was ongoing cause the ongoing one to drop many
seconds at a time. Not always, but often.

Note, btw, that even putting MySQL's DB on a separate spindle -and-
IDE controller did -not- help (enough); you'd think that recordings on
hda and the DB on hdc would separate out head motion and other I/O,
but it didn't. It helped somewhat (raising the number of simultaneous
streams and/or commflagging load that could typically be tolerated
before scheduler activity trashed encoding streams), but not by a lot.

And you're assuming that everybody runs Myth under particular Linux
kernels. They don't even all use Linux! So assuming that they can
exercise the sort of fine-grained control over the I/O subsystem that
-might- be possible under the most bleeding-edge Linux kernel isn't
something that can be applied to the entire user base. And if you
can't apply it everywhere, it needs an off switch.

So -my- assumptions are, "Any automatic system that tries to increase
I/O load on the MBE without my say-so is likely to cause trashed recordings."

> > What -did- work (sort of) was changing some MySQL parameters (I'll
> > send followup mail at some point for the archives), and even so, I
> > have to be cautious not to have too many processes flailing on the
> > MBE's disk (e.g., via NFS mounts from the SBE/FE, via other programs
> > moving things around as part of my automation, via commflagging, etc.)

> All of this would make a good argument for using an embedded libsqlite
> instead of MySQL, since then the parameters would be in MythTV's
> control.. I don't want to pick a fight by arguing that, though :)

I have no opinion on that, but it hasn't gotten a good reception on
the list when others have suggested it. (Or maybe it's that it just
wasn't up to the task; I don't recall.)

> > for whom it does, fine. But you can't claim "it works so well it
> > won't need an OFF switch" because you haven't seen everyone's
> > configurations, and there will always be new ones in the future.

> Sure. So is that "off switch" a ./configure param, a mythtv-setup
> param, or what? Does the old settings need to stay too? Is everyone
> really happy with the ever-increasing amount of config params and how
> they create an exponentially more complicated universe of different
> combinations?

Some parameter that can be set without recompiling. Lots of people
use prepackaged distributions; having to recompile just to turn this
off would shaft them. So presumably a mythtv-setup value or at least
a flag in the DB that can be set via a MySQL statement.
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

danielk at cuymedia

Dec 8, 2006, 2:49 PM

Post #4 of 10 (1112 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On Fri, 2006-12-08 at 23:57 +0200, osma ahvenlampi wrote:
> On 12/8/06, Daniel Kristjansson <danielk [at] cuymedia> wrote:
> > It's the seeks to the DB for writing the keyframe DB entries
> > which seems to throw the most randomness into the elevator.
>
> mysql is most likely running at the same nice value (0) as the backend
> writing the recordings. In this setting, it would most likely be the
> best approach to run recordings at (unnice) -5, mysql at 0, playback
> at (nice) +5 and transcodings and commflags at (nice) +10.

For most recorders people use the nice value of the backend and
mysql is the higher of the two since we need to write to both
the db and the filesystem when writing a file. I made this
same mistake before the 0.19 release when trying to maximise
the number of recorders you could use simultaneously with
a single disk to three. Lowering the recorder's niceness below
the mysql niceness has very little effect for MPEG recorders
because we need to write the keyframe rows to the DB once in
a while. (The NVP handles this much better, but fewer and fewer
people still use framegrabbers with the low cost of PVR-x50 and
digital cards these days.)

We also don't want playback to run at greater niceness than
recording. Playback has much higher real-time requirements.

> > If this is disk scheduler dependant how does this work on Macs,
> > the BSDs and MS Windows?
> If some OS can not balance its I/O well, that's their problem. Users
> will have to compensate by providing more I/O capacity. Or think of it
> this way -- Linux I/O scheduler can probably let the user get away
> with overcommitting their available I/O.

But the point of this patch is to make this scaling automatic.
If it in fact is just tuned for one person's system it doesn't
make much sense to apply it. If it can scale the number and
run speed of commercial flagging and transcoding processes for
multiple people then it becomes a very nice contribution.

> Networking can at least in theory be solved with the same principles
> using QoS settings on per-connection basis.

If you are volunteering to make MythTV configure QoS settings
for the 5-6 OS MythTV runs on you are a better man than I. :)

I think there must be a solution which doesn't require mucking
this much with all the different operating systems on which
MythTV runs.

> Although what's more
> likely to be a problem is that "network reads", which in many cases
> with Myth are in fact nfs/smb clients will cause disk access with the
> priority level of the nfs/smb daemon (and see above regarding nice
> levels..) -- to solve that either those daemons would have to be
> reprioritised or myth processes would in fact need to work as file
> servers (http on custom ports might do the trick, sort of like UPnP).

Classic priority inversion? I think the only way around this
without doing the scaling within MythTV would be to write a
MythTV disk elevator algorithm for the various operating systems
which gives transcoding and commercial flagging lower priority.

There has to be a better way.

> Or you could solve it the way Internet is usually solved -- brute
> force and more capacity than is going to be needed for the job at hand

But transcoding and commercial flagging are not real-time processes,
we should be able to run them when recording/playback is not
happening, or better yet use only the disk/cpu/network resources
that are currently going unused.

I would think that monitoring the buffer fill on recording and
playback processes would be a good enough metric to be control
the throttling of transcode & commercial flagging processes.
Something simple could probably be used to control the number
of transcode and commercial flagging processes we start as well.
Perhaps a simple count of the number of processes using various
resources and the number of processes using that resource which
caused problems in the past could be kept and used to limit the
number of processes we start up, with throttling kicking in when
an unexpected frontend connection arrives or a recording is
scheduled which begins before a running transcode or commercial
flagging process ends.

Ok, these two solutions aren't really simple when it comes to
filling in the blanks and engineering them into a working
system. But I think these two could address all the issues
no matter which of the big three bottlenecks are the problem.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 9, 2006, 3:35 AM

Post #5 of 10 (1122 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

(I'm new on this list, so please excuse a stupid question - would you
guys rather prefer me to reply to individual messages one by one, or
merge all together? I'm going to do the former now since it's a tiny
bit easier, but if getting many messages is a bother, just ask me to
switch)

On 12/9/06, f-myth-users [at] media <f-myth-users [at] media> wrote:
> > Did you try niceing MySQL? See my earlier message.
>
> If you're talking CPU niceness, it doesn't matter. MySQL uses a tiny
> amount of CPU while scheduling, but absolutely thrashes the disk
> unless it has huge caches.

My point exactly. The CFQ IO elevator *will consider "CPU nice"
(actually, the PROCESS nice value) when making IO operation scheduling
decisions*. It just happens that the value conveniently affects both
CPU and I/O priority, but for the purposes of discussing disk
trashing, the latter is more important.

> the encoders are writing too. I don't recall if Breezy fully
> supported ionice or only the I/O scheduling stuff, but I wasn't

you don't *need* to use ionice in order to benefit. I also thought you
did, but ionice is extra - it allows setting the priority class of a
process to "idle time only" or "realtime priority" (for root processes
only), but if you don't, CFQ will consider the process nice as an
equivalent of its I/O priority too.

> Don't create strawman arguments. The disks -aren't- slower than the
> recorders (not even 6 SD recorders), but they -are- slower if the
> heads are being thrashed by another process, and even a -single-

It's not my intention to create strawman arguments, or to make this
into fight. We are in complete agreement - it is indeed possible to
trash disk well enough to destroy ability to write fast enough. That's
where the OS can help by prioritising one class of access, and as an
application, MythTV can benefit if the whole system is considered.

In my day job, I'm working with applications that require hundreds of
megabits of network bandwidth, tens of thousands of simultaneous users
and thousands of persistent I/O transactions per second. Not saying
this to brag about it -- only as an indication that I've needed to
learn quite a bit of this kind of problems that I feel might be
applied here too.

> Note, btw, that even putting MySQL's DB on a separate spindle -and-
> IDE controller did -not- help (enough); you'd think that recordings on
> hda and the DB on hdc would separate out head motion and other I/O,
> but it didn't. It helped somewhat (raising the number of simultaneous
> streams and/or commflagging load that could typically be tolerated
> before scheduler activity trashed encoding streams), but not by a lot.

That's funny. Sounds more like buffer trashing than disk trashing
problem, actually. Might be interesting to look at the iostats of that
system, but I doubt that would be welcome on the list?

> And you're assuming that everybody runs Myth under particular Linux
> kernels. They don't even all use Linux! So assuming that they can

I'm not -- I'm just not assuming I can make it work for all those
other systems :)

> exercise the sort of fine-grained control over the I/O subsystem that
> -might- be possible under the most bleeding-edge Linux kernel isn't
> something that can be applied to the entire user base. And if you
> can't apply it everywhere, it needs an off switch.

I did split the patch to multiple parts - first one just fixing a bug
in the current jobqueue system, second one introducing the "smarts"
(this one I'm going to be glad to tune so that as few people as
possible would need to consider doing anything to default settings),
and a third one to actually remove the settings (which *I* think could
be ultimately removed). I'm not assuming all of those will be applied
immediately, but I made them known. What more could I do to contribute
here?

> So -my- assumptions are, "Any automatic system that tries to increase
> I/O load on the MBE without my say-so is likely to cause trashed recordings."

I'd like to try to overcome that assumption, if you'd like to help me
understand those circumstances and work with me to find solutions.
Have you reviewed the code? I am in fact trying to *reduce* the I/O
load by not starting competing jobs in situations where the system
indicates it has no spare capacity available, so I'm somewhat
surprised by what I perceive to be a rather hostile attitude towards
an attempt to make a benign contribution. Perhaps that's just because
I'm new here?

> Some parameter that can be set without recompiling. Lots of people
> use prepackaged distributions; having to recompile just to turn this
> off would shaft them. So presumably a mythtv-setup value or at least
> a flag in the DB that can be set via a MySQL statement.

I remember when I first tried MythTV over two years ago (I've been an
almost-daily user since), the number of parameters presented by
mythtv-setup and the settings section of mythfrontend was rather
overwhelming. That's my other wish here - to make it less intimidating
to try the system by either removing options where smart choices can
be made automatically, or by at least presenting them only after the
initial hurdle of getting basic functionality set up is over. But for
now, simply not applying the last part of the patch should avoid
anyone getting hurt by it.

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 9, 2006, 3:57 AM

Post #6 of 10 (1116 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/9/06, Daniel Kristjansson <danielk [at] cuymedia> wrote:
> On Fri, 2006-12-08 at 23:57 +0200, osma ahvenlampi wrote:
> > mysql is most likely running at the same nice value (0) as the backend
> > writing the recordings. In this setting, it would most likely be the
> > best approach to run recordings at (unnice) -5, mysql at 0, playback
> > at (nice) +5 and transcodings and commflags at (nice) +10.
>
> For most recorders people use the nice value of the backend and
> mysql is the higher of the two since we need to write to both
> the db and the filesystem when writing a file. I made this

I'm not sure I understood this. Which is less nice of the two? On my
backend, which also doubles as my primary frontend, mythbackend and
mysqld are both nice 0 processes. I haven't tuned that since my two
DVB tuners and generally not more than one simultaneous playback has
never become an I/O problem.

> same mistake before the 0.19 release when trying to maximise
> the number of recorders you could use simultaneously with
> a single disk to three. Lowering the recorder's niceness below
> the mysql niceness has very little effect for MPEG recorders
> because we need to write the keyframe rows to the DB once in
> a while. (The NVP handles this much better, but fewer and fewer
> people still use framegrabbers with the low cost of PVR-x50 and
> digital cards these days.)

But if mysql's buffer flushes are delayed a bit due to recording
buffer flushes, nothing is lost - and mysql writes much less. That's
why I suggested the recorder might benefit from being less nice. Also,
btw, might be a good reason to recommend (or even try to create)
innodb tables instead of myisam -- innodb is better at writing through
buffers.

> We also don't want playback to run at greater niceness than
> recording. Playback has much higher real-time requirements.

Well - is the priority to ensure correctly recorded programs, or to
avoid playback glitches? A recording error will cause a playback error
anyway - again, that's why felt it might be best if recording was the
least nice process.

> But the point of this patch is to make this scaling automatic.
> If it in fact is just tuned for one person's system it doesn't
> make much sense to apply it. If it can scale the number and
> run speed of commercial flagging and transcoding processes for
> multiple people then it becomes a very nice contribution.

I would venture to guess that people running their backends on Linux
2.6 kernels are in the overwhelming majority amongst the user base.
I'd be happy to try to work this out to support BSD, OSX and even
Windows if someone could point me to examples of how to read system
stats out of these systems and volunteer to test patches. Does the
backend even run on Windows?

> > Networking can at least in theory be solved with the same principles
> > using QoS settings on per-connection basis.
>
> If you are volunteering to make MythTV configure QoS settings
> for the 5-6 OS MythTV runs on you are a better man than I. :)

I did say "in theory" :)

> I think there must be a solution which doesn't require mucking
> this much with all the different operating systems on which
> MythTV runs.

The only thing this really requires is a way to read "how busy is the
CPU", and information on whether that busyness includes I/O wait or
not. Like I wrote in my previous message, under these circumstances
the patch will *reduce* competition for resources. Unfortunately,
getting even that much info is entirely OS-specific. I was thinking of
digging into libgtop to see how it's done on most systems. Which would
be more preferable, linking against that or just using it as an
example for a re-implementation?

> > Although what's more
> > likely to be a problem is that "network reads", which in many cases
> > with Myth are in fact nfs/smb clients will cause disk access with the
> > priority level of the nfs/smb daemon (and see above regarding nice
> > levels..) -- to solve that either those daemons would have to be
> > reprioritised or myth processes would in fact need to work as file
> > servers (http on custom ports might do the trick, sort of like UPnP).
>
> Classic priority inversion? I think the only way around this
> without doing the scaling within MythTV would be to write a
> MythTV disk elevator algorithm for the various operating systems
> which gives transcoding and commercial flagging lower priority.

Yikes. Speaking of system-specific. Since myth can already serve the
files out to frontends without nfs, I'd guess it would be far easier
to extend that for processing jobs as well than to start playing in
the kernel I/O elevator land.

> > Or you could solve it the way Internet is usually solved -- brute
> > force and more capacity than is going to be needed for the job at hand
>
> But transcoding and commercial flagging are not real-time processes,
> we should be able to run them when recording/playback is not
> happening, or better yet use only the disk/cpu/network resources
> that are currently going unused.

That's exactly what I started out to do, until I heard people want to
use resources that are NOT going unused, except sometimes they don't,
and I must prove I'm not going to trash a recording on a system
already 99% committed. That's a couple of objectives too many for my
original rather simple patch :)

> I would think that monitoring the buffer fill on recording and
> playback processes would be a good enough metric to be control
> the throttling of transcode & commercial flagging processes.

Not a bad idea, that. How would one go about monitoring it? Although
the buffers are worth what - a few seconds of I/O? The jobqueue
scheduling decisions are made over minutes, so it's not really in a
position to react to (possibly very temporary) buffer fill scenario.

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

danielk at cuymedia

Dec 9, 2006, 9:35 AM

Post #7 of 10 (1111 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On Sat, 2006-12-09 at 13:57 +0200, osma ahvenlampi wrote:
> On 12/9/06, Daniel Kristjansson <danielk [at] cuymedia> wrote:
> > For most recorders people use the nice value of the backend and
> > mysql is the higher of the two since we need to write to both
> > the db and the filesystem when writing a file. I made this
> I'm not sure I understood this. Which is less nice of the two? On my
> backend, which also doubles as my primary frontend, mythbackend and
> mysqld are both nice 0 processes. I haven't tuned that since my two
> DVB tuners and generally not more than one simultaneous playback has
> never become an I/O problem.

When recording an MPEG stream MythTV needs to write the
keyframe locations to the DB. This allows for quick seeking.
If the DB blocks, MythTV waits until the DB has has finished
before writing any more A/V data to the disk. The upside
of this is that changing the niceness of mysqld has little
effect on recorder performance for most of the recorders.

> But if mysql's buffer flushes are delayed a bit due to recording
> buffer flushes, nothing is lost - and mysql writes much less. That's
> why I suggested the recorder might benefit from being less nice. Also,
> btw, might be a good reason to recommend (or even try to create)
> innodb tables instead of myisam -- innodb is better at writing through
> buffers.

Changing the mysql buffering does have an effect. I believe
we have some recommendations for this in the docs somewhere.
However, when a reschedule happens (about every 5 minutes +/-
when EIT is being collected), there is enough activity that
mysql has to read/write from/to disk. And it has to do this
before our puny keyframe write is processed.

> > We also don't want playback to run at greater niceness than
> > recording. Playback has much higher real-time requirements.
> Well - is the priority to ensure correctly recorded programs, or to
> avoid playback glitches? A recording error will cause a playback error
> anyway - again, that's why felt it might be best if recording was the
> least nice process.

Well I run all three at 0 priority. But some people can't get
playback to work at all unless it is running at high priority.
So for those users playback needs to be run at high priority.
The decoder thread and the disk reading thread are not running
at high priority, they are running at the same priority as
the recorders. Having the commercial flagging and transcode
jobs run their disk reading and writing threads at low priority
would be (is?) fine. But disk access is notorious for creating
priority inversions. Mucking with niceness works well for
allocating CPU, but not so well for allocating other resources.

> I would venture to guess that people running their backends on Linux
> 2.6 kernels are in the overwhelming majority amongst the user base.
> I'd be happy to try to work this out to support BSD, OSX and even
> Windows if someone could point me to examples of how to read system
> stats out of these systems and volunteer to test patches.

Most of us are using Linux, but I'm weary of adding Linux
dependencies. If the only possible fix on some other operating
system fixes it for Linux too, then we probably want to go
with the more portable fix.

> Does the backend even run on Windows?

Almost, Jerry Rubinow is working on it. (see Ticket #1590).
He hasn't submitted a patch in a while because he ran into
some toolchain problems, and so far the new toolchain
appears to require more changes to MythTV.

> I did say "in theory" :)

:)

> The only thing this really requires is a way to read "how busy is the
> CPU", and information on whether that busyness includes I/O wait or
> not. Like I wrote in my previous message, under these circumstances
> the patch will *reduce* competition for resources.

I hadn't considered I/O wait, that might be the simplest way to
retrofit this metric.. The I/O wait stat includes both disk and
network, right?

> Unfortunately,
> getting even that much info is entirely OS-specific. I was thinking of
> digging into libgtop to see how it's done on most systems. Which would
> be more preferable, linking against that or just using it as an
> example for a re-implementation?

Generally, if it is small re-implement, if it is optional and
the library is generally available link, if it is big and not
optional import the library. But Issac is the final arbiter on
this issue since we all can easily disagree on where a library
fits on this continuum.

> > > Or you could solve it the way Internet is usually solved -- brute
> > > force and more capacity than is going to be needed for the job at hand
> >
> > But transcoding and commercial flagging are not real-time processes,
> > we should be able to run them when recording/playback is not
> > happening, or better yet use only the disk/cpu/network resources
> > that are currently going unused.
>
> That's exactly what I started out to do, until I heard people want to
> use resources that are NOT going unused, except sometimes they don't,
> and I must prove I'm not going to trash a recording on a system
> already 99% committed. That's a couple of objectives too many for my
> original rather simple patch :)

It's impossible to keep all resources 100% utilized with an
application like MythTV with some soft real-time requirements.
It is better to err on under-committing the resources we have.

> > I would think that monitoring the buffer fill on recording and
> > playback processes would be a good enough metric to be control
> > the throttling of transcode & commercial flagging processes.
>
> Not a bad idea, that. How would one go about monitoring it? Although
> the buffers are worth what - a few seconds of I/O? The jobqueue
> scheduling decisions are made over minutes, so it's not really in a
> position to react to (possibly very temporary) buffer fill scenario.

Together the buffers for recording are somewhere in the 30-60
second range, but they are generally far from full. I'm thinking
that we could just keep a running average & min & max stats over
a 5 minute window. If the max comes close to 75% or the average
approaches 50% we freeze jobs, for some values less than that
we throttle the jobs.

For playback the fill of the RingBuffers could be monitored as
well but there if the min comes close to 25% or the average
falls below 50% we would freeze or throttle the jobs, resp.
(You would need to throw out the outliers when seeking.)

But before adding this instrumentation, it might make sense to
look at I/O wait for this metric, and throttling the jobs
based on that. If that works, I'm sure we can find a way to
collect that stat on other operating systems..

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

danielk at cuymedia

Dec 9, 2006, 9:55 AM

Post #8 of 10 (1112 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On Fri, 2006-12-08 at 17:42 -0500, f-myth-users [at] media wrote:
> > From: "osma ahvenlampi" <oa [at] iki>
> > All of this would make a good argument for using an embedded libsqlite
> > instead of MySQL, since then the parameters would be in MythTV's
> > control.. I don't want to pick a fight by arguing that, though :)
> I have no opinion on that, but it hasn't gotten a good reception on
> the list when others have suggested it. (Or maybe it's that it just
> wasn't up to the task; I don't recall.)

It was tried last April, sqlite couldn't handle the scheduler
query. I don't remember if it was brought to the devs of sqlite
as an optimization challenge, but I remember doing that was
suggested by a couple folks.

It's PostgreSQL which has gotten the bad response, it is seen
as not adding enough value for the amount of work that would
be required to make MythTV compatible with it.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 9, 2006, 10:11 AM

Post #9 of 10 (1116 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/9/06, Daniel Kristjansson <danielk [at] cuymedia> wrote:
> When recording an MPEG stream MythTV needs to write the
> keyframe locations to the DB. This allows for quick seeking.
> If the DB blocks, MythTV waits until the DB has has finished
> before writing any more A/V data to the disk.

I don't think MySQL will block under this scenario if using InnoDB. I
haven't tested that, though. Even if it will, InnoDB commit is a 4096
byte sequential write to a ringbuffer, much faster than MyISAM's
seek-write-seek-write-index. FWIW, libsqlite is better with this than
MyISAM as well.

> Most of us are using Linux, but I'm weary of adding Linux
> dependencies. If the only possible fix on some other operating
> system fixes it for Linux too, then we probably want to go
> with the more portable fix.

Sure. I could not find a way to get to this information without being
OS-specific, *but* not getting to it doesn't hurt the system any
longer - the new version of the patch just won't start more than one
background job without having idle time.

> I hadn't considered I/O wait, that might be the simplest way to
> retrofit this metric.. The I/O wait stat includes both disk and
> network, right?

It should, but I'm not that deep of an expert on the kernel. This is
where I got the idea for the patch from - Linux 2.6 is the first OS
I've seen to separate iowait state out of system or user time, though
I'm sure there are others (haven't looked whether Solaris does it
these days). But I'm not depending on seeing the iowait, I'm looking
at idle vs all the rest, so I'm only depending on idle not including
iowait. As far as I know, that is true for most other Unix-like
systems.

> Generally, if it is small re-implement, if it is optional and
> the library is generally available link, if it is big and not
> optional import the library. But Issac is the final arbiter on
> this issue since we all can easily disagree on where a library
> fits on this continuum.

Well, libgtop is the best portable system info library I know of, but
it does a lot more than is needed here (full process lists and memory
maps etc) and is not installed by default with anything but GNOME
(AFAIK). KSysGuard hasn't been updated to look at I/O wait so it will
actually miscalculate total CPU (and resultingly the percentage of
idle time vs total time). I'd recommend using libgtop as a reference
to implementing support for other OSes, but I couldn't find any that
were actually advertised as alternative platforms for mythbackend.

> It's impossible to keep all resources 100% utilized with an
> application like MythTV with some soft real-time requirements.
> It is better to err on under-committing the resources we have.

Fully agree. I believe this patch will do exactly this, as it will now
permit *one* background job when they are enabled, but will not permit
more if idle time looks to be close to zero (well, less than 25% which
is in my empiric testing as good a threshold as any). That is, the old
functionality would have been to start up-to maxJobs under the same
circumstances, and potentially overcommit resources.

And once again, if the idle time can not be determined (as would be
the case for non-Linux platforms), it falls back to old behaviour.

Please review the patch (jobqueue_dynamic) -- the part about stripping
away settings is separate and doesn't need to be considered at this
point.

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

ylee at pobox

Dec 9, 2006, 11:25 AM

Post #10 of 10 (1110 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

f-myth-users [at] media <f-myth-users [at] media> says:
> That means that a recording that ended while another was ongoing
> cause the ongoing one to drop many seconds at a time. Not always,
> but often.

This is why I--with three HDTV recording streams and a not-so-fast NAS
to store recordings on--long since switched to manually patching the
ATrpms MythTV SRPMs with, among other things, significantly-increased
values in ThreadedFileWriter.cpp. See attached patch. This patch made
the single biggest difference in letting my frontend-backend system
record from all three streams simultaneously without dropping video
back when I was getting started with MythTV a year ago. (Switching to
CIFS was #2.) Even adding a 24x7 mythtranscode or mythcommflag process
didn't worsen things. Playback stutters are still almost impossible to
avoid on my setup when all three streams are recording away (and often
with only two), but at least the recordings are always intact; I much
prefer this to perfect playbacks with spotty recordings.

Please note that the patch is the result of a non-coder's
experimenting; I'd especially be curious to hear those who actally
know what they're talking about comment on the divisors I use for
TFW_MAX_BUF_SIZE and TFW_MIN_BUF_SIZE, and what the optimal values for
them would be in a typical remote recording-directory situation.

I submitted a ticket long ago but it was rejected, and properly so,
for forcing everyone, including non HDTV-using folks, to use
significantly-larger amounts of memory to record. That said, I
strongly suspect that the current values are simply too low.

> Some parameter that can be set without recompiling. Lots of people
> use prepackaged distributions; having to recompile just to turn this
> off would shaft them. So presumably a mythtv-setup value or at
> least a flag in the DB that can be set via a MySQL statement.

Absolutely. I'd *much* rather be able to tweak these settings from a
proper interface rather than recompiling. If not that, I'd take a
MythWeb-configurable setting.

Having more options are, nine out of ten times, a good thing. Yes,
there already are many pages of options between mythfrontend and
mythtv-setup. Those who don't want to add any more had better never,
ever claim that "the ability to configure things the way I want" is a
reason why they started MythTV in the first place!

Attachments:

patch-ThreadedFileWriter.cpp (0.98 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads