Mailing List Archive: MythTV: Dev

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling

Index | Next | Previous | View Threaded

adeffs.mythtv at gmail

Dec 7, 2006, 9:11 AM

Post #1 of 8 (1327 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling

On 12/6/06, MythTV <mythtv [at] cvs> wrote:
> #2782: automatic simultaneous jobs scaling
> -------------------------+--------------------------------------------------
> Reporter: oa [at] iki | Owner: cpinkham
> Type: enhancement | Status: new
> Priority: minor | Milestone: unknown
> Component: mythtv | Version: 0.20
> Severity: medium | Resolution:
> -------------------------+--------------------------------------------------
> Changes (by cpinkham):
>
> * owner: ijr => cpinkham

sweet, I've been looking for something like this to get put together!
I'll have to apply this to my next backend fixes compile.
Thanks for the work!

--
Steve
Before you ask, read the FAQ!
http://www.mythtv.org/wiki/index.php/Frequently_Asked_Questions
then search the Wiki, and this list,
http://www.gossamer-threads.com/lists/mythtv/
Mailinglist etiquette -
http://www.mythtv.org/wiki/index.php/Mailing_List_etiquette
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

cpinkham at bc2va

Dec 7, 2006, 10:00 AM

Post #2 of 8 (1266 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

* On Thu Dec 07, 2006 at 12:11:52PM -0500, Steven Adeff wrote:
> > #2782: automatic simultaneous jobs scaling

> > * owner: ijr => cpinkham
>
> sweet, I've been looking for something like this to get put together!
> I'll have to apply this to my next backend fixes compile.

I haven't looked at it yet to check the feasibility. We tried something
like this in the commercial flagger a long time ago where it would throttle
itself based on the system load at the time, but it caused problems when
using framegrabbers. With ?most? people using non-framegrabbing capture
cards now, this is more of a possibility, but there are other things that
we need to consider than just CPU. Just because my system can run X jobs
before the CPU usage gets above Y% doesn't mean I want to run X jobs because
that will affect things like storage, potentially network throughput, etc..
Users will probably always prefer a late commercial flag or transcode rather
than a corrupted recording.

All that being said, I haven't even looked at the patch yet, I just wanted
to put it in my queue since it is JobQueue related. Even if it is perfect,
don't expect it to go in over the next few weeks because my holidays are a
bit busy. That is one of the reasons I wanted to get Storage Groups committed
a few weeks ago to give it time to stabilize.

--
Chris
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 7, 2006, 11:25 AM

Post #3 of 8 (1267 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/7/06, Chris Pinkham <cpinkham [at] bc2va> wrote:
> cards now, this is more of a possibility, but there are other things that
> we need to consider than just CPU. Just because my system can run X jobs
> before the CPU usage gets above Y% doesn't mean I want to run X jobs because
> that will affect things like storage, potentially network throughput, etc..

The way I wrote the patch, it will only start a job if 25% of CPU is
"idle". That 25% is hardcoded because not only did I want to make this
automatic, I also really wanted not to introduce any more config
options (believing that software should "do the right thing right"
without configuring). 25% should also be 1) easily enough for
recording and playback, especially since the jobs will be run niced
anyway, and 2) actually relatively rare, especially on a 1-2 core
machine since most candidate jobs would consume all resources
available.

It also considers ONLY time reported truly "idle", ie. not used by
sys, user, nice OR iowait, so it should cover for the cases I think
you're referring to, although truly considering network load would be
very tricky, it not being entirely measurable on the host. Anyway, in
my environment I've only seen two kinds of situations that have been
problems with the old way, but both of these are very common: two
CPU-intensive concurrent processes, ie MPEG-2 -> MPEG-4 transcodes or
transcode plus delayed commflag, each finishing later than they would
have had only one been running at once, and two IO-intensive
concurrent processes, ie MPEG-2 -> MPEG-2 lossless transcodes, also
finishing (much) later than if run sequentially, AND hurting
concurrent recording and playback performance.

The patch fixes both of these situations "perfectly" (well, it's
worked fine for me for several days, while I record up to two DVB
streams at once on a combined FE/BE), and yet allows also two
real-time commflags (but only if no transcodes are running when the
commflags are candidates for starting - that's something that could
potentially be improved but would complicate the jobqueue scheduler
quite a bit).

The biggest drawback is that the "util" part of figuring out whether
idle time is available or not only supports Linux (it reads from
/proc/stat), so support for other OSs would need to be contributed by
others. The way it's written, it will allow jobs to start if it can't
figure out whether CPU is available.

> Users will probably always prefer a late commercial flag or transcode rather
> than a corrupted recording.

This has never been an issue for me, but I use DVB receivers instead
of framegrabbers.

> All that being said, I haven't even looked at the patch yet, I just wanted
> to put it in my queue since it is JobQueue related. Even if it is perfect,
> don't expect it to go in over the next few weeks because my holidays are a
> bit busy. That is one of the reasons I wanted to get Storage Groups committed
> a few weeks ago to give it time to stabilize.

Cool. Let me know if you have any questions or critique, I'd be happy
to work on that. However, I don't expect to upgrade from 0.20-fixes
any time soon, my install needs to remain WAF, so I hope that won't be
a problem.

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

mtdean at thirdcontact

Dec 7, 2006, 5:03 PM

Post #4 of 8 (1254 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/07/2006 02:25 PM, osma ahvenlampi wrote:
> On 12/7/06, Chris Pinkham <cpinkham [at] bc2va> wrote:
>
>> cards now, this is more of a possibility, but there are other things that
>> we need to consider than just CPU. Just because my system can run X jobs
>> before the CPU usage gets above Y% doesn't mean I want to run X jobs because
>> that will affect things like storage, potentially network throughput, etc..
>>
> The way I wrote the patch, it will only start a job if 25% of CPU is
> "idle". That 25% is hardcoded because not only did I want to make this
> automatic, I also really wanted not to introduce any more config
> options (believing that software should "do the right thing right"
> without configuring). 25% should also be 1) easily enough for
> recording and playback, especially since the jobs will be run niced
> anyway, and 2) actually relatively rare, especially on a 1-2 core
> machine since most candidate jobs would consume all resources
> available.
>
> It also considers ONLY time reported truly "idle", ie. not used by
> sys, user, nice OR iowait,
>

Guess I'll be disabling this if it goes in.

My master backend:
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
4 0 14788 5864 1816 252752 0 0 100 112 82 53 99 1
0 0

My slave backend:
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
1 0 48 50296 42792 339832 0 0 49 107 77 10 99 1
0 0

My dedicated frontend (with mythjobqueue):
(not powered on and I'm 2000 miles away from the power button, but its
idle value will say "0")

The idle value /never/ changes on any of the hosts, thanks to a little
program I call AthWarm (most call SETI/BOINC). :)

No problem, though, I wouldn't mind disabling job scaling. (Now,
disabling BOINC, OTOH, will not happen :).

>> Users will probably always prefer a late commercial flag or transcode rather
>> than a corrupted recording.
>>
> This has never been an issue for me, but I use DVB receivers instead
> of framegrabbers.
>

This will happen even without encoding video (i.e. even with DVB or
HDTV) if I/O wait becomes "critical".

Mike

My /proc/stat from my master backend:

cpu 838779 36201503 208958 10015 1537 16826 73892 0
cpu0 838779 36201503 208958 10015 1537 16826 73892 0
intr 202552568 93372945 8 0 1 1 1 0 1 0 0 1 1 1 0 1220376 3550776 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 366394 0 0 0 0 0 0 0 99468785 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 4573277 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 148767913
btime 1165165447
processes 8462
procs_running 2
procs_blocked 0

(only the I/O wait time is lower than idle time, showing that I don't
let my idle process get its share of the CPU--I'm pretty sure it will
never "finish" on my system :)
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 7, 2006, 11:29 PM

Post #5 of 8 (1255 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/8/06, Michael T. Dean <mtdean [at] thirdcontact> wrote:
> The idle value /never/ changes on any of the hosts, thanks to a little
> program I call AthWarm (most call SETI/BOINC). :)

Ah, I didn't consider someone wanting to run their HTPC hot - I've
been more concerned about running it quiet. How would you propose
detecting between time spent in SETI vs time spent by jobs? The kernel
won't distinguish between the niced processes..

> No problem, though, I wouldn't mind disabling job scaling. (Now,
> disabling BOINC, OTOH, will not happen :).

I didn't make this configurable, because I would have liked to make it
"work right" without configuring. That can be done though, but I'd
rather figure out a better solution.

> This will happen even without encoding video (i.e. even with DVB or
> HDTV) if I/O wait becomes "critical".

It *can* happen, but *will* it? With 2.6.15+ and CFQ I/O queues, I
can't see it being very realistic. Does anyone have empiric proof? On
Linux or some other system?

At least one change would need to happen for this to become
commit-worthy, I guess: right now jobqueue will not start *any*
processes if there's no idle time, but it will start them if it can't
read idle time at all. I'll change it so that it will start *one*
process in either of those situations, but scale up from there only if
there's still time available. Would that be satisfactory?

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

jppoet at gmail

Dec 8, 2006, 9:07 AM

Post #6 of 8 (1242 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/8/06, osma ahvenlampi <oa [at] iki> wrote:

<snip>

> This is why I referred to CFQ scheduling. With CFQ enabled and the
> jobs being niced, they won't be getting a whole lot of I/O bandwidth
> in this situation, so I really doubt that will be the thing taking you
> over the edge -- it'll be the too-many-recordings-and-playbacks alone
> that would be the real contributors, as you said (in the part I
> clipped).

IONICE is one of the cool things about CFQ

I experimented with using ionice to prevent auxiliary applications
from interfering with mythbackend, with very poor results. "ionice
-c3" is suppose to set a process to only get I/O if the disk is idle.
This did not seem to match reality.

After some searching, I finally found where someone said that ionice
*only* effects READS. WRITES are not effected. Since my auxiliary
app was trying to write to the same disk that mythbackend was, using
ionice did not prevent the aux app from stealing write bandwidth from
mythbackend.

I assume commflagging does not do much in the way of disk writing, but
transcoding does.

<snip>

John
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

mtdean at thirdcontact

Dec 8, 2006, 7:17 PM

Post #7 of 8 (1242 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/08/2006 02:29 AM, osma ahvenlampi wrote:
> On 12/8/06, Michael T. Dean <mtdean [at] thirdcontact> wrote:
>
>> The idle value /never/ changes on any of the hosts, thanks to a little
>> program I call AthWarm (most call SETI/BOINC). :)
>>
>
> Ah, I didn't consider someone wanting to run their HTPC hot - I've
> been more concerned about running it quiet. How would you propose
> detecting between time spent in SETI vs time spent by jobs? The kernel
> won't distinguish between the niced processes..
>

Yeah, my 2 backends and 1 frontend are not in the media room.
Therefore, noise from running hot has no effect on my viewing
experience. I like my media room clean--one 67" HDTV, speakers, and
places to sit--no ugly A/V receivers, STB's, PC's, tangle o' cables, ...

>> No problem, though, I wouldn't mind disabling job scaling. (Now,
>> disabling BOINC, OTOH, will not happen :).
>>
>
> I didn't make this configurable, because I would have liked to make it
> "work right" without configuring. That can be done though, but I'd
> rather figure out a better solution.
>

Not a problem for me--I keep my own patch tree. ;)

> At least one change would need to happen for this to become
> commit-worthy, I guess: right now jobqueue will not start *any*
> processes if there's no idle time, but it will start them if it can't
> read idle time at all. I'll change it so that it will start *one*
> process in either of those situations, but scale up from there only if
> there's still time available. Would that be satisfactory?
>

Yep. With that change, I wouldn't even bother disabling the scaling. I
run with a max of 1 job on all my hosts because of the I/O issues that
Daniel mentioned. So, if it were allowed to start one job even at 0%
idle (i.e. with 99% nice), it would be the same as I have currently.
Basically, I prefer running on the safe side. After all, I have > 480
hours of TV (including commercials) to catch up on, so I /never/ have to
watch something "as soon as possible" after it airs--even waiting until
the next day to give commflagging plenty of time to finish isn't a problem.*

Perhaps a more "general-purpose" approach that would work for others,
too, would be detecting the percentage of CPU time spent on idle plus
the CPU time spent on processes running at a higher nice level than the
job. IIRC, jobs are run at nice 17, 10 or 0 depending on the value of
JobQueueCPU, so any job running "nicer" than the nice setting for the
job queue should be treated as idle time (i.e. time that could be used
for the job queue). (This assumes that we'll still have a JobQueueCPU
setting after the patch--I haven't looked at the patch, yet.) I run
BOINC at nice 19--and expect most people running "hot" do so with
similarly-low-priority tasks. Any process running at higher priority
than the job queue jobs should probably be treated as more important,
anyway, so this seems like a good way to decide whether a new job is
warranted.

I don't know if doing this would require parsing every /proc/*/stat to
retrieve the priority/nice value(s) (fields 18, 19, and/or 38) and the
per-process CPU usage. If so, it's probably not worth the effort, but
it's the thought that counts, right?

Mike

*In fact, I have my Myth box record all new series every season. Then,
as networks cancel some of the new series, I delete them without
watching them. If "the word on the street" says the show isn't worth my
time, I delete the series without watching it. If everyone's talking
about the show--and saying good things about it--I'll actually watch
it. So, generally, I don't watch start to watch shows until they're at
least a couple months old. Then, I'm far enough behind "real-time" that
I don't catch up until after the season finishes.
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

oa at iki

Dec 9, 2006, 4:16 AM

Post #8 of 8 (1243 views)
Permalink

Re: [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling [In reply to]

On 12/9/06, Michael T. Dean <mtdean [at] thirdcontact> wrote:
> On 12/08/2006 02:29 AM, osma ahvenlampi wrote:
> > read idle time at all. I'll change it so that it will start *one*
> > process in either of those situations, but scale up from there only if
> > there's still time available. Would that be satisfactory?
>
> Yep. With that change, I wouldn't even bother disabling the scaling. I
> run with a max of 1 job on all my hosts because of the I/O issues that
> Daniel mentioned. So, if it were allowed to start one job even at 0%
> idle (i.e. with 99% nice), it would be the same as I have currently.

This is what I just submitted to Trac, replacing the original jobqueue_2 patch.

> Perhaps a more "general-purpose" approach that would work for others,
> too, would be detecting the percentage of CPU time spent on idle plus
> the CPU time spent on processes running at a higher nice level than the
> job. IIRC, jobs are run at nice 17, 10 or 0 depending on the value of
> JobQueueCPU, so any job running "nicer" than the nice setting for the
> job queue should be treated as idle time (i.e. time that could be used
> for the job queue).

It's not possible to detemine CPU used by individual nice levels
without going through the entire list of processes, which is not only
very complicated, requires a wee bit of CPU itself, but also suffers
from race conditions when processes come and go during the scan (look
at 'top' during 'make' -- CPU idle near 0%, yet no process seems to
use it - that's because the gcc that used it often just exited before
top got to see it).

Also, that's just on Linux. On some OSes I've seen, there's no way for
a user process to get to the information in the first place.

--
Osma Ahvenlampi <oa [at] iki> http://www.fishpool.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads