Mailing List Archive: Linux: Kernel

[RFC] PyTimechart

Index | Next | Previous | View Threaded

tardyp at gmail

May 11, 2010, 2:10 PM

Post #1 of 10 (273 views)
Permalink

[RFC] PyTimechart

Hello,

PyTimechart is another implementation of two very useful tools
available for the linux community:
perf-timechart ( http://blog.fenrus.org/?p=5 ) and bootchart (
http://www.bootchart.org/ )

The two tools share a common idea of making their output to SVG files.
While it is a very good idea for small traces, the generated SVG can
be very heavy, and turns out to be good stress tests for inkscape
developers...

PyTimechart is a tool that parses ftrace text traces, and display them
with the help of a very powerful dynamic plot framework, Chaco (
http://code.enthought.com/chaco/ )
The GUI makes the best it can to ease the browsing of huge traces.

The best is to look at those two 30s screencasts, to figure out how that work.

a look at mplayer startup:
http://tardyp.free.fr/pytimechart/mplayer_start.mp4
a look at ubuntu boot:
http://tardyp.free.fr/pytimechart/boot.mp4

This tool still is in the state of a prototype, I dont know if it
worth to continue to improve it or to implement it ideas in LTTV or
Kernel Shark.
It is actually very useful in its current form (
http://elinux.org/images/0/07/Effect_of_wakeups_on_Moorestown_power.pdf
), and will work without recompiling the kernel of recent distros.

the source code with build instruction is located here
http://gitorious.org/pytimechart

What do you think?

Regards,
--
Pierre
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fweisbec at gmail

May 11, 2010, 2:36 PM

Post #2 of 10 (269 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

On Tue, May 11, 2010 at 11:10:36PM +0200, Pierre Tardy wrote:
> Hello,
>
> PyTimechart is another implementation of two very useful tools
> available for the linux community:
> perf-timechart ( http://blog.fenrus.org/?p=5 ) and bootchart (
> http://www.bootchart.org/ )
>
> The two tools share a common idea of making their output to SVG files.
> While it is a very good idea for small traces, the generated SVG can
> be very heavy, and turns out to be good stress tests for inkscape
> developers...
>
> PyTimechart is a tool that parses ftrace text traces, and display them
> with the help of a very powerful dynamic plot framework, Chaco (
> http://code.enthought.com/chaco/ )
> The GUI makes the best it can to ease the browsing of huge traces.
>
> The best is to look at those two 30s screencasts, to figure out how that work.
>
> a look at mplayer startup:
> http://tardyp.free.fr/pytimechart/mplayer_start.mp4
> a look at ubuntu boot:
> http://tardyp.free.fr/pytimechart/boot.mp4
>
> This tool still is in the state of a prototype, I dont know if it
> worth to continue to improve it or to implement it ideas in LTTV or
> Kernel Shark.
> It is actually very useful in its current form (
> http://elinux.org/images/0/07/Effect_of_wakeups_on_Moorestown_power.pdf
> ), and will work without recompiling the kernel of recent distros.
>
> the source code with build instruction is located here
> http://gitorious.org/pytimechart
>
> What do you think?

I think I've been dreaming about this several times.
I've often used perf timechart lately and I'm really frustrated by
its inherent slowness due to its huge svg files. It's barely usable
with a small trace on two cpus, and it's impossible to go further,
which is really a pity for such a powerful tool.

IMO, this GUI is exactly what we want.

If you could plug it to the perf scripting facilities, I would
be very happy to see it merged in perf tools.

Plugging to the scripting API is really easy, run:

$ perf timechart record
$ sudo ./perf trace -g python
generated Python script: perf-trace.py

Now look at ./perf-trace.py, you'll find all the necessary hooks
generated with a default behaviour (displaying traces), which
you can observe by launching:

$ perf trace -s perf-trace.py

You just need to change it to plug your app on it.

(Adding more Cc).

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

tardyp at gmail

May 12, 2010, 6:37 AM

Post #3 of 10 (263 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

> Plugging to the scripting API is really easy, run:
>
> $ perf timechart record
> $ sudo ./perf trace -g python
> generated Python script: perf-trace.py

This is something that I want to try since I saw the perf scripting
patch, thanks for the recipe, I'll look at it in the next few days.

The next problem is that perf timechart record is frozen on the number
of event it records.
Pytimechart is able to display irq:* and workqueues:* events, as well
as trace_printks ( I dont know if perf is able to dump those )

I use trace_printk to mark the big traces with events I'm trying to debug.
Being able to look at ISR and workqueues is very useful, and there is
a special feature that warns when an ISR lasts more than 1ms.

For inclusion in mainline, this might take more time. The quality of
some portion of the code is far from being linux kernel's standards
:-}

Regards,
Pierre
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fweisbec at gmail

May 12, 2010, 7:48 AM

Post #4 of 10 (263 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

On Wed, May 12, 2010 at 03:37:27PM +0200, Pierre Tardy wrote:
> > Plugging to the scripting API is really easy, run:
> >
> > $ perf timechart record
> > $ sudo ./perf trace -g python
> > generated Python script: perf-trace.py
>
> This is something that I want to try since I saw the perf scripting
> patch, thanks for the recipe, I'll look at it in the next few days.
>
> The next problem is that perf timechart record is frozen on the number
> of event it records.

What do you mean? That it can't do incremental records or so?
It's supported by perf record, may be not by perf timechart but
that would be a matter of a little patch.

> Pytimechart is able to display irq:* and workqueues:* events, as well
> as trace_printks ( I dont know if perf is able to dump those )
>
> I use trace_printk to mark the big traces with events I'm trying to debug.
> Being able to look at ISR and workqueues is very useful, and there is
> a special feature that warns when an ISR lasts more than 1ms.

No problem with the irq and workqueues, if that's really useful, we
can change perf timechart to handle this.

But we don't yet support trace_printk in perf. May be we could wrap
them in trace events.

Why not starting with a release that does the same things
than timechart and trying to integrate the rest incrementally?

> For inclusion in mainline, this might take more time. The quality of
> some portion of the code is far from being linux kernel's standards
> :-}

The code I'm looking at here doesn't look dirty to me at a glance:
http://gitorious.org/pytimechart/pytimechart/trees/master

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

rostedt at goodmis

May 12, 2010, 8:36 AM

Post #5 of 10 (264 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

On Wed, 2010-05-12 at 16:48 +0200, Frederic Weisbecker wrote:
> On Wed, May 12, 2010 at 03:37:27PM +0200, Pierre Tardy wrote:

> But we don't yet support trace_printk in perf. May be we could wrap
> them in trace events.

Hmm, do we really want to do that?

We really need to get the perf and ftrace trace buffers combined. I
understand why perf chose to do the mmap buffers for the counting, but
for live streaming, it is very inefficient compared to splice.

I would hate to add more duplicate code to have perf support
trace_printk().

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

mingo at elte

May 12, 2010, 9:59 AM

Post #6 of 10 (268 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

* Steven Rostedt <rostedt [at] goodmis> wrote:

> On Wed, 2010-05-12 at 16:48 +0200, Frederic Weisbecker wrote:
> > On Wed, May 12, 2010 at 03:37:27PM +0200, Pierre Tardy wrote:
>
> > But we don't yet support trace_printk in perf. May be we could wrap
> > them in trace events.
>
> Hmm, do we really want to do that?
>
> We really need to get the perf and ftrace trace buffers combined. I
> understand why perf chose to do the mmap buffers for the counting, but for
> live streaming, it is very inefficient compared to splice.

The thing is that for a very long time ftrace didnt have splice support and
survived just fine. Even today most of the ftrace usage isnt utilizing splice.

Yes, splice might help in some situations but on average it's an independent
speedup on the order of magnitude of a few percents, not a 'must have' item.

So please keep these issues separate.

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

tardyp at gmail

May 12, 2010, 10:13 AM

Post #7 of 10 (268 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

On Wed, May 12, 2010 at 4:48 PM, Frederic Weisbecker <fweisbec [at] gmail> wrote:
> On Wed, May 12, 2010 at 03:37:27PM +0200, Pierre Tardy wrote:
>> > Plugging to the scripting API is really easy, run:
>> >
>> > $ perf timechart record
>> > $ sudo ./perf trace -g python
>> > generated Python script: perf-trace.py
>>
>> This is something that I want to try since I saw the perf scripting
>> patch, thanks for the recipe, I'll look at it in the next few days.
>>
>> The next problem is that perf timechart record is frozen on the number
>> of event it records.
>
>
>
> What do you mean? That it can't do incremental records or so?
> It's supported by perf record, may be not by perf timechart but
> that would be a matter of a little patch.

I mean that perf timechart record is implemented as an hardcoded perf
record call.
Typically, user wants to add is own events for pytimechart to display
them in a generic manner.
I agree this is a trivial change.

>
> Why not starting with a release that does the same things
> than timechart and trying to integrate the rest incrementally?
Yes, I'll see what I can do.

> The code I'm looking at here doesn't look dirty to me at a glance:
> http://gitorious.org/pytimechart/pytimechart/trees/master
That's because you did not read between the lines ;-)
The display core needs to be simplified.

Regards,
Pierre
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

rostedt at goodmis

May 12, 2010, 10:15 AM

Post #8 of 10 (265 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

On Wed, 2010-05-12 at 18:59 +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt [at] goodmis> wrote:
>
> > On Wed, 2010-05-12 at 16:48 +0200, Frederic Weisbecker wrote:
> > > On Wed, May 12, 2010 at 03:37:27PM +0200, Pierre Tardy wrote:
> >
> > > But we don't yet support trace_printk in perf. May be we could wrap
> > > them in trace events.
> >
> > Hmm, do we really want to do that?
> >
> > We really need to get the perf and ftrace trace buffers combined. I
> > understand why perf chose to do the mmap buffers for the counting, but for
> > live streaming, it is very inefficient compared to splice.
>
> The thing is that for a very long time ftrace didnt have splice support and
> survived just fine. Even today most of the ftrace usage isnt utilizing splice.

Actually, trace-cmd implements the splice interface and is used by
several people. I find myself using trace-cmd 90% of the time that I use
ftrace, specifically because of this speedup.

>
> Yes, splice might help in some situations but on average it's an independent
> speedup on the order of magnitude of a few percents, not a 'must have' item.

I'll have start running benchmarks to see what the actual speed up is.
I'm guessing it may be more than a few percent. It allows for zero copy
overhead and reuse of the data page.

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

mingo at elte

May 12, 2010, 11:23 AM

Post #9 of 10 (253 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

* Steven Rostedt <rostedt [at] goodmis> wrote:

> On Wed, 2010-05-12 at 18:59 +0200, Ingo Molnar wrote:
> > * Steven Rostedt <rostedt [at] goodmis> wrote:
> >
> > > On Wed, 2010-05-12 at 16:48 +0200, Frederic Weisbecker wrote:
> > > > On Wed, May 12, 2010 at 03:37:27PM +0200, Pierre Tardy wrote:
> > >
> > > > But we don't yet support trace_printk in perf. May be we could wrap
> > > > them in trace events.
> > >
> > > Hmm, do we really want to do that?
> > >
> > > We really need to get the perf and ftrace trace buffers combined. I
> > > understand why perf chose to do the mmap buffers for the counting, but
> > > for live streaming, it is very inefficient compared to splice.
> >
> > The thing is that for a very long time ftrace didnt have splice support
> > and survived just fine. Even today most of the ftrace usage isnt utilizing
> > splice.
>
> Actually, trace-cmd implements the splice interface and is used by several
> people. I find myself using trace-cmd 90% of the time that I use ftrace,
> specifically because of this speedup.

i know, but most people still use /debug/tracing/ bits not trace-cmd.

> > Yes, splice might help in some situations but on average it's an
> > independent speedup on the order of magnitude of a few percents, not a
> > 'must have' item.
>
> I'll have start running benchmarks to see what the actual speed up is. I'm
> guessing it may be more than a few percent. It allows for zero copy overhead
> and reuse of the data page.

Make sure you measure it in the context of a full app like PyTimechart.

You can measure the overhead using perf stat ;-)

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

acme at ghostprotocols

May 13, 2010, 8:02 AM

Post #10 of 10 (255 views)
Permalink

Re: [RFC] PyTimechart [In reply to]

Em Wed, May 12, 2010 at 07:13:00PM +0200, Pierre Tardy escreveu:
> > The code I'm looking at here doesn't look dirty to me at a glance:
> > http://gitorious.org/pytimechart/pytimechart/trees/master

> That's because you did not read between the lines ;-)
> The display core needs to be simplified.

Not even having looked at the code, I like your attitude! :-)

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads