Mailing List Archive: ivtv: devel

cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so

Index | Next | Previous | View Threaded

cwalls at radix

Mar 8, 2008, 8:37 PM

Post #1 of 10 (2710 views)
Permalink

cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so

Hans Verkuil wrote:
> > Here's my observation: There was no good reason for q_full to stay
> > empty for so long (> 5 seconds), just because data was sitting queued
> > in q_io. It was as if data stopped being moved from the encoder to
> > buffers and into q_full. What is both fortunate and unfortunate is
> > that draining q_full and q_io, seems to restart the transfers from
> > the encoder.
> >

> > So to test these hypotheses, I'll have to gain some understanding of
> > the new mailbox protocol and the Memory Descriptor List stuff and do
> > some testing with the fix to cx18_v4l2_enc_poll() backed out.
>
> Look for 'CX18_CPU_DE_SET_MDL' in cx18-fileops.c: this is where the
> processed buffer is returned to the firmware so that it can be used
> again. Based on your description I was wondering whether in some cases
> all or most buffers are in use and the firmware will just stop
> capturing due to a lack of available buffers.

Argh. I'm giving up on hunting this down for now. Here are my
observations and speculations with the return value of
cx18_v4l2_enc_poll() does not depend on q_io.length:

a) The most buffers I've ever seen in use in the driver are 1 in q_io
and 10 in q_full. That was when then encoder decided to give the driver
10 buffers all at once. (What good is q_io anyway in the cx18 driver?
It seems like a lot of book keeping just to store the 1 partially read
buffer. Would pushing a partially read buffer back on the head of
q_full be that hard?)

b) I can always get the transfer from the encoder to stall.

c) stopping the capture, reloading all the MDLs and restarting the
capture doesn't make transfers from the encoder work again.

d) a common sequence of failure looked like the following, with the
encoder sending a small (2048 bytes) buffer, the driver returning 2 MDLs
very close together, and then the encoder sending only one more buffer:

Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: open_id: 7 stream type = 0
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba00 buffers 61 length 2031616 bytesused 0
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9ba80 prev 0x21d9ba80 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9ba40 prev 0x21d9ba40 buffers 1 length 32768 bytesused 30720
Mar 8 19:27:09 palomino kernel: cx180 file: Encoder poll
Mar 8 19:27:09 palomino kernel: cx180 file: Post poll_wait
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba00 buffers 61 length 2031616 bytesused 0
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9ba80 prev 0x21d9ba80 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9ba40 prev 0x21d9ba40 buffers 1 length 32768 bytesused 30720
Mar 8 19:27:09 palomino kernel: cx180 file: read 4096 bytes from encoder MPEG
Mar 8 19:27:09 palomino kernel: cx180 file: read 4096 from encoder MPEG, got 4096
...
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: open_id: 7 stream type = 0
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:09 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba00 buffers 61 length 2031616 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9ba80 prev 0x21d9ba80 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9ba40 prev 0x21d9ba40 buffers 1 length 32768 bytesused 6144
Mar 8 19:27:10 palomino kernel: cx180 file: Encoder poll
Mar 8 19:27:10 palomino kernel: cx180 file: Post poll_wait
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba00 buffers 61 length 2031616 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9ba80 prev 0x21d9ba80 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9ba40 prev 0x21d9ba40 buffers 1 length 32768 bytesused 6144
Mar 8 19:27:10 palomino kernel: cx180 file: read 4096 bytes from encoder MPEG
Mar 8 19:27:10 palomino kernel: cx180 file: read 4096 from encoder MPEG, got 4096
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: open_id: 7 stream type = 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba00 buffers 61 length 2031616 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9ba80 prev 0x21d9ba80 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9ba40 prev 0x21d9ba40 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:10 palomino kernel: cx180 file: Encoder poll
Mar 8 19:27:10 palomino kernel: cx180 file: Post poll_wait
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba00 buffers 61 length 2031616 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9ba80 prev 0x21d9ba80 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9ba40 prev 0x21d9ba40 buffers 1 length 32768 bytesused 2048
Mar 8 19:27:10 palomino kernel: cx180 file: read 4096 bytes from encoder MPEG
Mar 8 19:27:10 palomino kernel: cx180 api: CX18_CPU_DE_SET_MDL
Mar 8 19:27:10 palomino kernel: cx180 api: req: 0x000035ab args: 0x00000001 0x00dc0cc8 0x00000001 0x00000003 0x00008000
Mar 8 19:27:10 palomino kernel: cx180 api: ack: 0x000035ab args: 0x00000001 0x00dc0cc8 0x00000001 0x00000003 0x00008000 0x00000000 error: 0x00000000
Mar 8 19:27:10 palomino kernel: cx180 api: CX18_CPU_DE_SET_MDL
Mar 8 19:27:10 palomino kernel: cx180 api: req: 0x000035ac args: 0x00000001 0x00dc0cd0 0x00000001 0x00000004 0x00008000
Mar 8 19:27:10 palomino kernel: cx180 api: ack: 0x000035ac args: 0x00000001 0x00dc0cd0 0x00000001 0x00000004 0x00008000 0x00000000 error: 0x00000000
Mar 8 19:27:10 palomino kernel: cx180 file: read 4096 from encoder MPEG, got 4096
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: open_id: 7 stream type = 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba80 buffers 63 length 2097152 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0xa440208 prev 0xa440208 buffers 0 length 0 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0xa440228 prev 0xa440228 buffers 0 length 0 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: Encoder poll
Mar 8 19:27:10 palomino kernel: cx180 file: Post poll_wait
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba80 buffers 63 length 2097152 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0xa440208 prev 0xa440208 buffers 0 length 0 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0xa440228 prev 0xa440228 buffers 0 length 0 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: open_id: 7 stream type = 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba80 buffers 62 length 2064384 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9bac0 prev 0x21d9bac0 buffers 1 length 32768 bytesused 26624
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0xa440228 prev 0xa440228 buffers 0 length 0 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: Encoder poll
Mar 8 19:27:10 palomino kernel: cx180 file: Post poll_wait
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba80 buffers 62 length 2064384 bytesused 0
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_full next 0x21d9bac0 prev 0x21d9bac0 buffers 1 length 32768 bytesused 26624
Mar 8 19:27:10 palomino kernel: cx180 file: enc_poll: stream q_io next 0xa440228 prev 0xa440228 buffers 0 length 0 bytesused 0
...
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: open_id: 7 stream type = 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba80 buffers 62 length 2064384 bytesused 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream q_full next 0xa440208 prev 0xa440208 buffers 0 length 0 bytesused 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9bac0 prev 0x21d9bac0 buffers 1 length 32768 bytesused 22528
Mar 8 19:27:13 palomino kernel: cx180 file: Encoder poll
Mar 8 19:27:13 palomino kernel: cx180 file: Post poll_wait
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream s_flags = 0x118
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream buffers 64 buf_size 32768 buffers_stolen 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream q_free next 0x21d9bd80 prev 0x21d9ba80 buffers 62 length 2064384 bytesused 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream q_full next 0xa440208 prev 0xa440208 buffers 0 length 0 bytesused 0
Mar 8 19:27:13 palomino kernel: cx180 file: enc_poll: stream q_io next 0x21d9bac0 prev 0x21d9bac0 buffers 1 length 32768 bytesused 22528
Mar 8 19:27:13 palomino kernel: cx18-0 info: close stopping capture
Mar 8 19:27:13 palomino kernel: cx18-0 info: close stopping embedded VBI capture

I changed the cx18_read() function to exit the loop and return once it
had returned 1 MDL to the encoder, but the problem still persisted. In
this case the short buffer in at least one trial was 4096 bytes.

e) I don't know what to make of it all without some specific commands to
query the encoder's status, use of MDLs, and timeouts. I could take a
closer look at the MDL acks and the irqs and irq acks, but I'm not that
motivated at this point. The driver/encoder transfer performance is
good enough for me for now with the return value logic of the
cx18_v4l2_enc_poll() function fixed.

Regards,
Andy

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

hverkuil at xs4all

Mar 9, 2008, 1:28 AM

Post #2 of 10 (2606 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so [In reply to]

On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> Hans Verkuil wrote:
> > > Here's my observation: There was no good reason for q_full to
> > > stay empty for so long (> 5 seconds), just because data was
> > > sitting queued in q_io. It was as if data stopped being moved
> > > from the encoder to buffers and into q_full. What is both
> > > fortunate and unfortunate is that draining q_full and q_io, seems
> > > to restart the transfers from the encoder.
> > >
> > >
> > > So to test these hypotheses, I'll have to gain some understanding
> > > of the new mailbox protocol and the Memory Descriptor List stuff
> > > and do some testing with the fix to cx18_v4l2_enc_poll() backed
> > > out.
> >
> > Look for 'CX18_CPU_DE_SET_MDL' in cx18-fileops.c: this is where the
> > processed buffer is returned to the firmware so that it can be used
> > again. Based on your description I was wondering whether in some
> > cases all or most buffers are in use and the firmware will just
> > stop capturing due to a lack of available buffers.
>
> Argh. I'm giving up on hunting this down for now. Here are my
> observations and speculations with the return value of
> cx18_v4l2_enc_poll() does not depend on q_io.length:
>
> a) The most buffers I've ever seen in use in the driver are 1 in q_io
> and 10 in q_full. That was when then encoder decided to give the
> driver 10 buffers all at once. (What good is q_io anyway in the cx18
> driver? It seems like a lot of book keeping just to store the 1
> partially read buffer. Would pushing a partially read buffer back on
> the head of q_full be that hard?)

This code is basically copied from ivtv (where bookkeeping is much more
complicated still). It might well be replaced by just pushing it back
to q_full for this driver.

> b) I can always get the transfer from the encoder to stall.
>
> c) stopping the capture, reloading all the MDLs and restarting the
> capture doesn't make transfers from the encoder work again.
>
> d) a common sequence of failure looked like the following, with the
> encoder sending a small (2048 bytes) buffer, the driver returning 2
> MDLs very close together, and then the encoder sending only one more
> buffer:

...

>
> I changed the cx18_read() function to exit the loop and return once
> it had returned 1 MDL to the encoder, but the problem still
> persisted. In this case the short buffer in at least one trial was
> 4096 bytes.
>
>
> e) I don't know what to make of it all without some specific commands
> to query the encoder's status, use of MDLs, and timeouts. I could
> take a closer look at the MDL acks and the irqs and irq acks, but I'm
> not that motivated at this point. The driver/encoder transfer
> performance is good enough for me for now with the return value logic
> of the cx18_v4l2_enc_poll() function fixed.

Is it possible to reproduce it by writing a small capture program that
acts similar to MythTV? If I had a program like this, then I could
investigate.

Regards,

Hans

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

cwalls at radix

Mar 9, 2008, 1:27 PM

Post #3 of 10 (2584 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so [In reply to]

Hans Verkuil wrote:
> On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> > Hans Verkuil wrote:
> > > > Here's my observation: There was no good reason for q_full to
> > > > stay empty for so long (> 5 seconds), just because data was
> > > > sitting queued in q_io. It was as if data stopped being moved
> > > > from the encoder to buffers and into q_full. What is both
> > > > fortunate and unfortunate is that draining q_full and q_io,
> seems
> > > > to restart the transfers from the encoder.
> > > >
> > > >
> > > > So to test these hypotheses, I'll have to gain some
> understanding
> > > > of the new mailbox protocol and the Memory Descriptor List stuff
> > > > and do some testing with the fix to cx18_v4l2_enc_poll() backed
> > > > out.
> > >
> > > Look for 'CX18_CPU_DE_SET_MDL' in cx18-fileops.c: this is where
> the
> > > processed buffer is returned to the firmware so that it can be
> used
> > > again. Based on your description I was wondering whether in some
> > > cases all or most buffers are in use and the firmware will just
> > > stop capturing due to a lack of available buffers.
> >
> > Argh. I'm giving up on hunting this down for now. Here are my
> > observations and speculations with the return value of
> > cx18_v4l2_enc_poll() does not depend on q_io.length:
> >
> > a) The most buffers I've ever seen in use in the driver are 1 in
> q_io
> > and 10 in q_full. That was when then encoder decided to give the
> > driver 10 buffers all at once. (What good is q_io anyway in the
> cx18
> > driver? It seems like a lot of book keeping just to store the 1
> > partially read buffer. Would pushing a partially read buffer back
> on
> > the head of q_full be that hard?)
>
> This code is basically copied from ivtv (where bookkeeping is much
> more
> complicated still). It might well be replaced by just pushing it back
> to q_full for this driver.

I thought so. I was also toying with the idea of more granular locking
for the queues: one lock for each q_free, q_full, and q_io. In which
case, if I could work out not deadlocking on q_free, I'd want q_io and
q_full to remain separate.

Anyway, q_io's fate isn't critical at the moment.

> > b) I can always get the transfer from the encoder to stall.
> >
> > c) stopping the capture, reloading all the MDLs and restarting the
> > capture doesn't make transfers from the encoder work again.
> >
> > d) a common sequence of failure looked like the following, with the
> > encoder sending a small (2048 bytes) buffer, the driver returning 2
> > MDLs very close together, and then the encoder sending only one more
> > buffer:
>
> ...
>
> >
> > I changed the cx18_read() function to exit the loop and return once
> > it had returned 1 MDL to the encoder, but the problem still
> > persisted. In this case the short buffer in at least one trial was
> > 4096 bytes.
> >
> >
> > e) I don't know what to make of it all without some specific
> commands
> > to query the encoder's status, use of MDLs, and timeouts. I could
> > take a closer look at the MDL acks and the irqs and irq acks, but
> I'm
> > not that motivated at this point. The driver/encoder transfer
> > performance is good enough for me for now with the return value
> logic
> > of the cx18_v4l2_enc_poll() function fixed.
>
> Is it possible to reproduce it by writing a small capture program
> that
> acts similar to MythTV? If I had a program like this, then I could
> investigate.

Just to be clear for list readers: for one stream (MPEG) being provided
by the firmware, the current cx18 driver works well for any application
that uses a minor amount of buffering. I also works well with a
strong , clean video signal for applications that don't buffer. MythTV
uses a lot of buffering. Mplayer doesn't do buffering (unless
specifically asked with the -cache option), but on most stations, it
performs well, with a minor glitch occasionally for stations with weak
signals.

Yes, if I have time, I can write a program to emulate the critical
aspects of MythTV's operation. You will have to back out the fix to
cx18_v4l2_enc_poll() to purposely induce the stall. Also tuning to a
weak, snowy channel (but not all snow) and changing channels will induce
the problem more rapidly. I assume the encoder can't compress as well
under these circumstances, so it sends more data to the host.

I will be on travel for the better part of the next two weeks. If I
don't post a test program on Monday, 10 March, don't expect anything
until Friday, 21 March at the earliest. (I'll be home for a few days in
the middle of that stretch, but that time is booked with home
maintenance and family time).

Regards,
Andy

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

hverkuil at xs4all

Mar 9, 2008, 4:05 PM

Post #4 of 10 (2583 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall , but I can't figure out why they do so [In reply to]

On Sunday 09 March 2008 21:27:53 Andy Walls wrote:
> Hans Verkuil wrote:
> > On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> > > Hans Verkuil wrote:
> > > > > Here's my observation: There was no good reason for q_full
> > > > > to stay empty for so long (> 5 seconds), just because data
> > > > > was sitting queued in q_io. It was as if data stopped being
> > > > > moved from the encoder to buffers and into q_full. What is
> > > > > both fortunate and unfortunate is that draining q_full and
> > > > > q_io,
> >
> > seems
> >
> > > > > to restart the transfers from the encoder.
> > > > >
> > > > >
> > > > > So to test these hypotheses, I'll have to gain some
> >
> > understanding
> >
> > > > > of the new mailbox protocol and the Memory Descriptor List
> > > > > stuff and do some testing with the fix to
> > > > > cx18_v4l2_enc_poll() backed out.
> > > >
> > > > Look for 'CX18_CPU_DE_SET_MDL' in cx18-fileops.c: this is where
> >
> > the
> >
> > > > processed buffer is returned to the firmware so that it can be
> >
> > used
> >
> > > > again. Based on your description I was wondering whether in
> > > > some cases all or most buffers are in use and the firmware will
> > > > just stop capturing due to a lack of available buffers.
> > >
> > > Argh. I'm giving up on hunting this down for now. Here are my
> > > observations and speculations with the return value of
> > > cx18_v4l2_enc_poll() does not depend on q_io.length:
> > >
> > > a) The most buffers I've ever seen in use in the driver are 1 in
> >
> > q_io
> >
> > > and 10 in q_full. That was when then encoder decided to give the
> > > driver 10 buffers all at once. (What good is q_io anyway in the
> >
> > cx18
> >
> > > driver? It seems like a lot of book keeping just to store the 1
> > > partially read buffer. Would pushing a partially read buffer
> > > back
> >
> > on
> >
> > > the head of q_full be that hard?)
> >
> > This code is basically copied from ivtv (where bookkeeping is much
> > more
> > complicated still). It might well be replaced by just pushing it
> > back to q_full for this driver.
>
> I thought so. I was also toying with the idea of more granular
> locking for the queues: one lock for each q_free, q_full, and q_io.
> In which case, if I could work out not deadlocking on q_free, I'd
> want q_io and q_full to remain separate.
>
> Anyway, q_io's fate isn't critical at the moment.
>
> > > b) I can always get the transfer from the encoder to stall.
> > >
> > > c) stopping the capture, reloading all the MDLs and restarting
> > > the capture doesn't make transfers from the encoder work again.
> > >
> > > d) a common sequence of failure looked like the following, with
> > > the encoder sending a small (2048 bytes) buffer, the driver
> > > returning 2 MDLs very close together, and then the encoder
> > > sending only one more buffer:
> >
> > ...
> >
> > > I changed the cx18_read() function to exit the loop and return
> > > once it had returned 1 MDL to the encoder, but the problem still
> > > persisted. In this case the short buffer in at least one trial
> > > was 4096 bytes.
> > >
> > >
> > > e) I don't know what to make of it all without some specific
> >
> > commands
> >
> > > to query the encoder's status, use of MDLs, and timeouts. I
> > > could take a closer look at the MDL acks and the irqs and irq
> > > acks, but
> >
> > I'm
> >
> > > not that motivated at this point. The driver/encoder transfer
> > > performance is good enough for me for now with the return value
> >
> > logic
> >
> > > of the cx18_v4l2_enc_poll() function fixed.
> >
> > Is it possible to reproduce it by writing a small capture program
> > that
> > acts similar to MythTV? If I had a program like this, then I could
> > investigate.
>
> Just to be clear for list readers: for one stream (MPEG) being
> provided by the firmware, the current cx18 driver works well for any
> application that uses a minor amount of buffering. I also works well
> with a strong , clean video signal for applications that don't
> buffer. MythTV uses a lot of buffering. Mplayer doesn't do
> buffering (unless specifically asked with the -cache option), but on
> most stations, it performs well, with a minor glitch occasionally for
> stations with weak signals.
>
>
> Yes, if I have time, I can write a program to emulate the critical
> aspects of MythTV's operation. You will have to back out the fix to
> cx18_v4l2_enc_poll() to purposely induce the stall. Also tuning to a
> weak, snowy channel (but not all snow) and changing channels will
> induce the problem more rapidly. I assume the encoder can't compress
> as well under these circumstances, so it sends more data to the host.
>
> I will be on travel for the better part of the next two weeks. If I
> don't post a test program on Monday, 10 March, don't expect anything
> until Friday, 21 March at the earliest. (I'll be home for a few days
> in the middle of that stretch, but that time is booked with home
> maintenance and family time).
>
> Regards,
> Andy

Well, the same is true for me, I doubt I'll get anything done until
after Easter.

Regards,

Hans

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

cwalls at radix

Mar 21, 2008, 7:03 PM

Post #5 of 10 (2541 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so [In reply to]

On Sun, 2008-03-09 at 16:28 -0400, Andy Walls wrote:
> Hans Verkuil wrote:
> > On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> > > Hans Verkuil wrote:
> > > > > Here's my observation: There was no good reason for q_full to
> > > > > stay empty for so long (> 5 seconds), just because data was
> > > > > sitting queued in q_io. It was as if data stopped being moved
> > > > > from the encoder to buffers and into q_full. What is both
> > > > > fortunate and unfortunate is that draining q_full and q_io,
> > seems
> > > > > to restart the transfers from the encoder.

> > > Argh. I'm giving up on hunting this down for now. Here are my
> > > observations and speculations with the return value of
> > > cx18_v4l2_enc_poll() does not depend on q_io.length:
> > >
> > > a) The most buffers I've ever seen in use in the driver are 1 in
> > q_io
> > > and 10 in q_full. That was when then encoder decided to give the
> > > driver 10 buffers all at once.

> > > b) I can always get the transfer from the encoder to stall.
> > >
> > > c) stopping the capture, reloading all the MDLs and restarting the
> > > capture doesn't make transfers from the encoder work again.
> > >
> > > d) a common sequence of failure looked like the following, with the
> > > encoder sending a small (2048 bytes) buffer, the driver returning 2
> > > MDLs very close together, and then the encoder sending only one more
> > > buffer:
> >
> > ...
> >
> > >
> > > I changed the cx18_read() function to exit the loop and return once
> > > it had returned 1 MDL to the encoder, but the problem still
> > > persisted. In this case the short buffer in at least one trial was
> > > 4096 bytes.
> > >
> > >
> >
> > Is it possible to reproduce it by writing a small capture program
> > that
> > acts similar to MythTV? If I had a program like this, then I could
> > investigate.
>

> Yes, if I have time, I can write a program to emulate the critical
> aspects of MythTV's operation. You will have to back out the fix to
> cx18_v4l2_enc_poll() to purposely induce the stall. Also tuning to a
> weak, snowy channel (but not all snow) and changing channels will induce
> the problem more rapidly. I assume the encoder can't compress as well
> under these circumstances, so it sends more data to the host.

Hans,

Attached is the test program for which you asked. With it, you should
be able to reproduce select() timeouts with cx18_v4l2_enc_poll()
modified to not watch q_io. So you can at least get the encoder
transfers to stall for 5 seconds.

I have not been able to get the test program to stall the encoder
transfers long term yet. I'll have to make it more MythTV-like, I
guess, by calling some of the ioctls that MythTV does.

-Andy

Attachments:

pollcx18.c (5.11 KB)

cwalls at radix

Mar 22, 2008, 7:27 PM

Post #6 of 10 (2519 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so [In reply to]

On Fri, 2008-03-21 at 22:04 -0400, Andy Walls wrote:
> On Sun, 2008-03-09 at 16:28 -0400, Andy Walls wrote:
> > Hans Verkuil wrote:
> > > On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> > > > Hans Verkuil wrote:
> > > > > > Here's my observation: There was no good reason for q_full to
> > > > > > stay empty for so long (> 5 seconds), just because data was
> > > > > > sitting queued in q_io. It was as if data stopped being moved
> > > > > > from the encoder to buffers and into q_full. What is both
> > > > > > fortunate and unfortunate is that draining q_full and q_io,
> > > seems
> > > > > > to restart the transfers from the encoder.
>
>
> > > > Argh. I'm giving up on hunting this down for now. Here are my
> > > > observations and speculations with the return value of
> > > > cx18_v4l2_enc_poll() does not depend on q_io.length:
> > > >
> > > > a) The most buffers I've ever seen in use in the driver are 1 in
> > > q_io
> > > > and 10 in q_full. That was when then encoder decided to give the
> > > > driver 10 buffers all at once.
>
>
> > > > b) I can always get the transfer from the encoder to stall.
> > > >
> > > > c) stopping the capture, reloading all the MDLs and restarting the
> > > > capture doesn't make transfers from the encoder work again.
> > > >
> > > > d) a common sequence of failure looked like the following, with the
> > > > encoder sending a small (2048 bytes) buffer, the driver returning 2
> > > > MDLs very close together, and then the encoder sending only one more
> > > > buffer:
> > >
> > > ...
> > >
> > > >
> > > > I changed the cx18_read() function to exit the loop and return once
> > > > it had returned 1 MDL to the encoder, but the problem still
> > > > persisted. In this case the short buffer in at least one trial was
> > > > 4096 bytes.
> > > >
> > > >
> > >
> > > Is it possible to reproduce it by writing a small capture program
> > > that
> > > acts similar to MythTV? If I had a program like this, then I could
> > > investigate.
> >
>
>
> > Yes, if I have time, I can write a program to emulate the critical
> > aspects of MythTV's operation. You will have to back out the fix to
> > cx18_v4l2_enc_poll() to purposely induce the stall. Also tuning to a
> > weak, snowy channel (but not all snow) and changing channels will induce
> > the problem more rapidly. I assume the encoder can't compress as well
> > under these circumstances, so it sends more data to the host.
>
> Hans,
>
> Attached is the test program for which you asked. With it, you should
> be able to reproduce select() timeouts with cx18_v4l2_enc_poll()
> modified to not watch q_io. So you can at least get the encoder
> transfers to stall for 5 seconds.
>
> I have not been able to get the test program to stall the encoder
> transfers long term yet. I'll have to make it more MythTV-like, I
> guess, by calling some of the ioctls that MythTV does.

Hans,

Attached is an updated version of the test program that sets up the
device with ioctl()s as MythTV would. The item that made the difference
on reproducing the "permanent" stall was the setup_vbi() function.

Note that upon further investigation the encoder isn't completely
non-responsive. After closing and reopening the stream, it will produce
one transfer into q_full. If nothing drains off q_full/q_io after that,
it will not produce another transfer until the stream is closed and
reopened again. For the MythTV user experience, this amount to a black
screen and repeated select timeouts.

Again, the fix to cx18_v4l2_enc_poll() to watch q_io now masks this bug.

Regards,
Andy

Attachments:

pollcx18.c (11.8 KB)

hverkuil at xs4all

Apr 6, 2008, 10:42 AM

Post #7 of 10 (2415 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall , but I can't figure out why they do so [In reply to]

On Sunday 23 March 2008 03:27:56 Andy Walls wrote:
> On Fri, 2008-03-21 at 22:04 -0400, Andy Walls wrote:
> > On Sun, 2008-03-09 at 16:28 -0400, Andy Walls wrote:
> > > Hans Verkuil wrote:
> > > > On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> > > > > Hans Verkuil wrote:
> > > > > > > Here's my observation: There was no good reason for
> > > > > > > q_full to stay empty for so long (> 5 seconds), just
> > > > > > > because data was sitting queued in q_io. It was as if
> > > > > > > data stopped being moved from the encoder to buffers and
> > > > > > > into q_full. What is both fortunate and unfortunate is
> > > > > > > that draining q_full and q_io,
> > > >
> > > > seems
> > > >
> > > > > > > to restart the transfers from the encoder.
> > > > >
> > > > > Argh. I'm giving up on hunting this down for now. Here are
> > > > > my observations and speculations with the return value of
> > > > > cx18_v4l2_enc_poll() does not depend on q_io.length:
> > > > >
> > > > > a) The most buffers I've ever seen in use in the driver are 1
> > > > > in
> > > >
> > > > q_io
> > > >
> > > > > and 10 in q_full. That was when then encoder decided to give
> > > > > the driver 10 buffers all at once.
> > > > >
> > > > >
> > > > > b) I can always get the transfer from the encoder to stall.
> > > > >
> > > > > c) stopping the capture, reloading all the MDLs and
> > > > > restarting the capture doesn't make transfers from the
> > > > > encoder work again.
> > > > >
> > > > > d) a common sequence of failure looked like the following,
> > > > > with the encoder sending a small (2048 bytes) buffer, the
> > > > > driver returning 2 MDLs very close together, and then the
> > > > > encoder sending only one more buffer:
> > > >
> > > > ...
> > > >
> > > > > I changed the cx18_read() function to exit the loop and
> > > > > return once it had returned 1 MDL to the encoder, but the
> > > > > problem still persisted. In this case the short buffer in at
> > > > > least one trial was 4096 bytes.
> > > >
> > > > Is it possible to reproduce it by writing a small capture
> > > > program that
> > > > acts similar to MythTV? If I had a program like this, then I
> > > > could investigate.
> > >
> > > Yes, if I have time, I can write a program to emulate the
> > > critical aspects of MythTV's operation. You will have to back
> > > out the fix to cx18_v4l2_enc_poll() to purposely induce the
> > > stall. Also tuning to a weak, snowy channel (but not all snow)
> > > and changing channels will induce the problem more rapidly. I
> > > assume the encoder can't compress as well under these
> > > circumstances, so it sends more data to the host.
> >
> > Hans,
> >
> > Attached is the test program for which you asked. With it, you
> > should be able to reproduce select() timeouts with
> > cx18_v4l2_enc_poll() modified to not watch q_io. So you can at
> > least get the encoder transfers to stall for 5 seconds.
> >
> > I have not been able to get the test program to stall the encoder
> > transfers long term yet. I'll have to make it more MythTV-like, I
> > guess, by calling some of the ioctls that MythTV does.
>
> Hans,
>
> Attached is an updated version of the test program that sets up the
> device with ioctl()s as MythTV would. The item that made the
> difference on reproducing the "permanent" stall was the setup_vbi()
> function.

Thanks Andy,

I can reproduce it but it is not entirely clear what is going on. It
looks like a buffer that isn't thrown back into the MDL pool fast
enough. Or something like that. To be continued.

BTW: check out my latest cx18 tree, apparently ATSC is now working
thanks to the hard work of Steve Toth.

Regards,

Hans

>
> Note that upon further investigation the encoder isn't completely
> non-responsive. After closing and reopening the stream, it will
> produce one transfer into q_full. If nothing drains off q_full/q_io
> after that, it will not produce another transfer until the stream is
> closed and reopened again. For the MythTV user experience, this
> amount to a black screen and repeated select timeouts.
>
> Again, the fix to cx18_v4l2_enc_poll() to watch q_io now masks this
> bug.
>
> Regards,
> Andy

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

cwalls at radix

Apr 6, 2008, 11:04 AM

Post #8 of 10 (2420 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall, but I can't figure out why they do so [In reply to]

Hans Verkuil wrote:
> On Sunday 23 March 2008 03:27:56 Andy Walls wrote:
> > On Fri, 2008-03-21 at 22:04 -0400, Andy Walls wrote:
> > > On Sun, 2008-03-09 at 16:28 -0400, Andy Walls wrote:
> > > > Hans Verkuil wrote:
> > > > > On Sunday 09 March 2008 05:37:04 Andy Walls wrote:
> > > > > > Hans Verkuil wrote:
> > > > > > > > Here's my observation: There was no good reason for
> > > > > > > > q_full to stay empty for so long (> 5 seconds), just
> > > > > > > > because data was sitting queued in q_io. It was as if
> > > > > > > > data stopped being moved from the encoder to buffers and
> > > > > > > > into q_full. What is both fortunate and unfortunate is
> > > > > > > > that draining q_full and q_io,
> > > > >
> > > > > seems
> > > > >
> > > > > > > > to restart the transfers from the encoder.
> > > > > >
> > > > > > Argh. I'm giving up on hunting this down for now. Here are
> > > > > > my observations and speculations with the return value of
> > > > > > cx18_v4l2_enc_poll() does not depend on q_io.length:
> > > > > >
> > > > > > a) The most buffers I've ever seen in use in the driver are 1
> > > > > > in
> > > > >
> > > > > q_io
> > > > >
> > > > > > and 10 in q_full. That was when then encoder decided to give
> > > > > > the driver 10 buffers all at once.
> > > > > >
> > > > > >
> > > > > > b) I can always get the transfer from the encoder to stall.
> > > > > >
> > > > > > c) stopping the capture, reloading all the MDLs and
> > > > > > restarting the capture doesn't make transfers from the
> > > > > > encoder work again.
> > > > > >
> > > > > > d) a common sequence of failure looked like the following,
> > > > > > with the encoder sending a small (2048 bytes) buffer, the
> > > > > > driver returning 2 MDLs very close together, and then the
> > > > > > encoder sending only one more buffer:
> > > > >
> > > > > ...
> > > > >
> > > > > > I changed the cx18_read() function to exit the loop and
> > > > > > return once it had returned 1 MDL to the encoder, but the
> > > > > > problem still persisted. In this case the short buffer in at
> > > > > > least one trial was 4096 bytes.
> > > > >
> > > > > Is it possible to reproduce it by writing a small capture
> > > > > program that
> > > > > acts similar to MythTV? If I had a program like this, then I
> > > > > could investigate.
> > > >
> > > > Yes, if I have time, I can write a program to emulate the
> > > > critical aspects of MythTV's operation. You will have to back
> > > > out the fix to cx18_v4l2_enc_poll() to purposely induce the
> > > > stall. Also tuning to a weak, snowy channel (but not all snow)
> > > > and changing channels will induce the problem more rapidly. I
> > > > assume the encoder can't compress as well under these
> > > > circumstances, so it sends more data to the host.
> > >
> > > Hans,
> > >
> > > Attached is the test program for which you asked. With it, you
> > > should be able to reproduce select() timeouts with
> > > cx18_v4l2_enc_poll() modified to not watch q_io. So you can at
> > > least get the encoder transfers to stall for 5 seconds.
> > >
> > > I have not been able to get the test program to stall the encoder
> > > transfers long term yet. I'll have to make it more MythTV-like, I
> > > guess, by calling some of the ioctls that MythTV does.
> >
> > Hans,
> >
> > Attached is an updated version of the test program that sets up the
> > device with ioctl()s as MythTV would. The item that made the
> > difference on reproducing the "permanent" stall was the setup_vbi()
> > function.
>
> Thanks Andy,
>
> I can reproduce it but it is not entirely clear what is going on. It
> looks like a buffer that isn't thrown back into the MDL pool fast
> enough. Or something like that. To be continued.

Again, if you eliminate the call to setup_vbi() in the test program,
things are "fine".

> BTW: check out my latest cx18 tree, apparently ATSC is now working
> thanks to the hard work of Steve Toth.

I saw Steve making check-ins last night and you making some changes
today, so I was waiting for the "churn" to die down.

Later this evening I may get a chance to test things. Linux's DVB
device nodes and API are new to me, so I'll have to climb the learning
curve. (Hopefully mplayer or MythTV has it figured out for me already.)

I'll only be able to test ATSC 8VSB, as that's the only modulation I
have ever received over the air. I don't have cable, so ATSC QAM
modulations (and the little used 16VSB modulation) have to be tested by
someone else.

> Regards,
>
> Hans
>
> >
> > Note that upon further investigation the encoder isn't completely
> > non-responsive. After closing and reopening the stream, it will
> > produce one transfer into q_full. If nothing drains off q_full/q_io
> > after that, it will not produce another transfer until the stream is
> > closed and reopened again. For the MythTV user experience, this
> > amount to a black screen and repeated select timeouts.
> >
> > Again, the fix to cx18_v4l2_enc_poll() to watch q_io now masks this
> > bug.
> >
> > Regards,
> > Andy

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

jeanfrancois at sagetv

Apr 6, 2008, 12:52 PM

Post #9 of 10 (2415 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall , but I can't figure out why they do so [In reply to]

> BTW: check out my latest cx18 tree, apparently ATSC is now working
> thanks to the hard work of Steve Toth.
>
> Regards,
>
> Hans

Hello Hans,

This seems to be working except only on the first capture. If you modify the
first test in cx18_stop_v4l2_encode_stream to

if (s->v4l2dev == NULL && s->type!=CX18_ENC_STREAM_TYPE_TS)

then it works multiple times.

Thanks

Jean-Francois

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

hverkuil at xs4all

Apr 6, 2008, 10:59 PM

Post #10 of 10 (2402 views)
Permalink

Re: cx18: I can purposely get the transfers from the encoder to stall , but I can't figure out why they do so [In reply to]

On Sunday 06 April 2008 21:52:47 Jean-Francois Thibert wrote:
> > BTW: check out my latest cx18 tree, apparently ATSC is now working
> > thanks to the hard work of Steve Toth.
> >
> > Regards,
> >
> > Hans
>
> Hello Hans,
>
> This seems to be working except only on the first capture. If you
> modify the first test in cx18_stop_v4l2_encode_stream to
>
> if (s->v4l2dev == NULL && s->type!=CX18_ENC_STREAM_TYPE_TS)
>
> then it works multiple times.
>
> Thanks
>
> Jean-Francois

Please try again with my latest tree, it should be fixed.

Regards,

Hans

_______________________________________________
ivtv-devel mailing list
ivtv-devel [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads