Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ivtv: users

Help With Troubleshooting

 

 

ivtv users RSS feed   Index | Next | Previous | View Threaded


jpreston at longlines

Jan 25, 2012, 2:57 PM

Post #1 of 11 (987 views)
Permalink
Help With Troubleshooting

Hello, I have two servers running Mythbuntu 11.04 and MythTV 0.24.1. I have
three PVR-150 tuners in the server configured as a slave backend, and
some digital tuners in the master backend. The problem I have is that
whenever the analog tuners are called for by any frontend (even a local
one), I get distorted video, with stuttering and image freezes. There is
currently a bug report in with MythTV on a “Analog Channel Change” issue.
I have tried the work-a-round (tune a digital channel before using one of
the analog tuners), but it does not work for me. I finally saw some
troubleshooting steps for IVTV drivers, and followed some of those steps.
The one that caught my attention, is the “cat /dev/video1 > test.mpg”
test. I looked at the file that was produced while the command was
active, and VLC would only play a few seconds of the file and then skip
to the end, and then terminate the viewing window. The video jumped and
was distorted, just the same as it was in MythTV. So the issue,
evidently, is not MythTV, but perhaps with IVTV or some related item.
On a related note, the same hardware works without issues, when running
MythTV 0.21 under Mythbuntu 8.10.
What would you do next and where would you look for a resolution? Thanks, Jeff


awalls at md

Jan 25, 2012, 7:40 PM

Post #2 of 11 (967 views)
Permalink
Re: Help With Troubleshooting [In reply to]

jpreston [at] longlines wrote:

>Hello,>>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
>I have
>three PVR-150 tuners in the server configured as a slave backend, and
>some digital tuners in the master backend. The problem I have is that
>whenever the analog tuners are called for by any frontend (even a local
>one), I get distorted video, with stuttering and image freezes. There
>is
>currently a bug report in with MythTV on a �Analog Channel Change�
>issue.
>I have tried the work-a-round (tune a digital channel before using one
>of
>the analog tuners), but it does not work for me. I finally saw some
>troubleshooting steps for IVTV drivers, and followed some of those
>steps.
>The one that caught my attention, is the �cat /dev/video1 > test.mpg�
>test. I looked at the file that was produced while the command was
>active, and VLC would only play a few seconds of the file and then skip
>to the end, and then terminate the viewing window. The video jumped
>and
>was distorted, just the same as it was in MythTV. So the issue,
>evidently, is not MythTV, but perhaps with IVTV or some related item.
>>On a related note, the same hardware works without issues, when
>running
>MythTV 0.21 under Mythbuntu 8.10.
>>What would you do next and where would you look for a
>resolution?>>Thanks,>>Jeff
>
>_______________________________________________
>ivtv-users mailing list
>ivtv-users [at] ivtvdriver
>http://ivtvdriver.org/mailman/listinfo/ivtv-users

What are the kernel versions shown with uname -a?

What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1 --log-status?

What other devices are shown sharing the interrupt lines with ivtv using cat /proc/interrupts?

What is the cpu loading shown with top during video capture? Are there high numbers for user or io wait percentages?

Have you tried loading the ivtv module with various debug flags to get extra logging: modinfo ivtv.

Regards,
Andy W.

_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


jpreston at longlines

Jan 26, 2012, 4:39 PM

Post #3 of 11 (976 views)
Permalink
Re: Help With Troubleshooting [In reply to]

==============Original message text=============== On Wed, 25 Jan 2012 9:40:11 pm CST Andy Walls wrote: jpreston [at] longlines wrote: >>Hello,>>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1. >>I have >>three PVR-150 tuners in the server configured as a slave backend, and >>some digital tuners in the master backend. The problem I have is that >>whenever the analog tuners are called for by any frontend (even a local >>one), I get distorted video, with stuttering and image freezes. There >>is >>currently a bug report in with MythTV on a �Analog Channel Change� >>issue. >>I have tried the work-a-round (tune a digital channel before using one >>of >>the analog tuners), but it does not work for me. I finally saw some >>troubleshooting steps for IVTV drivers, and followed some of those >>steps. >>The one that caught my attention, is the �cat /dev/video1 > test.mpg� >>test. I looked at the file that was produced while the command was >>active, and VLC would only play a few seconds of the file and then skip >>to the end, and then terminate the viewing window. The video jumped >>and >>was distorted, just the same as it was in MythTV. So the issue, >>evidently, is not MythTV, but perhaps with IVTV or some related item. >>On a related note, the same hardware works without issues, when >>running >>MythTV 0.21 under Mythbuntu 8.10. >>What would you do next and where would you look for a >>resolution?>>Thanks,>>Jeff >> >>_______________________________________________ >>ivtv-users mailing list >>ivtv-users [at] ivtvdriver >>http://ivtvdriver.org/mailman/listinfo/ivtv-users >What are the kernel versions shown with uname -a? Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 UTC
2012 i686 i686 i386 GNU/Linux
>What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1
>--log-status?
ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150 >What other devices are shown sharing the interrupt lines with ivtv using
>cat /proc/interrupts?
CPU0 CPU1 0: 580 0 IO-APIC-edge timer 1: 3649 0 IO-APIC-edge i8042 6: 3 0 IO-APIC-edge floppy 7: 0 0 IO-APIC-edge parport0 8: 1 0 IO-APIC-edge rtc0 9: 0 0 IO-APIC-fasteoi acpi 14: 63335 0 IO-APIC-edge ata_piix 15: 1294927 0 IO-APIC-edge ata_piix 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2,
uhci_hcd:usb5, nvidia
17: 12238 7179 IO-APIC-fasteoi ivtv1 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,
eth0, Ensoniq AudioPCI
19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2 22: 1829 0 IO-APIC-fasteoi ivtv0 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1 NMI: 0 0 Non-maskable interrupts LOC: 14667959 15620982 Local timer interrupts SPU: 0 0 Spurious interrupts PMI: 0 0 Performance monitoring interrupts IWI: 0 0 IRQ work interrupts RES: 2407721 2344290 Rescheduling interrupts CAL: 789 2664 Function call interrupts TLB: 13663 20105 TLB shootdowns TRM: 0 0 Thermal event interrupts THR: 0 0 Threshold APIC interrupts MCE: 0 0 Machine check exceptions MCP: 540 540 Machine check polls ERR: 0 MIS: 0 Looks like one of the USB ports shares its IRQ with one of the cards, but
my issue happens with all three cards, so that doesn't seem to be the
likely cause of the issue.
>What is the cpu loading shown with top during video capture? Are there
>high numbers for user or io wait percentages?
Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers Swap: 2095100k total, 0k used, 2095100k free, 688688k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54
xfce4-terminal
1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers This was captured during a cat /dev/video1 > test.mpg, and the CPU usage
never went above a few percent for any process. The file was still
distorted and flawed.
Where else can I look for this? Do you have any other ideas? Thank you much for your assistance so far. Jeff


awalls at md

Jan 28, 2012, 6:25 AM

Post #4 of 11 (966 views)
Permalink
Re: Help With Troubleshooting [In reply to]

On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> ==============Original message text===============
> On Wed, 25 Jan 2012 9:40:11 pm CST Andy Walls wrote:
> jpreston [at] longlines wrote:
> >>Hello,
> >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> >>I have
> >>three PVR-150 tuners in the server configured as a slave backend, and
> >>some digital tuners in the master backend. The problem I have is that
> >>whenever the analog tuners are called for by any frontend (even a local
> >>one), I get distorted video, with stuttering and image freezes. There
> >>is
> >>currently a bug report in with MythTV on a �Analog Channel Change�
> >>issue.
> >>I have tried the work-a-round (tune a digital channel before using one
> >>of
> >>the analog tuners), but it does not work for me. I finally saw some
> >>troubleshooting steps for IVTV drivers, and followed some of those
> >>steps.
> >>The one that caught my attention, is the �cat /dev/video1 > test.mpg�
> >>test. I looked at the file that was produced while the command was
> >>active, and VLC would only play a few seconds of the file and then skip
> >>to the end, and then terminate the viewing window. The video jumped
> >>and
> >>was distorted, just the same as it was in MythTV. So the issue,
> >>evidently, is not MythTV, but perhaps with IVTV or some related item.
> >>On a related note, the same hardware works without issues, when
> >>running
> >>MythTV 0.21 under Mythbuntu 8.10.
> >>What would you do next and where would you look for a>>resolution?
> >>Thanks,
> >>Jeff

> >What are the kernel versions shown with uname -a?

> Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 UTC
> 2012 i686 i686 i386 GNU/Linux

OK, fairly modern; that's good. Do you happen to know the kernel
version under Mythbuntu 8.10? (so I check the differences between the
two kernels)

> >What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1
> >--log-status?

> ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150

OK, good.

> >What other devices are shown sharing the interrupt lines with ivtv using
> >cat /proc/interrupts?

> CPU0 CPU1
> 0: 580 0 IO-APIC-edge timer
> 1: 3649 0 IO-APIC-edge i8042
> 6: 3 0 IO-APIC-edge floppy
> 7: 0 0 IO-APIC-edge parport0
> 8: 1 0 IO-APIC-edge rtc0
> 9: 0 0 IO-APIC-fasteoi acpi
> 14: 63335 0 IO-APIC-edge ata_piix
> 15: 1294927 0 IO-APIC-edge ata_piix
> 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, nvidia
> 17: 12238 7179 IO-APIC-fasteoi ivtv1
> 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> 22: 1829 0 IO-APIC-fasteoi ivtv0
> 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> NMI: 0 0 Non-maskable interrupts
> LOC: 14667959 15620982 Local timer interrupts
> SPU: 0 0 Spurious interrupts
> PMI: 0 0 Performance monitoring interrupts
> IWI: 0 0 IRQ work interrupts
> RES: 2407721 2344290 Rescheduling interrupts
> CAL: 789 2664 Function call interrupts
> TLB: 13663 20105 TLB shootdowns
> TRM: 0 0 Thermal event interrupts
> THR: 0 0 Threshold APIC interrupts
> MCE: 0 0 Machine check exceptions
> MCP: 540 540 Machine check polls
> ERR: 0
> MIS: 0

This looks pretty imbalanced: CPU0 handles a lot more of the interrupt
service than CPU1. This imbalance of CPU0 handling most of the hardware
interrupts might be a source of problems, *if* CPU0 is actually busy
with higher priority interrupts (ata_piix, nvidia) when the ivtv* card
interrupts come in.

The disk drive (ata_piix) and network card (eth0) interrupts have only
been handled by CPU0.

Is "irqbalance" running on your system?

For a back end system, I have to wonder what the nvidia card is doing
for you, assuming it it the only device generating interrupts on IRQ16.
It has genearted 9.98M interrupts in the same time of 30.3M local timer
interrupts (1 nvidia card interrupt per every 3 timer interrupts).



> Looks like one of the USB ports shares its IRQ with one of the cards, but
> my issue happens with all three cards, so that doesn't seem to be the
> likely cause of the issue.

Right, especially if there are no USB devices connected to the port that
IRQ19 serves.

> >What is the cpu loading shown with top during video capture? Are there
> >high numbers for user or io wait percentages?
> Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
> 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54 xfce4-terminal
> 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
> 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
> 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
> 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
> 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
> 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
> 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
> 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
> 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
> 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
> 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
> 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
> 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
> 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers

> This was captured during a cat /dev/video1 > test.mpg, and the CPU usage
> never went above a few percent for any process. The file was still
> distorted and flawed.

OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8%
idle), nothing waiting on I/O (0.0% wa).

'cat' consuming only 1% of CPU may be a little low, since the ivtv
driver and cat collectively do some buffer copies in the read() call
that cat makes into the ivtv driver. (May indicate not many filled
video buffers being provided by the ivtv driver).


Given this, the best guesses I have are:
1. an interrupt handling latency problem.
2. a PCI bus/chipset problem: DMA transfers aborting or IO transactions
to registers and memory on the CX23416 chip are being aborted
3. an ivtv driver bug/race in setting up the CX23416 decoder, that
causes problems in the decoder actually running
4. a cx25840 driver bug in setting up the CX25843 chip to lock on to the
video signal and digitize it properly
5. an analog tuner driver bug that causes tuning to fail

> Where else can I look for this? Do you have any other ideas?

For #4 and #5 above, the output of several runs of 'v4l2-ctl
--log-status' should let you know if the CX25843 has a good video
signal.

For #2 and #3, setting the debug=... option to the ivtv driver
in /etc/modprobe.conf or on the modprobe ivtv command line will turn on
various levels of logging in the ivtv driver (with messages emitted
to /var/log/messages normally).

$ modinfo ivtv
[...]
parm: debug:Debug level (bitmask). Default: 0
1/0x0001: warning
2/0x0002: info
4/0x0004: mailbox
8/0x0008: ioctl
16/0x0010: file
32/0x0020: dma
64/0x0040: irq
128/0x0080: decoder
256/0x0100: yuv
512/0x0200: i2c
1024/0x0400: high volume
[...]

So maybe, as root,

# modprobe -r ivtv
# modprobe ivtv debug=0x4f (or =0x44f)

And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew
in /var/log/messages to see what is wrong.


For #1, I'd experiment with how does video capture behave with

a. having irqbalance running (or not running)
b. booting into only run-level 3 (text console, no GUI/X) and unloading
the nvidia driver module


I would also look at IRQ handling latency with ftrace and other kernel
tracing mechanisms:

http://lwn.net/Articles/322666/
http://lwn.net/Articles/322731/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/365835/
http://www.spinics.net/lists/linux-media/msg15762.html

Regards,
Andy

> Thank you much for your assistance so far.

> Jeff
>



_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


jpreston at longlines

Jan 30, 2012, 1:02 PM

Post #5 of 11 (957 views)
Permalink
Re: Help With Troubleshooting [In reply to]

Kernel version on 8.10 was/is: 2.6.24 - 30-generic

Loading Mythbuntu version 8.10 results in a rock solid system and all three of my PVR150's are happy as clams. Input change between them, and channel change all day without so much as one single hiccup.

The CPU imbalance may be due to the fact that it is a hyperthreaded P4 processor. Based on some reading I did on the PVR150's that another user pointed me toward, there was a theory postulated that P4's with hyperthreading enabled would cause some issues. I tried this, and it seemed to improve things until I started switching channels on one of the other PVR150's. Then I got the same distorted images. I thought I was really onto something there, but I would not be that lucky. I am leaving it off until I track the source of this issue down.

Ivtv version running under 8.10 is 1.1.0

The nVidia card is only in the backend so I can have a local frontend for testing. Once the backend is stable, the high performance video card would come out and be replaced by a passively cooled card.

I get the same sort of processor imbalance with 8.10 loaded running "cat /proc/interrupts" and hyperthreading enabled.

I have tried to adjust the latency timer values on the PVR150's while running under Mythbuntu 11.04, but the 'setpci' command has no effect. I issued the command "sudo setpci -v -s 03:03.0 latency-timer=70", but the timer comes back under "lspci -v" with the same value as before the setpci command. This same command, however, works under version 8.10. Strange....

I am trying to now run ivtv with enhanced logging and see what that brings. What do I look for in the v4l2-ctl --log-status output to know if the card is working properly? Can I track the other cards using that command? How?

I will need more time to track down and use the other suggestions you gave, as I have to learn to use them first. But I will get there. Thanks for your help so far.

-----Original Message-----
From: Andy Walls [mailto:awalls [at] md]
Sent: Saturday, January 28, 2012 8:25 AM
To: jpreston [at] longlines
Cc: ivtv-users [at] ivtvdriver
Subject: Re: [ivtv-users] Help With Troubleshooting

On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> ==============Original message text=============== On Wed, 25 Jan 2012
> 9:40:11 pm CST Andy Walls wrote:
> jpreston [at] longlines wrote:
> >>Hello,
> >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> >>I have
> >>three PVR-150 tuners in the server configured as a slave backend,
> >>and some digital tuners in the master backend. The problem I have
> >>is that whenever the analog tuners are called for by any frontend
> >>(even a local one), I get distorted video, with stuttering and image
> >>freezes. There is currently a bug report in with MythTV on a
> >> Analog Channel Change issue.
> >>I have tried the work-a-round (tune a digital channel before using
> >>one of the analog tuners), but it does not work for me. I finally
> >>saw some troubleshooting steps for IVTV drivers, and followed some
> >>of those steps.
> >>The one that caught my attention, is the cat /dev/video1 >
> >>test.mpg test. I looked at the file that was produced while the
> >>command was active, and VLC would only play a few seconds of the
> >>file and then skip to the end, and then terminate the viewing
> >>window. The video jumped and was distorted, just the same as it was
> >>in MythTV. So the issue, evidently, is not MythTV, but perhaps with
> >>IVTV or some related item.
> >>On a related note, the same hardware works without issues, when
> >>running MythTV 0.21 under Mythbuntu 8.10.
> >>What would you do next and where would you look for a>>resolution?
> >>Thanks,
> >>Jeff

> >What are the kernel versions shown with uname -a?

> Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 UTC
> 2012 i686 i686 i386 GNU/Linux

OK, fairly modern; that's good. Do you happen to know the kernel version under Mythbuntu 8.10? (so I check the differences between the two kernels)

> >What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1
> >--log-status?

> ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150

OK, good.

> >What other devices are shown sharing the interrupt lines with ivtv
> >using cat /proc/interrupts?

> CPU0 CPU1
> 0: 580 0 IO-APIC-edge timer
> 1: 3649 0 IO-APIC-edge i8042
> 6: 3 0 IO-APIC-edge floppy
> 7: 0 0 IO-APIC-edge parport0
> 8: 1 0 IO-APIC-edge rtc0
> 9: 0 0 IO-APIC-fasteoi acpi
> 14: 63335 0 IO-APIC-edge ata_piix
> 15: 1294927 0 IO-APIC-edge ata_piix
> 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, nvidia
> 17: 12238 7179 IO-APIC-fasteoi ivtv1
> 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> 22: 1829 0 IO-APIC-fasteoi ivtv0
> 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> NMI: 0 0 Non-maskable interrupts
> LOC: 14667959 15620982 Local timer interrupts
> SPU: 0 0 Spurious interrupts
> PMI: 0 0 Performance monitoring interrupts
> IWI: 0 0 IRQ work interrupts
> RES: 2407721 2344290 Rescheduling interrupts
> CAL: 789 2664 Function call interrupts
> TLB: 13663 20105 TLB shootdowns
> TRM: 0 0 Thermal event interrupts
> THR: 0 0 Threshold APIC interrupts
> MCE: 0 0 Machine check exceptions
> MCP: 540 540 Machine check polls
> ERR: 0
> MIS: 0

This looks pretty imbalanced: CPU0 handles a lot more of the interrupt service than CPU1. This imbalance of CPU0 handling most of the hardware interrupts might be a source of problems, *if* CPU0 is actually busy with higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts come in.

The disk drive (ata_piix) and network card (eth0) interrupts have only been handled by CPU0.

Is "irqbalance" running on your system?

For a back end system, I have to wonder what the nvidia card is doing for you, assuming it it the only device generating interrupts on IRQ16.
It has genearted 9.98M interrupts in the same time of 30.3M local timer interrupts (1 nvidia card interrupt per every 3 timer interrupts).



> Looks like one of the USB ports shares its IRQ with one of the cards,
> but my issue happens with all three cards, so that doesn't seem to be
> the likely cause of the issue.

Right, especially if there are no USB devices connected to the port that
IRQ19 serves.

> >What is the cpu loading shown with top during video capture? Are
> >there high numbers for user or io wait percentages?
> Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
> 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54 xfce4-terminal
> 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
> 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
> 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
> 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
> 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
> 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
> 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
> 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
> 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
> 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
> 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
> 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
> 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
> 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers

> This was captured during a cat /dev/video1 > test.mpg, and the CPU
> usage never went above a few percent for any process. The file was
> still distorted and flawed.

OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle), nothing waiting on I/O (0.0% wa).

'cat' consuming only 1% of CPU may be a little low, since the ivtv driver and cat collectively do some buffer copies in the read() call that cat makes into the ivtv driver. (May indicate not many filled video buffers being provided by the ivtv driver).


Given this, the best guesses I have are:
1. an interrupt handling latency problem.
2. a PCI bus/chipset problem: DMA transfers aborting or IO transactions to registers and memory on the CX23416 chip are being aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder, that causes problems in the decoder actually running 4. a cx25840 driver bug in setting up the CX25843 chip to lock on to the video signal and digitize it properly 5. an analog tuner driver bug that causes tuning to fail

> Where else can I look for this? Do you have any other ideas?

For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status' should let you know if the CX25843 has a good video signal.

For #2 and #3, setting the debug=... option to the ivtv driver in /etc/modprobe.conf or on the modprobe ivtv command line will turn on various levels of logging in the ivtv driver (with messages emitted to /var/log/messages normally).

$ modinfo ivtv
[...]
parm: debug:Debug level (bitmask). Default: 0
1/0x0001: warning
2/0x0002: info
4/0x0004: mailbox
8/0x0008: ioctl
16/0x0010: file
32/0x0020: dma
64/0x0040: irq
128/0x0080: decoder
256/0x0100: yuv
512/0x0200: i2c
1024/0x0400: high volume
[...]

So maybe, as root,

# modprobe -r ivtv
# modprobe ivtv debug=0x4f (or =0x44f)

And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew in /var/log/messages to see what is wrong.


For #1, I'd experiment with how does video capture behave with

a. having irqbalance running (or not running) b. booting into only run-level 3 (text console, no GUI/X) and unloading the nvidia driver module


I would also look at IRQ handling latency with ftrace and other kernel tracing mechanisms:

http://lwn.net/Articles/322666/
http://lwn.net/Articles/322731/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/365835/
http://www.spinics.net/lists/linux-media/msg15762.html

Regards,
Andy

> Thank you much for your assistance so far.

> Jeff
>




-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date: 01/28/12


_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


jpreston at longlines

Jan 30, 2012, 10:49 PM

Post #6 of 11 (955 views)
Permalink
Re: Help With Troubleshooting [In reply to]

Update:

Tried enabling debug messages on ivtv...

Sudo modprobe ivtv debug=0x44f

But I cannot find where the log file is created. There is no \var\log\messages file or folder.

Tried setting the PCI latency timers to 128 from 64, but it made no difference.

-----Original Message-----
From: Andy Walls [mailto:awalls [at] md]
Sent: Saturday, January 28, 2012 8:25 AM
To: jpreston [at] longlines
Cc: ivtv-users [at] ivtvdriver
Subject: Re: [ivtv-users] Help With Troubleshooting

On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> ==============Original message text=============== On Wed, 25 Jan 2012
> 9:40:11 pm CST Andy Walls wrote:
> jpreston [at] longlines wrote:
> >>Hello,
> >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> >>I have
> >>three PVR-150 tuners in the server configured as a slave backend,
> >>and some digital tuners in the master backend. The problem I have
> >>is that whenever the analog tuners are called for by any frontend
> >>(even a local one), I get distorted video, with stuttering and image
> >>freezes. There is currently a bug report in with MythTV on a
> >> Analog Channel Change issue.
> >>I have tried the work-a-round (tune a digital channel before using
> >>one of the analog tuners), but it does not work for me. I finally
> >>saw some troubleshooting steps for IVTV drivers, and followed some
> >>of those steps.
> >>The one that caught my attention, is the cat /dev/video1 >
> >>test.mpg test. I looked at the file that was produced while the
> >>command was active, and VLC would only play a few seconds of the
> >>file and then skip to the end, and then terminate the viewing
> >>window. The video jumped and was distorted, just the same as it was
> >>in MythTV. So the issue, evidently, is not MythTV, but perhaps with
> >>IVTV or some related item.
> >>On a related note, the same hardware works without issues, when
> >>running MythTV 0.21 under Mythbuntu 8.10.
> >>What would you do next and where would you look for a>>resolution?
> >>Thanks,
> >>Jeff

> >What are the kernel versions shown with uname -a?

> Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 UTC
> 2012 i686 i686 i386 GNU/Linux

OK, fairly modern; that's good. Do you happen to know the kernel version under Mythbuntu 8.10? (so I check the differences between the two kernels)

> >What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1
> >--log-status?

> ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150

OK, good.

> >What other devices are shown sharing the interrupt lines with ivtv
> >using cat /proc/interrupts?

> CPU0 CPU1
> 0: 580 0 IO-APIC-edge timer
> 1: 3649 0 IO-APIC-edge i8042
> 6: 3 0 IO-APIC-edge floppy
> 7: 0 0 IO-APIC-edge parport0
> 8: 1 0 IO-APIC-edge rtc0
> 9: 0 0 IO-APIC-fasteoi acpi
> 14: 63335 0 IO-APIC-edge ata_piix
> 15: 1294927 0 IO-APIC-edge ata_piix
> 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, nvidia
> 17: 12238 7179 IO-APIC-fasteoi ivtv1
> 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> 22: 1829 0 IO-APIC-fasteoi ivtv0
> 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> NMI: 0 0 Non-maskable interrupts
> LOC: 14667959 15620982 Local timer interrupts
> SPU: 0 0 Spurious interrupts
> PMI: 0 0 Performance monitoring interrupts
> IWI: 0 0 IRQ work interrupts
> RES: 2407721 2344290 Rescheduling interrupts
> CAL: 789 2664 Function call interrupts
> TLB: 13663 20105 TLB shootdowns
> TRM: 0 0 Thermal event interrupts
> THR: 0 0 Threshold APIC interrupts
> MCE: 0 0 Machine check exceptions
> MCP: 540 540 Machine check polls
> ERR: 0
> MIS: 0

This looks pretty imbalanced: CPU0 handles a lot more of the interrupt service than CPU1. This imbalance of CPU0 handling most of the hardware interrupts might be a source of problems, *if* CPU0 is actually busy with higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts come in.

The disk drive (ata_piix) and network card (eth0) interrupts have only been handled by CPU0.

Is "irqbalance" running on your system?

For a back end system, I have to wonder what the nvidia card is doing for you, assuming it it the only device generating interrupts on IRQ16.
It has genearted 9.98M interrupts in the same time of 30.3M local timer interrupts (1 nvidia card interrupt per every 3 timer interrupts).



> Looks like one of the USB ports shares its IRQ with one of the cards,
> but my issue happens with all three cards, so that doesn't seem to be
> the likely cause of the issue.

Right, especially if there are no USB devices connected to the port that
IRQ19 serves.

> >What is the cpu loading shown with top during video capture? Are
> >there high numbers for user or io wait percentages?
> Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
> 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54 xfce4-terminal
> 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
> 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
> 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
> 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
> 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
> 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
> 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
> 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
> 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
> 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
> 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
> 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
> 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
> 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers

> This was captured during a cat /dev/video1 > test.mpg, and the CPU
> usage never went above a few percent for any process. The file was
> still distorted and flawed.

OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle), nothing waiting on I/O (0.0% wa).

'cat' consuming only 1% of CPU may be a little low, since the ivtv driver and cat collectively do some buffer copies in the read() call that cat makes into the ivtv driver. (May indicate not many filled video buffers being provided by the ivtv driver).


Given this, the best guesses I have are:
1. an interrupt handling latency problem.
2. a PCI bus/chipset problem: DMA transfers aborting or IO transactions to registers and memory on the CX23416 chip are being aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder, that causes problems in the decoder actually running 4. a cx25840 driver bug in setting up the CX25843 chip to lock on to the video signal and digitize it properly 5. an analog tuner driver bug that causes tuning to fail

> Where else can I look for this? Do you have any other ideas?

For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status' should let you know if the CX25843 has a good video signal.

For #2 and #3, setting the debug=... option to the ivtv driver in /etc/modprobe.conf or on the modprobe ivtv command line will turn on various levels of logging in the ivtv driver (with messages emitted to /var/log/messages normally).

$ modinfo ivtv
[...]
parm: debug:Debug level (bitmask). Default: 0
1/0x0001: warning
2/0x0002: info
4/0x0004: mailbox
8/0x0008: ioctl
16/0x0010: file
32/0x0020: dma
64/0x0040: irq
128/0x0080: decoder
256/0x0100: yuv
512/0x0200: i2c
1024/0x0400: high volume
[...]

So maybe, as root,

# modprobe -r ivtv
# modprobe ivtv debug=0x4f (or =0x44f)

And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew in /var/log/messages to see what is wrong.


For #1, I'd experiment with how does video capture behave with

a. having irqbalance running (or not running) b. booting into only run-level 3 (text console, no GUI/X) and unloading the nvidia driver module


I would also look at IRQ handling latency with ftrace and other kernel tracing mechanisms:

http://lwn.net/Articles/322666/
http://lwn.net/Articles/322731/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/365835/
http://www.spinics.net/lists/linux-media/msg15762.html

Regards,
Andy

> Thank you much for your assistance so far.

> Jeff
>




-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date: 01/28/12


_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


jpreston at longlines

Feb 9, 2012, 9:22 AM

Post #7 of 11 (935 views)
Permalink
Re: Help With Troubleshooting [In reply to]

Andy,

I have tried the following things:

1 - I disabled Hyperthreading on my P4 3.2Ghz CPU, and for several minutes, this seemed to resolve the issue, but then it started happening again, and the picture went distorted, with lots of skipping, and jumping. MythTV would sometimes exit with "Error opening jump program file", or "Video frame buffering failed too many times".

2 - I changed grub and rebooted the machine without the GUI loaded. So this accomplished two things....no nVidia driver loaded and no Window managers running, so a fairly clean system as far as processes running. I started mythbackend, but then it exited at the end (maybe it is supposed to do that). I then started a remote frontend and the first station it tuned to gave me the same trouble. Re-started the system with graphics again.

3 - I ran the command 'v4l2-ctl --log-status' several times, but could see nothing in the output that would give me any idea of the signal quality or strength. Maybe I am not looking at the output correctly.

4 - I issued the command "modprobe ivtv debug=0x4f", but could not find any logs in the folder you specified. I did look at the output from "dmesg" and found what I believe to be the results of the debug output from the ivtv module. However, there is only so much space, it seems, in the output from dmesg, and I could not find anything that was there that would give me a clue as to what is going on. Is there a way to get the debug mode for the ivtv module to redirect its output to a log file, so I can comb through it without worrying about the buffer space in dmesg?

5 - I looked to see if I could load IRQbalance as you suggested, and found that it was already running under Mythbuntu 11.04. I then unloaded it, rebooted and tried again - no difference.

I tried ftrace, but I believe it is beyond my abilities to configure and run at the moment. I do not know how to mount the debug file system or enable the option on the Ubuntu kernel. I have seen the MythTV bug reports, but they do not seem to apply exactly to my situation, however, I cannot help but think that they are at least linked to what is going on with my system. I do not believe it is a latency issue, as adjusting the latency of the cards and other things on my system made no difference in the behavior of the analog tuner (PVR-150's) at all. I also do not think it has anything to do with changing channels, as I have had distorted video come up when first bring up my frontend...any frontend.

I know I should not let this get to me, but I had a working system under Mythbuntu 8.10, and it doesn't work under 11.04. I found that the issue actually began, or has something to do with the changes between Mythbuntu 8.10 and 9.10. I don't know where the changes are located that would affect analog tuners, but I will guarantee you the source of the issue is there. The problem still existed in Mythbuntu 10.10, as well.

Perhaps someone that is familiar with the design architectures of Linux and Mythbuntu could point us all in the right direction.

Andy, I want to thank you for your help so far. I would not have gotten this far without your help. Thanks for taking pity on an aging engineer.

-----Original Message-----
From: Andy Walls [mailto:awalls [at] md]
Sent: Saturday, January 28, 2012 8:25 AM
To: jpreston [at] longlines
Cc: ivtv-users [at] ivtvdriver
Subject: Re: [ivtv-users] Help With Troubleshooting

On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> ==============Original message text=============== On Wed, 25 Jan 2012
> 9:40:11 pm CST Andy Walls wrote:
> jpreston [at] longlines wrote:
> >>Hello,
> >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> >>I have
> >>three PVR-150 tuners in the server configured as a slave backend,
> >>and some digital tuners in the master backend. The problem I have
> >>is that whenever the analog tuners are called for by any frontend
> >>(even a local one), I get distorted video, with stuttering and image
> >>freezes. There is currently a bug report in with MythTV on a
> >> Analog Channel Change issue.
> >>I have tried the work-a-round (tune a digital channel before using
> >>one of the analog tuners), but it does not work for me. I finally
> >>saw some troubleshooting steps for IVTV drivers, and followed some
> >>of those steps.
> >>The one that caught my attention, is the cat /dev/video1 >
> >>test.mpg test. I looked at the file that was produced while the
> >>command was active, and VLC would only play a few seconds of the
> >>file and then skip to the end, and then terminate the viewing
> >>window. The video jumped and was distorted, just the same as it was
> >>in MythTV. So the issue, evidently, is not MythTV, but perhaps with
> >>IVTV or some related item.
> >>On a related note, the same hardware works without issues, when
> >>running MythTV 0.21 under Mythbuntu 8.10.
> >>What would you do next and where would you look for a>>resolution?
> >>Thanks,
> >>Jeff

> >What are the kernel versions shown with uname -a?

> Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 UTC
> 2012 i686 i686 i386 GNU/Linux

OK, fairly modern; that's good. Do you happen to know the kernel version under Mythbuntu 8.10? (so I check the differences between the two kernels)

> >What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1
> >--log-status?

> ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150

OK, good.

> >What other devices are shown sharing the interrupt lines with ivtv
> >using cat /proc/interrupts?

> CPU0 CPU1
> 0: 580 0 IO-APIC-edge timer
> 1: 3649 0 IO-APIC-edge i8042
> 6: 3 0 IO-APIC-edge floppy
> 7: 0 0 IO-APIC-edge parport0
> 8: 1 0 IO-APIC-edge rtc0
> 9: 0 0 IO-APIC-fasteoi acpi
> 14: 63335 0 IO-APIC-edge ata_piix
> 15: 1294927 0 IO-APIC-edge ata_piix
> 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, nvidia
> 17: 12238 7179 IO-APIC-fasteoi ivtv1
> 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> 22: 1829 0 IO-APIC-fasteoi ivtv0
> 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> NMI: 0 0 Non-maskable interrupts
> LOC: 14667959 15620982 Local timer interrupts
> SPU: 0 0 Spurious interrupts
> PMI: 0 0 Performance monitoring interrupts
> IWI: 0 0 IRQ work interrupts
> RES: 2407721 2344290 Rescheduling interrupts
> CAL: 789 2664 Function call interrupts
> TLB: 13663 20105 TLB shootdowns
> TRM: 0 0 Thermal event interrupts
> THR: 0 0 Threshold APIC interrupts
> MCE: 0 0 Machine check exceptions
> MCP: 540 540 Machine check polls
> ERR: 0
> MIS: 0

This looks pretty imbalanced: CPU0 handles a lot more of the interrupt service than CPU1. This imbalance of CPU0 handling most of the hardware interrupts might be a source of problems, *if* CPU0 is actually busy with higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts come in.

The disk drive (ata_piix) and network card (eth0) interrupts have only been handled by CPU0.

Is "irqbalance" running on your system?

For a back end system, I have to wonder what the nvidia card is doing for you, assuming it it the only device generating interrupts on IRQ16.
It has genearted 9.98M interrupts in the same time of 30.3M local timer interrupts (1 nvidia card interrupt per every 3 timer interrupts).



> Looks like one of the USB ports shares its IRQ with one of the cards,
> but my issue happens with all three cards, so that doesn't seem to be
> the likely cause of the issue.

Right, especially if there are no USB devices connected to the port that
IRQ19 serves.

> >What is the cpu loading shown with top during video capture? Are
> >there high numbers for user or io wait percentages?
> Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
> 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54 xfce4-terminal
> 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
> 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
> 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
> 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
> 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
> 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
> 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
> 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
> 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
> 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
> 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
> 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
> 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
> 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers

> This was captured during a cat /dev/video1 > test.mpg, and the CPU
> usage never went above a few percent for any process. The file was
> still distorted and flawed.

OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle), nothing waiting on I/O (0.0% wa).

'cat' consuming only 1% of CPU may be a little low, since the ivtv driver and cat collectively do some buffer copies in the read() call that cat makes into the ivtv driver. (May indicate not many filled video buffers being provided by the ivtv driver).


Given this, the best guesses I have are:
1. an interrupt handling latency problem.
2. a PCI bus/chipset problem: DMA transfers aborting or IO transactions to registers and memory on the CX23416 chip are being aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder, that causes problems in the decoder actually running 4. a cx25840 driver bug in setting up the CX25843 chip to lock on to the video signal and digitize it properly 5. an analog tuner driver bug that causes tuning to fail

> Where else can I look for this? Do you have any other ideas?

For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status' should let you know if the CX25843 has a good video signal.

For #2 and #3, setting the debug=... option to the ivtv driver in /etc/modprobe.conf or on the modprobe ivtv command line will turn on various levels of logging in the ivtv driver (with messages emitted to /var/log/messages normally).

$ modinfo ivtv
[...]
parm: debug:Debug level (bitmask). Default: 0
1/0x0001: warning
2/0x0002: info
4/0x0004: mailbox
8/0x0008: ioctl
16/0x0010: file
32/0x0020: dma
64/0x0040: irq
128/0x0080: decoder
256/0x0100: yuv
512/0x0200: i2c
1024/0x0400: high volume
[...]

So maybe, as root,

# modprobe -r ivtv
# modprobe ivtv debug=0x4f (or =0x44f)

And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew in /var/log/messages to see what is wrong.


For #1, I'd experiment with how does video capture behave with

a. having irqbalance running (or not running) b. booting into only run-level 3 (text console, no GUI/X) and unloading the nvidia driver module


I would also look at IRQ handling latency with ftrace and other kernel tracing mechanisms:

http://lwn.net/Articles/322666/
http://lwn.net/Articles/322731/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/365835/
http://www.spinics.net/lists/linux-media/msg15762.html

Regards,
Andy

> Thank you much for your assistance so far.

> Jeff
>




-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date: 01/28/12


_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


awalls at md

Feb 11, 2012, 9:01 AM

Post #8 of 11 (933 views)
Permalink
Re: Help With Troubleshooting [In reply to]

On Thu, 2012-02-09 at 11:22 -0600, Jeffrey Preston wrote:
> Andy,
>
> I have tried the following things:
>
> 1 - I disabled Hyperthreading on my P4 3.2Ghz CPU, and for several
> minutes, this seemed to resolve the issue, but then it started
> happening again, and the picture went distorted, with lots of
> skipping, and jumping. MythTV would sometimes exit with "Error
> opening jump program file", or "Video frame buffering failed too many
> times".
>
> 2 - I changed grub and rebooted the machine without the GUI loaded.
> So this accomplished two things....no nVidia driver loaded and no
> Window managers running, so a fairly clean system as far as processes
> running. I started mythbackend, but then it exited at the end (maybe
> it is supposed to do that). I then started a remote frontend and the
> first station it tuned to gave me the same trouble. Re-started the
> system with graphics again.
>
> 3 - I ran the command 'v4l2-ctl --log-status' several times, but could
> see nothing in the output that would give me any idea of the signal
> quality or strength. Maybe I am not looking at the output correctly.

You just want to see that a signal is present consistently. I suspect
it was.


> 4 - I issued the command "modprobe ivtv debug=0x4f", but could not
> find any logs in the folder you specified. I did look at the output
> from "dmesg" and found what I believe to be the results of the debug
> output from the ivtv module. However, there is only so much space, it
> seems, in the output from dmesg, and I could not find anything that
> was there that would give me a clue as to what is going on. Is there
> a way to get the debug mode for the ivtv module to redirect its output
> to a log file, so I can comb through it without worrying about the
> buffer space in dmesg?

That's a syslog daemon configuration. The ivtv driver uses KERN_INFO
and KERN_DEBUG (and KERN_ERROR and KERN_WARN) logging facilities.

As root, in /etc/rsyslod.conf or /etc/syslogd.conf you would want to
direct "kern.*" logging to /var/log/kernelmsgs or whatever. Then send
syslod or rsyslogd a SIGHUP to re-read the configuration:

# kill -HUP (process ID of system log daemon)


> 5 - I looked to see if I could load IRQbalance as you suggested, and
> found that it was already running under Mythbuntu 11.04. I then
> unloaded it, rebooted and tried again - no difference.

OK.

> I tried ftrace, but I believe it is beyond my abilities to configure
> and run at the moment. I do not know how to mount the debug file
> system or enable the option on the Ubuntu kernel.

I agree, it is complex. I really don't expect normal users to be able
to use it as an effective troubleshooting tool.

> I have seen the MythTV bug reports, but they do not seem to apply
> exactly to my situation, however, I cannot help but think that they
> are at least linked to what is going on with my system. I do not
> believe it is a latency issue, as adjusting the latency of the cards
> and other things on my system made no difference in the behavior of
> the analog tuner (PVR-150's) at all.

The PCI latency timer is just so that no one card can hog and hang the
PCI bus (remember the days of ISA cards?).

The PCI bus has a Round-Robin arbiter for bus access, and the latency
timer is the maximu amount of time that device can master the bus before
having to yield the bus back to the next bus grantee by the arbiter.

You can only set PCI latency timer values in multiples of 8 BTW. The
PCI clock runs at 33 MHz which makes 8 cycles to be approximately 250
nanoseconds (242 ns actually).

The ivtv driver tries to set the card's latency timer to allow its
internal DMA engine to have the PCI bus for 64 cycles (~2 microseconds)
before the card has to yield the PCI bus to another waiting PCI device.

You might want to check the latency timers of any PCI bridges upstream
of the PVR-150. If their latency timer is shorter than the PVR-150s,
then the latency timer of the PVR-150 doesn't really matter.

> I also do not think it has anything to do with changing channels, as
> I have had distorted video come up when first bring up my
> frontend...any frontend.

> I know I should not let this get to me, but I had a working system
> under Mythbuntu 8.10, and it doesn't work under 11.04. I found that
> the issue actually began, or has something to do with the changes
> between Mythbuntu 8.10 and 9.10. I don't know where the changes are
> located that would affect analog tuners, but I will guarantee you the
> source of the issue is there.

To find the root cause blindly but deterministically, with no
guess-work, one would have to

- be able to reproduce your problem with a known bad kernel
- have a known working kernel version
- do a git bisect on the kernel source between those two versions,
- rebuild the kernel and install it and test it
- iterate the bisection process until the bad kernel change is isolated

Each iteration takes ~30 minutes for me. Unfortunately that's time I
really don't have available. :(


> The problem still existed in Mythbuntu 10.10, as well.
>
> Perhaps someone that is familiar with the design architectures of
> Linux and Mythbuntu could point us all in the right direction.
>
> Andy, I want to thank you for your help so far. I would not have
> gotten this far without your help. Thanks for taking pity on an aging
> engineer.

You're welcome. Sorry I don't have a lot of time available lately.

Aging engineers have to watch out for each other. ;)

Regards,
Andy


> -----Original Message-----
> From: Andy Walls [mailto:awalls [at] md]
> Sent: Saturday, January 28, 2012 8:25 AM
> To: jpreston [at] longlines
> Cc: ivtv-users [at] ivtvdriver
> Subject: Re: [ivtv-users] Help With Troubleshooting
>
> On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> > ==============Original message text=============== On Wed, 25 Jan 2012
> > 9:40:11 pm CST Andy Walls wrote:
> > jpreston [at] longlines wrote:
> > >>Hello,
> > >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> > >>I have
> > >>three PVR-150 tuners in the server configured as a slave backend,
> > >>and some digital tuners in the master backend. The problem I have
> > >>is that whenever the analog tuners are called for by any frontend
> > >>(even a local one), I get distorted video, with stuttering and image
> > >>freezes. There is currently a bug report in with MythTV on a
> > >> Analog Channel Change issue.
> > >>I have tried the work-a-round (tune a digital channel before using
> > >>one of the analog tuners), but it does not work for me. I finally
> > >>saw some troubleshooting steps for IVTV drivers, and followed some
> > >>of those steps.
> > >>The one that caught my attention, is the cat /dev/video1 >
> > >>test.mpg test. I looked at the file that was produced while the
> > >>command was active, and VLC would only play a few seconds of the
> > >>file and then skip to the end, and then terminate the viewing
> > >>window. The video jumped and was distorted, just the same as it was
> > >>in MythTV. So the issue, evidently, is not MythTV, but perhaps with
> > >>IVTV or some related item.
> > >>On a related note, the same hardware works without issues, when
> > >>running MythTV 0.21 under Mythbuntu 8.10.
> > >>What would you do next and where would you look for a>>resolution?
> > >>Thanks,
> > >>Jeff
>
> > >What are the kernel versions shown with uname -a?
>
> > Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 UTC
> > 2012 i686 i686 i386 GNU/Linux
>
> OK, fairly modern; that's good. Do you happen to know the kernel version under Mythbuntu 8.10? (so I check the differences between the two kernels)
>
> > >What are the ivtv-driver versions shown with v4l2-ctl -d /dev/video1
> > >--log-status?
>
> > ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150
>
> OK, good.
>
> > >What other devices are shown sharing the interrupt lines with ivtv
> > >using cat /proc/interrupts?
>
> > CPU0 CPU1
> > 0: 580 0 IO-APIC-edge timer
> > 1: 3649 0 IO-APIC-edge i8042
> > 6: 3 0 IO-APIC-edge floppy
> > 7: 0 0 IO-APIC-edge parport0
> > 8: 1 0 IO-APIC-edge rtc0
> > 9: 0 0 IO-APIC-fasteoi acpi
> > 14: 63335 0 IO-APIC-edge ata_piix
> > 15: 1294927 0 IO-APIC-edge ata_piix
> > 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, nvidia
> > 17: 12238 7179 IO-APIC-fasteoi ivtv1
> > 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> > 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> > 22: 1829 0 IO-APIC-fasteoi ivtv0
> > 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> > NMI: 0 0 Non-maskable interrupts
> > LOC: 14667959 15620982 Local timer interrupts
> > SPU: 0 0 Spurious interrupts
> > PMI: 0 0 Performance monitoring interrupts
> > IWI: 0 0 IRQ work interrupts
> > RES: 2407721 2344290 Rescheduling interrupts
> > CAL: 789 2664 Function call interrupts
> > TLB: 13663 20105 TLB shootdowns
> > TRM: 0 0 Thermal event interrupts
> > THR: 0 0 Threshold APIC interrupts
> > MCE: 0 0 Machine check exceptions
> > MCP: 540 540 Machine check polls
> > ERR: 0
> > MIS: 0
>
> This looks pretty imbalanced: CPU0 handles a lot more of the interrupt service than CPU1. This imbalance of CPU0 handling most of the hardware interrupts might be a source of problems, *if* CPU0 is actually busy with higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts come in.
>
> The disk drive (ata_piix) and network card (eth0) interrupts have only been handled by CPU0.
>
> Is "irqbalance" running on your system?
>
> For a back end system, I have to wonder what the nvidia card is doing for you, assuming it it the only device generating interrupts on IRQ16.
> It has genearted 9.98M interrupts in the same time of 30.3M local timer interrupts (1 nvidia card interrupt per every 3 timer interrupts).
>
>
>
> > Looks like one of the USB ports shares its IRQ with one of the cards,
> > but my issue happens with all three cards, so that doesn't seem to be
> > the likely cause of the issue.
>
> Right, especially if there are no USB devices connected to the port that
> IRQ19 serves.
>
> > >What is the cpu loading shown with top during video capture? Are
> > >there high numbers for user or io wait percentages?
> > Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> > Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> > Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
> > 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54 xfce4-terminal
> > 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
> > 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
> > 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
> > 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
> > 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
> > 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
> > 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
> > 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
> > 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
> > 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
> > 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
> > 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
> > 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
> > 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
> > 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers
>
> > This was captured during a cat /dev/video1 > test.mpg, and the CPU
> > usage never went above a few percent for any process. The file was
> > still distorted and flawed.
>
> OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle), nothing waiting on I/O (0.0% wa).
>
> 'cat' consuming only 1% of CPU may be a little low, since the ivtv driver and cat collectively do some buffer copies in the read() call that cat makes into the ivtv driver. (May indicate not many filled video buffers being provided by the ivtv driver).
>
>
> Given this, the best guesses I have are:
> 1. an interrupt handling latency problem.
> 2. a PCI bus/chipset problem: DMA transfers aborting or IO transactions to registers and memory on the CX23416 chip are being aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder, that causes problems in the decoder actually running 4. a cx25840 driver bug in setting up the CX25843 chip to lock on to the video signal and digitize it properly 5. an analog tuner driver bug that causes tuning to fail
>
> > Where else can I look for this? Do you have any other ideas?
>
> For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status' should let you know if the CX25843 has a good video signal.
>
> For #2 and #3, setting the debug=... option to the ivtv driver in /etc/modprobe.conf or on the modprobe ivtv command line will turn on various levels of logging in the ivtv driver (with messages emitted to /var/log/messages normally).
>
> $ modinfo ivtv
> [...]
> parm: debug:Debug level (bitmask). Default: 0
> 1/0x0001: warning
> 2/0x0002: info
> 4/0x0004: mailbox
> 8/0x0008: ioctl
> 16/0x0010: file
> 32/0x0020: dma
> 64/0x0040: irq
> 128/0x0080: decoder
> 256/0x0100: yuv
> 512/0x0200: i2c
> 1024/0x0400: high volume
> [...]
>
> So maybe, as root,
>
> # modprobe -r ivtv
> # modprobe ivtv debug=0x4f (or =0x44f)
>
> And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew in /var/log/messages to see what is wrong.
>
>
> For #1, I'd experiment with how does video capture behave with
>
> a. having irqbalance running (or not running) b. booting into only run-level 3 (text console, no GUI/X) and unloading the nvidia driver module
>
>
> I would also look at IRQ handling latency with ftrace and other kernel tracing mechanisms:
>
> http://lwn.net/Articles/322666/
> http://lwn.net/Articles/322731/
> http://lwn.net/Articles/366796/
> http://lwn.net/Articles/365835/
> http://www.spinics.net/lists/linux-media/msg15762.html
>
> Regards,
> Andy
>
> > Thank you much for your assistance so far.
>
> > Jeff
> >
>
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date: 01/28/12
>



_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


jpreston at longlines

Feb 13, 2012, 10:07 AM

Post #9 of 11 (914 views)
Permalink
Re: Help With Troubleshooting [In reply to]

Andy,

Thank you for all the help. I will use whatever resources I can in order to isolate and resolve this issue. To further the research a little, I decided to try the Mythdora distro, and found some very interesting results. Loading up and trying version 12.23 got me a 100% working system right away, without any modifications. I then performed updates, and it all went to hell. Didn't get the jumping distorted pictures, but it would lock up at a blank screen anytime I tried live TV. I am going to go back and check the version of the backend / frontends, where it was working and not, but then this could be an entirely new issue, and not even related to the issue I am hunting down.

I will post the information here when I get a chance. Maybe it will help...

-----Original Message-----
From: Andy Walls [mailto:awalls [at] md]
Sent: Saturday, February 11, 2012 11:02 AM
To: Jeffrey Preston
Cc: ivtv-users [at] ivtvdriver
Subject: RE: [ivtv-users] Help With Troubleshooting

On Thu, 2012-02-09 at 11:22 -0600, Jeffrey Preston wrote:
> Andy,
>
> I have tried the following things:
>
> 1 - I disabled Hyperthreading on my P4 3.2Ghz CPU, and for several
> minutes, this seemed to resolve the issue, but then it started
> happening again, and the picture went distorted, with lots of
> skipping, and jumping. MythTV would sometimes exit with "Error
> opening jump program file", or "Video frame buffering failed too many
> times".
>
> 2 - I changed grub and rebooted the machine without the GUI loaded.
> So this accomplished two things....no nVidia driver loaded and no
> Window managers running, so a fairly clean system as far as processes
> running. I started mythbackend, but then it exited at the end (maybe
> it is supposed to do that). I then started a remote frontend and the
> first station it tuned to gave me the same trouble. Re-started the
> system with graphics again.
>
> 3 - I ran the command 'v4l2-ctl --log-status' several times, but could
> see nothing in the output that would give me any idea of the signal
> quality or strength. Maybe I am not looking at the output correctly.

You just want to see that a signal is present consistently. I suspect it was.


> 4 - I issued the command "modprobe ivtv debug=0x4f", but could not
> find any logs in the folder you specified. I did look at the output
> from "dmesg" and found what I believe to be the results of the debug
> output from the ivtv module. However, there is only so much space, it
> seems, in the output from dmesg, and I could not find anything that
> was there that would give me a clue as to what is going on. Is there
> a way to get the debug mode for the ivtv module to redirect its output
> to a log file, so I can comb through it without worrying about the
> buffer space in dmesg?

That's a syslog daemon configuration. The ivtv driver uses KERN_INFO and KERN_DEBUG (and KERN_ERROR and KERN_WARN) logging facilities.

As root, in /etc/rsyslod.conf or /etc/syslogd.conf you would want to direct "kern.*" logging to /var/log/kernelmsgs or whatever. Then send syslod or rsyslogd a SIGHUP to re-read the configuration:

# kill -HUP (process ID of system log daemon)


> 5 - I looked to see if I could load IRQbalance as you suggested, and
> found that it was already running under Mythbuntu 11.04. I then
> unloaded it, rebooted and tried again - no difference.

OK.

> I tried ftrace, but I believe it is beyond my abilities to configure
> and run at the moment. I do not know how to mount the debug file
> system or enable the option on the Ubuntu kernel.

I agree, it is complex. I really don't expect normal users to be able to use it as an effective troubleshooting tool.

> I have seen the MythTV bug reports, but they do not seem to apply
> exactly to my situation, however, I cannot help but think that they
> are at least linked to what is going on with my system. I do not
> believe it is a latency issue, as adjusting the latency of the cards
> and other things on my system made no difference in the behavior of
> the analog tuner (PVR-150's) at all.

The PCI latency timer is just so that no one card can hog and hang the PCI bus (remember the days of ISA cards?).

The PCI bus has a Round-Robin arbiter for bus access, and the latency timer is the maximu amount of time that device can master the bus before having to yield the bus back to the next bus grantee by the arbiter.

You can only set PCI latency timer values in multiples of 8 BTW. The PCI clock runs at 33 MHz which makes 8 cycles to be approximately 250 nanoseconds (242 ns actually).

The ivtv driver tries to set the card's latency timer to allow its internal DMA engine to have the PCI bus for 64 cycles (~2 microseconds) before the card has to yield the PCI bus to another waiting PCI device.

You might want to check the latency timers of any PCI bridges upstream of the PVR-150. If their latency timer is shorter than the PVR-150s, then the latency timer of the PVR-150 doesn't really matter.

> I also do not think it has anything to do with changing channels, as
> I have had distorted video come up when first bring up my
> frontend...any frontend.

> I know I should not let this get to me, but I had a working system
> under Mythbuntu 8.10, and it doesn't work under 11.04. I found that
> the issue actually began, or has something to do with the changes
> between Mythbuntu 8.10 and 9.10. I don't know where the changes are
> located that would affect analog tuners, but I will guarantee you the
> source of the issue is there.

To find the root cause blindly but deterministically, with no guess-work, one would have to

- be able to reproduce your problem with a known bad kernel
- have a known working kernel version
- do a git bisect on the kernel source between those two versions,
- rebuild the kernel and install it and test it
- iterate the bisection process until the bad kernel change is isolated

Each iteration takes ~30 minutes for me. Unfortunately that's time I really don't have available. :(


> The problem still existed in Mythbuntu 10.10, as well.
>
> Perhaps someone that is familiar with the design architectures of
> Linux and Mythbuntu could point us all in the right direction.
>
> Andy, I want to thank you for your help so far. I would not have
> gotten this far without your help. Thanks for taking pity on an aging
> engineer.

You're welcome. Sorry I don't have a lot of time available lately.

Aging engineers have to watch out for each other. ;)

Regards,
Andy


> -----Original Message-----
> From: Andy Walls [mailto:awalls [at] md]
> Sent: Saturday, January 28, 2012 8:25 AM
> To: jpreston [at] longlines
> Cc: ivtv-users [at] ivtvdriver
> Subject: Re: [ivtv-users] Help With Troubleshooting
>
> On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> > ==============Original message text=============== On Wed, 25 Jan
> > 2012
> > 9:40:11 pm CST Andy Walls wrote:
> > jpreston [at] longlines wrote:
> > >>Hello,
> > >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> > >>I have
> > >>three PVR-150 tuners in the server configured as a slave backend,
> > >>and some digital tuners in the master backend. The problem I have
> > >>is that whenever the analog tuners are called for by any frontend
> > >>(even a local one), I get distorted video, with stuttering and
> > >>image freezes. There is currently a bug report in with MythTV on
> > >>a Analog Channel Change issue.
> > >>I have tried the work-a-round (tune a digital channel before using
> > >>one of the analog tuners), but it does not work for me. I finally
> > >>saw some troubleshooting steps for IVTV drivers, and followed some
> > >>of those steps.
> > >>The one that caught my attention, is the cat /dev/video1 >
> > >>test.mpg test. I looked at the file that was produced while the
> > >>command was active, and VLC would only play a few seconds of the
> > >>file and then skip to the end, and then terminate the viewing
> > >>window. The video jumped and was distorted, just the same as it
> > >>was in MythTV. So the issue, evidently, is not MythTV, but
> > >>perhaps with IVTV or some related item.
> > >>On a related note, the same hardware works without issues, when
> > >>running MythTV 0.21 under Mythbuntu 8.10.
> > >>What would you do next and where would you look for a>>resolution?
> > >>Thanks,
> > >>Jeff
>
> > >What are the kernel versions shown with uname -a?
>
> > Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52
> > UTC
> > 2012 i686 i686 i386 GNU/Linux
>
> OK, fairly modern; that's good. Do you happen to know the kernel
> version under Mythbuntu 8.10? (so I check the differences between the
> two kernels)
>
> > >What are the ivtv-driver versions shown with v4l2-ctl -d
> > >/dev/video1 --log-status?
>
> > ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150
>
> OK, good.
>
> > >What other devices are shown sharing the interrupt lines with ivtv
> > >using cat /proc/interrupts?
>
> > CPU0 CPU1
> > 0: 580 0 IO-APIC-edge timer
> > 1: 3649 0 IO-APIC-edge i8042
> > 6: 3 0 IO-APIC-edge floppy
> > 7: 0 0 IO-APIC-edge parport0
> > 8: 1 0 IO-APIC-edge rtc0
> > 9: 0 0 IO-APIC-fasteoi acpi
> > 14: 63335 0 IO-APIC-edge ata_piix
> > 15: 1294927 0 IO-APIC-edge ata_piix
> > 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, nvidia
> > 17: 12238 7179 IO-APIC-fasteoi ivtv1
> > 18: 5690498 0 IO-APIC-fasteoi ata_piix, uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> > 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> > 22: 1829 0 IO-APIC-fasteoi ivtv0
> > 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> > NMI: 0 0 Non-maskable interrupts
> > LOC: 14667959 15620982 Local timer interrupts
> > SPU: 0 0 Spurious interrupts
> > PMI: 0 0 Performance monitoring interrupts
> > IWI: 0 0 IRQ work interrupts
> > RES: 2407721 2344290 Rescheduling interrupts
> > CAL: 789 2664 Function call interrupts
> > TLB: 13663 20105 TLB shootdowns
> > TRM: 0 0 Thermal event interrupts
> > THR: 0 0 Threshold APIC interrupts
> > MCE: 0 0 Machine check exceptions
> > MCP: 540 540 Machine check polls
> > ERR: 0
> > MIS: 0
>
> This looks pretty imbalanced: CPU0 handles a lot more of the interrupt service than CPU1. This imbalance of CPU0 handling most of the hardware interrupts might be a source of problems, *if* CPU0 is actually busy with higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts come in.
>
> The disk drive (ata_piix) and network card (eth0) interrupts have only been handled by CPU0.
>
> Is "irqbalance" running on your system?
>
> For a back end system, I have to wonder what the nvidia card is doing for you, assuming it it the only device generating interrupts on IRQ16.
> It has genearted 9.98M interrupts in the same time of 30.3M local timer interrupts (1 nvidia card interrupt per every 3 timer interrupts).
>
>
>
> > Looks like one of the USB ports shares its IRQ with one of the
> > cards, but my issue happens with all three cards, so that doesn't
> > seem to be the likely cause of the issue.
>
> Right, especially if there are no USB devices connected to the port
> that
> IRQ19 serves.
>
> > >What is the cpu loading shown with top during video capture? Are
> > >there high numbers for user or io wait percentages?
> > Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> > Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> > Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
> > 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54 xfce4-terminal
> > 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
> > 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
> > 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
> > 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88 xscreensaver
> > 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
> > 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
> > 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
> > 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
> > 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
> > 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
> > 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
> > 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
> > 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
> > 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
> > 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers
>
> > This was captured during a cat /dev/video1 > test.mpg, and the CPU
> > usage never went above a few percent for any process. The file was
> > still distorted and flawed.
>
> OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle), nothing waiting on I/O (0.0% wa).
>
> 'cat' consuming only 1% of CPU may be a little low, since the ivtv driver and cat collectively do some buffer copies in the read() call that cat makes into the ivtv driver. (May indicate not many filled video buffers being provided by the ivtv driver).

>
>
> Given this, the best guesses I have are:
> 1. an interrupt handling latency problem.
> 2. a PCI bus/chipset problem: DMA transfers aborting or IO
> transactions to registers and memory on the CX23416 chip are being
> aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder,
> that causes problems in the decoder actually running 4. a cx25840
> driver bug in setting up the CX25843 chip to lock on to the video
> signal and digitize it properly 5. an analog tuner driver bug that
> causes tuning to fail
>
> > Where else can I look for this? Do you have any other ideas?
>
> For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status' should let you know if the CX25843 has a good video signal.
>
> For #2 and #3, setting the debug=... option to the ivtv driver in /etc/modprobe.conf or on the modprobe ivtv command line will turn on various levels of logging in the ivtv driver (with messages emitted to /var/log/messages normally).
>
> $ modinfo ivtv
> [...]
> parm: debug:Debug level (bitmask). Default: 0
> 1/0x0001: warning
> 2/0x0002: info
> 4/0x0004: mailbox
> 8/0x0008: ioctl
> 16/0x0010: file
> 32/0x0020: dma
> 64/0x0040: irq
> 128/0x0080: decoder
> 256/0x0100: yuv
> 512/0x0200: i2c
> 1024/0x0400: high volume
> [...]
>
> So maybe, as root,
>
> # modprobe -r ivtv
> # modprobe ivtv debug=0x4f (or =0x44f)
>
> And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew in /var/log/messages to see what is wrong.
>
>
> For #1, I'd experiment with how does video capture behave with
>
> a. having irqbalance running (or not running) b. booting into only
> run-level 3 (text console, no GUI/X) and unloading the nvidia driver
> module
>
>
> I would also look at IRQ handling latency with ftrace and other kernel tracing mechanisms:
>
> http://lwn.net/Articles/322666/
> http://lwn.net/Articles/322731/
> http://lwn.net/Articles/366796/
> http://lwn.net/Articles/365835/
> http://www.spinics.net/lists/linux-media/msg15762.html
>
> Regards,
> Andy
>
> > Thank you much for your assistance so far.
>
> > Jeff
> >
>
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date:
> 01/28/12
>




-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2112/4804 - Release Date: 02/11/12


_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


jpreston at longlines

Feb 16, 2012, 4:58 PM

Post #10 of 11 (904 views)
Permalink
Re: Help With Troubleshooting [In reply to]

OK,

I have experimented by loading Ubuntu 11.04, and then used Synaptic to load
MythTV. I found that I had a stable system with my three PVR-150's and
perfect video coming from all three tuners. However, about once every dozen
or so channel changes, I get the error "Video frame buffering failed too
many times.", and it exits out to the main menu. Going right back into
LiveTV again results in perfect video, at the channel that was changed to
prior to the error, EVERY time. It was my impression, that Mythbuntu was
basically Ubuntu 11.04 and MythTV combined, but when I load Mythbuntu, it
results in an system that gives me jumping, distorted video, with errors
"Video frame buffering failed too many times" or "Error opening jump program
file", almost every time I change channels.

Does have any suggestions on where to look next? I am thinking now, that
IVTV is probably not at fault, but I cannot rule it out.

-----Original Message-----
From: ivtv-users-bounces [at] ivtvdriver
[mailto:ivtv-users-bounces [at] ivtvdriver] On Behalf Of Jeffrey Preston
Sent: Monday, February 13, 2012 12:08 PM
To: 'Andy Walls'
Cc: ivtv-users [at] ivtvdriver
Subject: Re: [ivtv-users] Help With Troubleshooting

Andy,

Thank you for all the help. I will use whatever resources I can in order to
isolate and resolve this issue. To further the research a little, I decided
to try the Mythdora distro, and found some very interesting results.
Loading up and trying version 12.23 got me a 100% working system right away,
without any modifications. I then performed updates, and it all went to
hell. Didn't get the jumping distorted pictures, but it would lock up at a
blank screen anytime I tried live TV. I am going to go back and check the
version of the backend / frontends, where it was working and not, but then
this could be an entirely new issue, and not even related to the issue I am
hunting down.

I will post the information here when I get a chance. Maybe it will help...

-----Original Message-----
From: Andy Walls [mailto:awalls [at] md]
Sent: Saturday, February 11, 2012 11:02 AM
To: Jeffrey Preston
Cc: ivtv-users [at] ivtvdriver
Subject: RE: [ivtv-users] Help With Troubleshooting

On Thu, 2012-02-09 at 11:22 -0600, Jeffrey Preston wrote:
> Andy,
>
> I have tried the following things:
>
> 1 - I disabled Hyperthreading on my P4 3.2Ghz CPU, and for several
> minutes, this seemed to resolve the issue, but then it started
> happening again, and the picture went distorted, with lots of
> skipping, and jumping. MythTV would sometimes exit with "Error
> opening jump program file", or "Video frame buffering failed too many
> times".
>
> 2 - I changed grub and rebooted the machine without the GUI loaded.
> So this accomplished two things....no nVidia driver loaded and no
> Window managers running, so a fairly clean system as far as processes
> running. I started mythbackend, but then it exited at the end (maybe
> it is supposed to do that). I then started a remote frontend and the
> first station it tuned to gave me the same trouble. Re-started the
> system with graphics again.
>
> 3 - I ran the command 'v4l2-ctl --log-status' several times, but could
> see nothing in the output that would give me any idea of the signal
> quality or strength. Maybe I am not looking at the output correctly.

You just want to see that a signal is present consistently. I suspect it
was.


> 4 - I issued the command "modprobe ivtv debug=0x4f", but could not
> find any logs in the folder you specified. I did look at the output
> from "dmesg" and found what I believe to be the results of the debug
> output from the ivtv module. However, there is only so much space, it
> seems, in the output from dmesg, and I could not find anything that
> was there that would give me a clue as to what is going on. Is there
> a way to get the debug mode for the ivtv module to redirect its output
> to a log file, so I can comb through it without worrying about the
> buffer space in dmesg?

That's a syslog daemon configuration. The ivtv driver uses KERN_INFO and
KERN_DEBUG (and KERN_ERROR and KERN_WARN) logging facilities.

As root, in /etc/rsyslod.conf or /etc/syslogd.conf you would want to direct
"kern.*" logging to /var/log/kernelmsgs or whatever. Then send syslod or
rsyslogd a SIGHUP to re-read the configuration:

# kill -HUP (process ID of system log daemon)


> 5 - I looked to see if I could load IRQbalance as you suggested, and
> found that it was already running under Mythbuntu 11.04. I then
> unloaded it, rebooted and tried again - no difference.

OK.

> I tried ftrace, but I believe it is beyond my abilities to configure
> and run at the moment. I do not know how to mount the debug file
> system or enable the option on the Ubuntu kernel.

I agree, it is complex. I really don't expect normal users to be able to
use it as an effective troubleshooting tool.

> I have seen the MythTV bug reports, but they do not seem to apply
> exactly to my situation, however, I cannot help but think that they
> are at least linked to what is going on with my system. I do not
> believe it is a latency issue, as adjusting the latency of the cards
> and other things on my system made no difference in the behavior of
> the analog tuner (PVR-150's) at all.

The PCI latency timer is just so that no one card can hog and hang the PCI
bus (remember the days of ISA cards?).

The PCI bus has a Round-Robin arbiter for bus access, and the latency timer
is the maximu amount of time that device can master the bus before having to
yield the bus back to the next bus grantee by the arbiter.

You can only set PCI latency timer values in multiples of 8 BTW. The PCI
clock runs at 33 MHz which makes 8 cycles to be approximately 250
nanoseconds (242 ns actually).

The ivtv driver tries to set the card's latency timer to allow its internal
DMA engine to have the PCI bus for 64 cycles (~2 microseconds) before the
card has to yield the PCI bus to another waiting PCI device.

You might want to check the latency timers of any PCI bridges upstream of
the PVR-150. If their latency timer is shorter than the PVR-150s, then the
latency timer of the PVR-150 doesn't really matter.

> I also do not think it has anything to do with changing channels, as
> I have had distorted video come up when first bring up my
> frontend...any frontend.

> I know I should not let this get to me, but I had a working system
> under Mythbuntu 8.10, and it doesn't work under 11.04. I found that
> the issue actually began, or has something to do with the changes
> between Mythbuntu 8.10 and 9.10. I don't know where the changes are
> located that would affect analog tuners, but I will guarantee you the
> source of the issue is there.

To find the root cause blindly but deterministically, with no guess-work,
one would have to

- be able to reproduce your problem with a known bad kernel
- have a known working kernel version
- do a git bisect on the kernel source between those two versions,
- rebuild the kernel and install it and test it
- iterate the bisection process until the bad kernel change is isolated

Each iteration takes ~30 minutes for me. Unfortunately that's time I really
don't have available. :(


> The problem still existed in Mythbuntu 10.10, as well.
>
> Perhaps someone that is familiar with the design architectures of
> Linux and Mythbuntu could point us all in the right direction.
>
> Andy, I want to thank you for your help so far. I would not have
> gotten this far without your help. Thanks for taking pity on an aging
> engineer.

You're welcome. Sorry I don't have a lot of time available lately.

Aging engineers have to watch out for each other. ;)

Regards,
Andy


> -----Original Message-----
> From: Andy Walls [mailto:awalls [at] md]
> Sent: Saturday, January 28, 2012 8:25 AM
> To: jpreston [at] longlines
> Cc: ivtv-users [at] ivtvdriver
> Subject: Re: [ivtv-users] Help With Troubleshooting
>
> On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
> > ==============Original message text=============== On Wed, 25 Jan
> > 2012
> > 9:40:11 pm CST Andy Walls wrote:
> > jpreston [at] longlines wrote:
> > >>Hello,
> > >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> > >>I have
> > >>three PVR-150 tuners in the server configured as a slave backend,
> > >>and some digital tuners in the master backend. The problem I have
> > >>is that whenever the analog tuners are called for by any frontend
> > >>(even a local one), I get distorted video, with stuttering and
> > >>image freezes. There is currently a bug report in with MythTV on
> > >>a Analog Channel Change issue.
> > >>I have tried the work-a-round (tune a digital channel before using
> > >>one of the analog tuners), but it does not work for me. I finally
> > >>saw some troubleshooting steps for IVTV drivers, and followed some
> > >>of those steps.
> > >>The one that caught my attention, is the cat /dev/video1 >
> > >>test.mpg test. I looked at the file that was produced while the
> > >>command was active, and VLC would only play a few seconds of the
> > >>file and then skip to the end, and then terminate the viewing
> > >>window. The video jumped and was distorted, just the same as it
> > >>was in MythTV. So the issue, evidently, is not MythTV, but
> > >>perhaps with IVTV or some related item.
> > >>On a related note, the same hardware works without issues, when
> > >>running MythTV 0.21 under Mythbuntu 8.10.
> > >>What would you do next and where would you look for a>>resolution?
> > >>Thanks,
> > >>Jeff
>
> > >What are the kernel versions shown with uname -a?
>
> > Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52
> > UTC
> > 2012 i686 i686 i386 GNU/Linux
>
> OK, fairly modern; that's good. Do you happen to know the kernel
> version under Mythbuntu 8.10? (so I check the differences between the
> two kernels)
>
> > >What are the ivtv-driver versions shown with v4l2-ctl -d
> > >/dev/video1 --log-status?
>
> > ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150
>
> OK, good.
>
> > >What other devices are shown sharing the interrupt lines with ivtv
> > >using cat /proc/interrupts?
>
> > CPU0 CPU1
> > 0: 580 0 IO-APIC-edge timer
> > 1: 3649 0 IO-APIC-edge i8042
> > 6: 3 0 IO-APIC-edge floppy
> > 7: 0 0 IO-APIC-edge parport0
> > 8: 1 0 IO-APIC-edge rtc0
> > 9: 0 0 IO-APIC-fasteoi acpi
> > 14: 63335 0 IO-APIC-edge ata_piix
> > 15: 1294927 0 IO-APIC-edge ata_piix
> > 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2,
uhci_hcd:usb5, nvidia
> > 17: 12238 7179 IO-APIC-fasteoi ivtv1
> > 18: 5690498 0 IO-APIC-fasteoi ata_piix,
uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> > 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
> > 22: 1829 0 IO-APIC-fasteoi ivtv0
> > 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
> > NMI: 0 0 Non-maskable interrupts
> > LOC: 14667959 15620982 Local timer interrupts
> > SPU: 0 0 Spurious interrupts
> > PMI: 0 0 Performance monitoring interrupts
> > IWI: 0 0 IRQ work interrupts
> > RES: 2407721 2344290 Rescheduling interrupts
> > CAL: 789 2664 Function call interrupts
> > TLB: 13663 20105 TLB shootdowns
> > TRM: 0 0 Thermal event interrupts
> > THR: 0 0 Threshold APIC interrupts
> > MCE: 0 0 Machine check exceptions
> > MCP: 540 540 Machine check polls
> > ERR: 0
> > MIS: 0
>
> This looks pretty imbalanced: CPU0 handles a lot more of the interrupt
service than CPU1. This imbalance of CPU0 handling most of the hardware
interrupts might be a source of problems, *if* CPU0 is actually busy with
higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts
come in.
>
> The disk drive (ata_piix) and network card (eth0) interrupts have only
been handled by CPU0.
>
> Is "irqbalance" running on your system?
>
> For a back end system, I have to wonder what the nvidia card is doing for
you, assuming it it the only device generating interrupts on IRQ16.
> It has genearted 9.98M interrupts in the same time of 30.3M local timer
interrupts (1 nvidia card interrupt per every 3 timer interrupts).
>
>
>
> > Looks like one of the USB ports shares its IRQ with one of the
> > cards, but my issue happens with all three cards, so that doesn't
> > seem to be the likely cause of the issue.
>
> Right, especially if there are no USB devices connected to the port
> that
> IRQ19 serves.
>
> > >What is the cpu loading shown with top during video capture? Are
> > >there high numbers for user or io wait percentages?
> > Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
> > Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
> > Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

> > 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4

> > 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54
xfce4-terminal
> > 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg

> > 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat

> > 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top

> > 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88
xscreensaver
> > 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init

> > 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd

> > 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0

> > 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0

> > 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0

> > 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1

> > 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1

> > 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset

> > 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper

> > 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns

> > 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers
>
> > This was captured during a cat /dev/video1 > test.mpg, and the CPU
> > usage never went above a few percent for any process. The file was
> > still distorted and flawed.
>
> OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle),
nothing waiting on I/O (0.0% wa).
>
> 'cat' consuming only 1% of CPU may be a little low, since the ivtv driver
and cat collectively do some buffer copies in the read() call that cat makes
into the ivtv driver. (May indicate not many filled video buffers being
provided by the ivtv driver).


>
>
> Given this, the best guesses I have are:
> 1. an interrupt handling latency problem.
> 2. a PCI bus/chipset problem: DMA transfers aborting or IO
> transactions to registers and memory on the CX23416 chip are being
> aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder,
> that causes problems in the decoder actually running 4. a cx25840
> driver bug in setting up the CX25843 chip to lock on to the video
> signal and digitize it properly 5. an analog tuner driver bug that
> causes tuning to fail
>
> > Where else can I look for this? Do you have any other ideas?
>
> For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status'
should let you know if the CX25843 has a good video signal.
>
> For #2 and #3, setting the debug=... option to the ivtv driver in
/etc/modprobe.conf or on the modprobe ivtv command line will turn on various
levels of logging in the ivtv driver (with messages emitted to
/var/log/messages normally).
>
> $ modinfo ivtv
> [...]
> parm: debug:Debug level (bitmask). Default: 0
> 1/0x0001: warning
> 2/0x0002: info
> 4/0x0004: mailbox
> 8/0x0008: ioctl
> 16/0x0010: file
> 32/0x0020: dma
> 64/0x0040: irq
> 128/0x0080: decoder
> 256/0x0100: yuv
> 512/0x0200: i2c
> 1024/0x0400: high volume
> [...]
>
> So maybe, as root,
>
> # modprobe -r ivtv
> # modprobe ivtv debug=0x4f (or =0x44f)
>
> And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew
in /var/log/messages to see what is wrong.
>
>
> For #1, I'd experiment with how does video capture behave with
>
> a. having irqbalance running (or not running) b. booting into only
> run-level 3 (text console, no GUI/X) and unloading the nvidia driver
> module
>
>
> I would also look at IRQ handling latency with ftrace and other kernel
tracing mechanisms:
>
> http://lwn.net/Articles/322666/
> http://lwn.net/Articles/322731/
> http://lwn.net/Articles/366796/
> http://lwn.net/Articles/365835/
> http://www.spinics.net/lists/linux-media/msg15762.html
>
> Regards,
> Andy
>
> > Thank you much for your assistance so far.
>
> > Jeff
> >
>
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date:
> 01/28/12
>




-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2112/4804 - Release Date: 02/11/12


_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2112/4807 - Release Date: 02/13/12


_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users


peter.schneider at schneideris

Feb 16, 2012, 7:28 PM

Post #11 of 11 (920 views)
Permalink
Re: Help With Troubleshooting [In reply to]

I have seen the same problem and backed out of the NVIDIA proprietary driver that I think may have been causing it. I am on 11.10 and the NVIDIA driver has been a disaster for me with it. The big down side is the lack of VDPUW.

I am thinking of moving to Debian instead to see if it gets me past the issues with Ubuntu.

Sent from my iPhone

On 2012-02-16, at 5:58 PM, "Jeffrey Preston" <jpreston [at] longlines> wrote:

> OK,
>
> I have experimented by loading Ubuntu 11.04, and then used Synaptic to load
> MythTV. I found that I had a stable system with my three PVR-150's and
> perfect video coming from all three tuners. However, about once every dozen
> or so channel changes, I get the error "Video frame buffering failed too
> many times.", and it exits out to the main menu. Going right back into
> LiveTV again results in perfect video, at the channel that was changed to
> prior to the error, EVERY time. It was my impression, that Mythbuntu was
> basically Ubuntu 11.04 and MythTV combined, but when I load Mythbuntu, it
> results in an system that gives me jumping, distorted video, with errors
> "Video frame buffering failed too many times" or "Error opening jump program
> file", almost every time I change channels.
>
> Does have any suggestions on where to look next? I am thinking now, that
> IVTV is probably not at fault, but I cannot rule it out.
>
> -----Original Message-----
> From: ivtv-users-bounces [at] ivtvdriver
> [mailto:ivtv-users-bounces [at] ivtvdriver] On Behalf Of Jeffrey Preston
> Sent: Monday, February 13, 2012 12:08 PM
> To: 'Andy Walls'
> Cc: ivtv-users [at] ivtvdriver
> Subject: Re: [ivtv-users] Help With Troubleshooting
>
> Andy,
>
> Thank you for all the help. I will use whatever resources I can in order to
> isolate and resolve this issue. To further the research a little, I decided
> to try the Mythdora distro, and found some very interesting results.
> Loading up and trying version 12.23 got me a 100% working system right away,
> without any modifications. I then performed updates, and it all went to
> hell. Didn't get the jumping distorted pictures, but it would lock up at a
> blank screen anytime I tried live TV. I am going to go back and check the
> version of the backend / frontends, where it was working and not, but then
> this could be an entirely new issue, and not even related to the issue I am
> hunting down.
>
> I will post the information here when I get a chance. Maybe it will help...
>
> -----Original Message-----
> From: Andy Walls [mailto:awalls [at] md]
> Sent: Saturday, February 11, 2012 11:02 AM
> To: Jeffrey Preston
> Cc: ivtv-users [at] ivtvdriver
> Subject: RE: [ivtv-users] Help With Troubleshooting
>
> On Thu, 2012-02-09 at 11:22 -0600, Jeffrey Preston wrote:
>> Andy,
>>
>> I have tried the following things:
>>
>> 1 - I disabled Hyperthreading on my P4 3.2Ghz CPU, and for several
>> minutes, this seemed to resolve the issue, but then it started
>> happening again, and the picture went distorted, with lots of
>> skipping, and jumping. MythTV would sometimes exit with "Error
>> opening jump program file", or "Video frame buffering failed too many
>> times".
>>
>> 2 - I changed grub and rebooted the machine without the GUI loaded.
>> So this accomplished two things....no nVidia driver loaded and no
>> Window managers running, so a fairly clean system as far as processes
>> running. I started mythbackend, but then it exited at the end (maybe
>> it is supposed to do that). I then started a remote frontend and the
>> first station it tuned to gave me the same trouble. Re-started the
>> system with graphics again.
>>
>> 3 - I ran the command 'v4l2-ctl --log-status' several times, but could
>> see nothing in the output that would give me any idea of the signal
>> quality or strength. Maybe I am not looking at the output correctly.
>
> You just want to see that a signal is present consistently. I suspect it
> was.
>
>
>> 4 - I issued the command "modprobe ivtv debug=0x4f", but could not
>> find any logs in the folder you specified. I did look at the output
>> from "dmesg" and found what I believe to be the results of the debug
>> output from the ivtv module. However, there is only so much space, it
>> seems, in the output from dmesg, and I could not find anything that
>> was there that would give me a clue as to what is going on. Is there
>> a way to get the debug mode for the ivtv module to redirect its output
>> to a log file, so I can comb through it without worrying about the
>> buffer space in dmesg?
>
> That's a syslog daemon configuration. The ivtv driver uses KERN_INFO and
> KERN_DEBUG (and KERN_ERROR and KERN_WARN) logging facilities.
>
> As root, in /etc/rsyslod.conf or /etc/syslogd.conf you would want to direct
> "kern.*" logging to /var/log/kernelmsgs or whatever. Then send syslod or
> rsyslogd a SIGHUP to re-read the configuration:
>
> # kill -HUP (process ID of system log daemon)
>
>
>> 5 - I looked to see if I could load IRQbalance as you suggested, and
>> found that it was already running under Mythbuntu 11.04. I then
>> unloaded it, rebooted and tried again - no difference.
>
> OK.
>
>> I tried ftrace, but I believe it is beyond my abilities to configure
>> and run at the moment. I do not know how to mount the debug file
>> system or enable the option on the Ubuntu kernel.
>
> I agree, it is complex. I really don't expect normal users to be able to
> use it as an effective troubleshooting tool.
>
>> I have seen the MythTV bug reports, but they do not seem to apply
>> exactly to my situation, however, I cannot help but think that they
>> are at least linked to what is going on with my system. I do not
>> believe it is a latency issue, as adjusting the latency of the cards
>> and other things on my system made no difference in the behavior of
>> the analog tuner (PVR-150's) at all.
>
> The PCI latency timer is just so that no one card can hog and hang the PCI
> bus (remember the days of ISA cards?).
>
> The PCI bus has a Round-Robin arbiter for bus access, and the latency timer
> is the maximu amount of time that device can master the bus before having to
> yield the bus back to the next bus grantee by the arbiter.
>
> You can only set PCI latency timer values in multiples of 8 BTW. The PCI
> clock runs at 33 MHz which makes 8 cycles to be approximately 250
> nanoseconds (242 ns actually).
>
> The ivtv driver tries to set the card's latency timer to allow its internal
> DMA engine to have the PCI bus for 64 cycles (~2 microseconds) before the
> card has to yield the PCI bus to another waiting PCI device.
>
> You might want to check the latency timers of any PCI bridges upstream of
> the PVR-150. If their latency timer is shorter than the PVR-150s, then the
> latency timer of the PVR-150 doesn't really matter.
>
>> I also do not think it has anything to do with changing channels, as
>> I have had distorted video come up when first bring up my
>> frontend...any frontend.
>
>> I know I should not let this get to me, but I had a working system
>> under Mythbuntu 8.10, and it doesn't work under 11.04. I found that
>> the issue actually began, or has something to do with the changes
>> between Mythbuntu 8.10 and 9.10. I don't know where the changes are
>> located that would affect analog tuners, but I will guarantee you the
>> source of the issue is there.
>
> To find the root cause blindly but deterministically, with no guess-work,
> one would have to
>
> - be able to reproduce your problem with a known bad kernel
> - have a known working kernel version
> - do a git bisect on the kernel source between those two versions,
> - rebuild the kernel and install it and test it
> - iterate the bisection process until the bad kernel change is isolated
>
> Each iteration takes ~30 minutes for me. Unfortunately that's time I really
> don't have available. :(
>
>
>> The problem still existed in Mythbuntu 10.10, as well.
>>
>> Perhaps someone that is familiar with the design architectures of
>> Linux and Mythbuntu could point us all in the right direction.
>>
>> Andy, I want to thank you for your help so far. I would not have
>> gotten this far without your help. Thanks for taking pity on an aging
>> engineer.
>
> You're welcome. Sorry I don't have a lot of time available lately.
>
> Aging engineers have to watch out for each other. ;)
>
> Regards,
> Andy
>
>
>> -----Original Message-----
>> From: Andy Walls [mailto:awalls [at] md]
>> Sent: Saturday, January 28, 2012 8:25 AM
>> To: jpreston [at] longlines
>> Cc: ivtv-users [at] ivtvdriver
>> Subject: Re: [ivtv-users] Help With Troubleshooting
>>
>> On Thu, 2012-01-26 at 18:39 -0600, jpreston [at] longlines wrote:
>>> ==============Original message text=============== On Wed, 25 Jan
>>> 2012
>>> 9:40:11 pm CST Andy Walls wrote:
>>> jpreston [at] longlines wrote:
>>>>> Hello,
>>>>> I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
>>>>> I have
>>>>> three PVR-150 tuners in the server configured as a slave backend,
>>>>> and some digital tuners in the master backend. The problem I have
>>>>> is that whenever the analog tuners are called for by any frontend
>>>>> (even a local one), I get distorted video, with stuttering and
>>>>> image freezes. There is currently a bug report in with MythTV on
>>>>> a Analog Channel Change issue.
>>>>> I have tried the work-a-round (tune a digital channel before using
>>>>> one of the analog tuners), but it does not work for me. I finally
>>>>> saw some troubleshooting steps for IVTV drivers, and followed some
>>>>> of those steps.
>>>>> The one that caught my attention, is the cat /dev/video1 >
>>>>> test.mpg test. I looked at the file that was produced while the
>>>>> command was active, and VLC would only play a few seconds of the
>>>>> file and then skip to the end, and then terminate the viewing
>>>>> window. The video jumped and was distorted, just the same as it
>>>>> was in MythTV. So the issue, evidently, is not MythTV, but
>>>>> perhaps with IVTV or some related item.
>>>>> On a related note, the same hardware works without issues, when
>>>>> running MythTV 0.21 under Mythbuntu 8.10.
>>>>> What would you do next and where would you look for a>>resolution?
>>>>> Thanks,
>>>>> Jeff
>>
>>>> What are the kernel versions shown with uname -a?
>>
>>> Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52
>>> UTC
>>> 2012 i686 i686 i386 GNU/Linux
>>
>> OK, fairly modern; that's good. Do you happen to know the kernel
>> version under Mythbuntu 8.10? (so I check the differences between the
>> two kernels)
>>
>>>> What are the ivtv-driver versions shown with v4l2-ctl -d
>>>> /dev/video1 --log-status?
>>
>>> ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150
>>
>> OK, good.
>>
>>>> What other devices are shown sharing the interrupt lines with ivtv
>>>> using cat /proc/interrupts?
>>
>>> CPU0 CPU1
>>> 0: 580 0 IO-APIC-edge timer
>>> 1: 3649 0 IO-APIC-edge i8042
>>> 6: 3 0 IO-APIC-edge floppy
>>> 7: 0 0 IO-APIC-edge parport0
>>> 8: 1 0 IO-APIC-edge rtc0
>>> 9: 0 0 IO-APIC-fasteoi acpi
>>> 14: 63335 0 IO-APIC-edge ata_piix
>>> 15: 1294927 0 IO-APIC-edge ata_piix
>>> 16: 7153671 2827260 IO-APIC-fasteoi uhci_hcd:usb2,
> uhci_hcd:usb5, nvidia
>>> 17: 12238 7179 IO-APIC-fasteoi ivtv1
>>> 18: 5690498 0 IO-APIC-fasteoi ata_piix,
> uhci_hcd:usb4,eth0, Ensoniq AudioPCI
>>> 19: 74042 0 IO-APIC-fasteoi uhci_hcd:usb3, ivtv2
>>> 22: 1829 0 IO-APIC-fasteoi ivtv0
>>> 23: 3 0 IO-APIC-fasteoi ehci_hcd:usb1
>>> NMI: 0 0 Non-maskable interrupts
>>> LOC: 14667959 15620982 Local timer interrupts
>>> SPU: 0 0 Spurious interrupts
>>> PMI: 0 0 Performance monitoring interrupts
>>> IWI: 0 0 IRQ work interrupts
>>> RES: 2407721 2344290 Rescheduling interrupts
>>> CAL: 789 2664 Function call interrupts
>>> TLB: 13663 20105 TLB shootdowns
>>> TRM: 0 0 Thermal event interrupts
>>> THR: 0 0 Threshold APIC interrupts
>>> MCE: 0 0 Machine check exceptions
>>> MCP: 540 540 Machine check polls
>>> ERR: 0
>>> MIS: 0
>>
>> This looks pretty imbalanced: CPU0 handles a lot more of the interrupt
> service than CPU1. This imbalance of CPU0 handling most of the hardware
> interrupts might be a source of problems, *if* CPU0 is actually busy with
> higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts
> come in.
>>
>> The disk drive (ata_piix) and network card (eth0) interrupts have only
> been handled by CPU0.
>>
>> Is "irqbalance" running on your system?
>>
>> For a back end system, I have to wonder what the nvidia card is doing for
> you, assuming it it the only device generating interrupts on IRQ16.
>> It has genearted 9.98M interrupts in the same time of 30.3M local timer
> interrupts (1 nvidia card interrupt per every 3 timer interrupts).
>>
>>
>>
>>> Looks like one of the USB ports shares its IRQ with one of the
>>> cards, but my issue happens with all three cards, so that doesn't
>>> seem to be the likely cause of the issue.
>>
>> Right, especially if there are no USB devices connected to the port
>> that
>> IRQ19 serves.
>>
>>>> What is the cpu loading shown with top during video capture? Are
>>>> there high numbers for user or io wait percentages?
>>> Tasks: 146 total, 1 running, 145 sleeping, 0 stopped, 0 zombie
>>> Cpu(s): 7.2%us, 4.0%sy, 0.0%ni, 88.8%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
>>> Mem: 2059632k total, 1143540k used, 916092k free, 112388k buffers
>>> Swap: 2095100k total, 0k used, 2095100k free, 688688k cached
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>>> 2799 jpreston 20 0 214m 49m 24m S 2 2.5 36:37.44 knotify4
>
>>> 7959 jpreston 20 0 72220 12m 9.8m S 2 0.6 0:04.54
> xfce4-terminal
>>> 1263 root 20 0 88408 51m 14m S 1 2.6 1:35.49 Xorg
>
>>> 8028 jpreston 20 0 3360 504 440 S 1 0.0 0:00.35 cat
>
>>> 8082 jpreston 20 0 2632 1136 848 R 1 0.1 0:00.08 top
>
>>> 1345 jpreston 20 0 6464 2516 1788 S 0 0.1 0:06.88
> xscreensaver
>>> 1 root 20 0 3028 1792 1216 S 0 0.1 0:00.90 init
>
>>> 2 root 20 0 0 0 0 S 0 0.0 0:00.04 kthreadd
>
>>> 3 root 20 0 0 0 0 S 0 0.0 0:01.95 ksoftirqd/0
>
>>> 5 root 20 0 0 0 0 S 0 0.0 0:03.60 kworker/u:0
>
>>> 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
>
>>> 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
>
>>> 9 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/1
>
>>> 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
>
>>> 12 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
>
>>> 13 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
>
>>> 15 root 20 0 0 0 0 S 0 0.0 0:00.27 sync_supers
>>
>>> This was captured during a cat /dev/video1 > test.mpg, and the CPU
>>> usage never went above a few percent for any process. The file was
>>> still distorted and flawed.
>>
>> OK. Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle),
> nothing waiting on I/O (0.0% wa).
>>
>> 'cat' consuming only 1% of CPU may be a little low, since the ivtv driver
> and cat collectively do some buffer copies in the read() call that cat makes
> into the ivtv driver. (May indicate not many filled video buffers being
> provided by the ivtv driver).
>
>
>>
>>
>> Given this, the best guesses I have are:
>> 1. an interrupt handling latency problem.
>> 2. a PCI bus/chipset problem: DMA transfers aborting or IO
>> transactions to registers and memory on the CX23416 chip are being
>> aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder,
>> that causes problems in the decoder actually running 4. a cx25840
>> driver bug in setting up the CX25843 chip to lock on to the video
>> signal and digitize it properly 5. an analog tuner driver bug that
>> causes tuning to fail
>>
>>> Where else can I look for this? Do you have any other ideas?
>>
>> For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status'
> should let you know if the CX25843 has a good video signal.
>>
>> For #2 and #3, setting the debug=... option to the ivtv driver in
> /etc/modprobe.conf or on the modprobe ivtv command line will turn on various
> levels of logging in the ivtv driver (with messages emitted to
> /var/log/messages normally).
>>
>> $ modinfo ivtv
>> [...]
>> parm: debug:Debug level (bitmask). Default: 0
>> 1/0x0001: warning
>> 2/0x0002: info
>> 4/0x0004: mailbox
>> 8/0x0008: ioctl
>> 16/0x0010: file
>> 32/0x0020: dma
>> 64/0x0040: irq
>> 128/0x0080: decoder
>> 256/0x0100: yuv
>> 512/0x0200: i2c
>> 1024/0x0400: high volume
>> [...]
>>
>> So maybe, as root,
>>
>> # modprobe -r ivtv
>> # modprobe ivtv debug=0x4f (or =0x44f)
>>
>> And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew
> in /var/log/messages to see what is wrong.
>>
>>
>> For #1, I'd experiment with how does video capture behave with
>>
>> a. having irqbalance running (or not running) b. booting into only
>> run-level 3 (text console, no GUI/X) and unloading the nvidia driver
>> module
>>
>>
>> I would also look at IRQ handling latency with ftrace and other kernel
> tracing mechanisms:
>>
>> http://lwn.net/Articles/322666/
>> http://lwn.net/Articles/322731/
>> http://lwn.net/Articles/366796/
>> http://lwn.net/Articles/365835/
>> http://www.spinics.net/lists/linux-media/msg15762.html
>>
>> Regards,
>> Andy
>>
>>> Thank you much for your assistance so far.
>>
>>> Jeff
>>>
>>
>>
>>
>>
>> -----
>> No virus found in this message.
>> Checked by AVG - www.avg.com
>> Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date:
>> 01/28/12
>>
>
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.1913 / Virus Database: 2112/4804 - Release Date: 02/11/12
>
>
> _______________________________________________
> ivtv-users mailing list
> ivtv-users [at] ivtvdriver
> http://ivtvdriver.org/mailman/listinfo/ivtv-users
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.1913 / Virus Database: 2112/4807 - Release Date: 02/13/12
>
>
> _______________________________________________
> ivtv-users mailing list
> ivtv-users [at] ivtvdriver
> http://ivtvdriver.org/mailman/listinfo/ivtv-users

_______________________________________________
ivtv-users mailing list
ivtv-users [at] ivtvdriver
http://ivtvdriver.org/mailman/listinfo/ivtv-users

ivtv users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.