Mailing List Archive: MythTV: Users

Mythbuntu stalling with HDD light solid

Index | Next | Previous | View Threaded

lists at whitehouse

Apr 12, 2010, 3:05 AM

Post #1 of 8 (1141 views)
Permalink

Mythbuntu stalling with HDD light solid

Hello all,

I've spent a long time trying to narrow this down, but I'm really
struggling and could do with any help/suggestions that you can come up with.

I use an AMD 780G board with a 1TB Western Digital Green Power EAS HDD
and a 2.4GHz (or thereabouts) Sempron (Sparta) LE-1200CPU. I have 2 GB
RAM and have just added an nVidia 8600GT 512MB. I was previously using
analogue TV over a pvr150 tuner and a bttv tuner. I have just replaced
the pvr150 and bttv with a Hauppauge hvr-2200.

I have run Mythbuntu for quite a long time. We used 9.10 without too
much trouble. At some point near the beginning of this year (I don't
know what triggered it), Mythbuntu started stalling for 1-3 minutes at a
time, with the hard disk light on solidly and nothing else working. The
system would be virtually unresponsive, but would slowly crawl to a
terminal etc. Then, after about 1-3 mins, it would be back to normal. I
can't see anything that could be causing the problem in top, iotop or
the system logs. It is very difficult as the issue causes a whole bunch
of other issues in the logs (such as buffers filling up or kernel events
timing out), making it hard to pick the symptoms from the cause. On a
practical note, stalls during a recording often stop the recording
taking place, which is pretty unhelpful for a PVR. In top there is often
a very high percentage of "wa", which I understand means that a lot of
things are waiting for I/O.

I have now changed the card to an nVidia and am using the proprietary
drivers. I've replaced the tuner cards. I upgraded the system to Lucid
Beta and downloaded all the latest updates. The problem is still
occurring, even when all I was doing was browsing folders in Thunar. To
give an idea of the problem, running a 42 minute video in VLC took 53
mins with the stalls. The memory looks okay (I have 2GB in it), with
free -m showing most "used", but "cached".

On a few occasions when it has stalled, I have killed mythbackend (sudo
stop mythtv-backend) and the system continues stalling for another
couple of minutes. I had decided that the problem could be the hard
drive itself, because I would get a lot of stalls whenever it was
accessing the drive (such as when browsing in Firefox or copying files)
so I tried to copy all of the files from the disk to an eSATA disk. I
had also heard some rumours about issues with Green Power drives.
Mythbuntu/Thunar said that this would take 42 hours for 200GB over eSATA
(and seemed to be taking about that long for the first couple of hours
or so — it initially said 7ish hours, then I set the ionice to BE7 so
that it could do a recording, but it went up to 42 and didn't speed up
after the recording finished), but rebooting to a live CD of Ubuntu
Lucid Beta 2 did the job in 2 hours. This suggests to me that it isn't
the disk, but that talking to the disk in Mythbuntu has issues that
aren't in Ubuntu. I saw stalls in Mythbuntu before I configured it or
imported my database, which makes me think that it is unlikely a corrupt
database or similar. I'm struggling to think of anything that is
different between Mythbuntu and Ubuntu that I haven't tested and
eliminated as a cause.

Given that it isn't showing in top/iotop, I'm wondering if it is
something really low-level. On the other hand, I would have thought it
would need to be hardware/Mythbuntu-specific or everyone would be
complaining. I've replaced most of the hardware that I thought could be
causing the issues.

If anyone can help me, it would make a huge difference -- even if only
to suggest more diagnostics or logs to review.

Please note that the issue is very intermittent. Sometimes it does it
about every 10 mins and sometimes I can watch whole programs (recorded
or in LiveTV) without it happening.

I couldn't get Mythbuntu Log Grabber to work (it just gives me
http://mythbuntu.pastebin.com/ ), so I did it manually:
/var/log/syslog:
http://mythbuntu.pastebin.com/wvChWtfQ

/var/log/mythtv/mythbackend.log:
http://mythbuntu.pastebin.com/AdtsnkVT

/var/log/mythtv/mythfrontend.log:
http://mythbuntu.pastebin.com/UUya70RT

/var/log/dmesg:
http://mythbuntu.pastebin.com/jVr1JV5i

lspci:
http://mythbuntu.pastebin.com/ecPJk4Tz

Please let me know if anything else would be helpful.

The most recent stalls that I have recorded were approximately:
9 Apr 06:50
9 Apr 22:10
11 Apr 21:10
12 Apr 20:12

I am also tracking this issue at:
https://answers.launchpad.net/mythbuntu/+question/106466
in the hope that the next person who comes across a similar issue will
not have to bother everyone.

Thanks in advance,

Aaron
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

support at drdos

Apr 12, 2010, 5:29 AM

Post #2 of 8 (1111 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

Aaron Whitehouse wrote:
>
> On a few occasions when it has stalled, I have killed mythbackend
> (sudo stop mythtv-backend) and the system continues stalling for
> another couple of minutes. I had decided that the problem could be the
> hard drive itself, because I would get a lot of stalls whenever it was
> accessing

That would be my guess. What does dmesg|grep -i dma show?

Sounds like the drive is going bad, you'll probably see a lot of dma
timeouts. You may want to install smartmontools and use:

smartctl -H /dev/sd(x) (Or whatever your drive is mapped to)

smartctl --all /dev/sd(x)

And see if the drive is reporting errors.

Doug

--

Ben Franklin quote:

"Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety."

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

justin.johnson3 at gmail

Apr 12, 2010, 10:33 AM

Post #3 of 8 (1085 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

On Mon, Apr 12, 2010 at 7:29 AM, Doug Lytle <support [at] drdos> wrote:
> Aaron Whitehouse wrote:
>>
>> On a few occasions when it has stalled, I have killed mythbackend (sudo
>> stop mythtv-backend) and the system continues stalling for another couple of
>> minutes. I had decided that the problem could be the hard drive itself,
>> because I would get a lot of stalls whenever it was accessing
>
> That would be my guess. What does dmesg|grep -i dma show?
>
> Sounds like the drive is going bad, you'll probably see a lot of dma
> timeouts. You may want to install smartmontools and use:
>
> smartctl -H /dev/sd(x) (Or whatever your drive is mapped to)
>
> smartctl --all /dev/sd(x)
>
> And see if the drive is reporting errors.
>
> Doug
>

I had a similar issue with my WD EADS 2TB drive shortly after I bought
it, though my "timeouts" were only a few seconds. I didn't know what
was causing it and just let it go. After a couple weeks the entire
system pretty much failed. I couldn't do anything. After I got a
replacement drive from WD, I checked the original with smartctl and
found an incredible number of pending sectors and a number of other
errors. I second the suggestion to check smartctl. On a side note, I
highly recommend setting up smartd and have it e-mail you when it
finds a problem.
--Justin Johnson
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

lists at whitehouse

Apr 14, 2010, 3:05 AM

Post #4 of 8 (1042 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

Thanks very much for your reply!

On Mon, 12 Apr 2010 08:29:19 -0400, Doug Lytle <support [at] drdos> wrote:
> Aaron Whitehouse wrote:
>> I had decided that the problem could be the hard drive itself
> That would be my guess. What does dmesg|grep -i dma show?

$ dmesg|grep -i dma
[ 0.000000] DMA 0x00000000 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 101 pages reserved
[ 0.000000] DMA zone: 3837 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 7105 pages used for memmap
[ 0.000000] DMA32 zone: 512543 pages, LIFO batch:31
[ 0.000000] Policy zone: DMA32
[ 1.407342] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xfa00 irq
14
[ 1.407345] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xfa08 irq
15
[ 1.412451] ata3: SATA max UDMA/133 abar m1024 [at] 0xfe02f00 port
0xfe02f100 irq 22
[ 1.412455] ata4: SATA max UDMA/133 abar m1024 [at] 0xfe02f00 port
0xfe02f180 irq 22
[ 1.412459] ata5: SATA max UDMA/133 abar m1024 [at] 0xfe02f00 port
0xfe02f200 irq 22
[ 1.412463] ata6: SATA max UDMA/133 abar m1024 [at] 0xfe02f00 port
0xfe02f280 irq 22
[ 2.120252] ata6.00: ATAPI: ATAPI iHAS120 6, 7L0F, max UDMA/100
[ 2.120938] ata6.00: configured for UDMA/100
[ 2.130472] ata3.00: ATA-8: WDC WD10EADS-00P8B0, 01.00A01, max UDMA/133
[ 2.132978] ata3.00: configured for UDMA/133
[ 14.022630] parport0: PC-style at 0x378 (0x778), irq 7, dma 3
[PCSPP,TRISTATE,COMPAT,ECP,DMA]

> Sounds like the drive is going bad, you'll probably see a lot of dma
> timeouts. You may want to install smartmontools and [...]

I had already installed smartmontools and thought that the output looked
okay, but I'm not very good at reading it.

$ sudo smartctl -H /dev/sda
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

> smartctl --all /dev/sd(x)
sudo smartctl --all /dev/sda > smartctl_all.log
gave the attached file.

Thanks again for your help.

If nobody can come up with anything better, I'll try moving to a different
HDD this weekend -- it's just a bit of a pain with a 1TB drive (and only a
500GB spare).

Regards,

Aaron

Attachments:

smartctl_all.log (4.51 KB)

mythtv-list at dinkum

Apr 14, 2010, 3:43 AM

Post #5 of 8 (1038 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

On 14 Apr 2010, at 11:05, <lists [at] whitehouse> <lists [at] whitehouse> wrote:
>
> If nobody can come up with anything better, I'll try moving to a different
> HDD this weekend -- it's just a bit of a pain with a 1TB drive (and only a
> 500GB spare).

There's a lot of faqs around about those WD drives, especially in raid arrays but they pause for long periods during sector remaps and often the linux sata driver will decide the drive is bad in this time. There's some problems with power management too, I'd go looking for firmware updates and workarounds before swapping it out.

I had some similar trouble with six Samsung F1's since the kernel update in Ubuntu 9.10, some sata controller drivers are a long way from perfect too, some combinations of drive firmware and controller driver get very smelly.

Andre
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

support at drdos

Apr 14, 2010, 4:07 AM

Post #6 of 8 (1036 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

lists [at] whitehouse wrote:
>
> I had already installed smartmontools and thought that the output looked
> okay, but I'm not very good at reading it.
>
>

The logs show that the drive is fine.

Doug

--

Ben Franklin quote:

"Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety."

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

rkulagow at gmail

Apr 14, 2010, 7:20 AM

Post #7 of 8 (1194 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

Interestingly enough, I'm also experiencing something similar lately
and I haven't gotten to the bottom of it. Mythbuntu 10.04 Beta 2

3ware 9000 Storage Controller device driver for Linux v2.26.02.012.
Drives are configured as JBOD.

I've also noticed that dmesg used to tell me what each drive was
(Samsung, Seagate, etc), but now I've got:
[ 57.086475] 3w-9xxx: scsi4: ERROR: (0x03:0x0101): Invalid command
opcode:opcode=0x85.
[ 57.094863] 3w-9xxx: scsi4: ERROR: (0x03:0x0101): Invalid command
opcode:opcode=0x85.
[ 57.116270] 3w-9xxx: scsi4: ERROR: (0x03:0x0101): Invalid command
opcode:opcode=0x85.
[ 57.118369] 3w-9xxx: scsi4: ERROR: (0x03:0x0101): Invalid command
opcode:opcode=0x85.
[ 57.127340] 3w-9xxx: scsi4: ERROR: (0x03:0x0101): Invalid command
opcode:opcode=0x85.
[ 57.133543] 3w-9xxx: scsi4: ERROR: (0x03:0x0101): Invalid command
opcode:opcode=0x85.

Installed drives are a mix of Samsung, WD and Seagate.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

lists at whitehouse

Apr 25, 2010, 2:40 AM

Post #8 of 8 (908 views)
Permalink

Re: Mythbuntu stalling with HDD light solid [In reply to]

Doug Lytle wrote:
> The logs show that the drive is fine.

It seems that, after all of that, it was a hardware issue. I have
replaced the hard drive (a WD Green 1TB) with a spare one that I had and
all is working really well. It's a pity that I can't get the hours back,
but I'm really glad to have found and solved the problem.

That drive was itself a warranty replacement because the first one
didn't work past 10GB. It doesn't leave me with a lot of love for WD.
For anyone else out there, there are specific ranges of drives designed
for 24/7 video recording called "AV" for WD and "SV" for Seagate.

It was interesting to me that SMART didn't show anything.

Thanks everyone for your help!

Regards,

Aaron
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads