Mailing List Archive: Xen: Users

Gplpv drivers make reset raid controller

Index | Next | Previous | View Threaded

matthieu.lejeune at exxoss

Jun 18, 2012, 11:22 AM

Post #1 of 6 (388 views)
Permalink

Gplpv drivers make reset raid controller

Hi,

I have a very stange problem with my xen 4.0.1

Debian squeeze, Xen 4.0.1
Areca 1882i with 14 HDD Raid5 + 2 spare
I'm using lvm volume for my DomU

I have a Windows SBS 2K8 on this server with unsigned drivers and "test
mode" on an he is running ok !

If I make a second DomU, after installing Windows 2008R2 and Gplpv
Signed drivers, I shutdown my DomU.
When I start my DomU at the second time, my areca controler make hw
reset here is the log :

Jun 18 21:55:51 xentemp kernel: [514308.568415] scsi cmnd aborted,
scsi_cmnd(0xffff880002169400), cmnd[0x8a,0x 0,0x 0,0x 0,0x 0,0x
1,0x86,0xcf,0x30,0xc0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
Jun 18 21:55:54 xentemp kernel: [514311.800391] scsi cmnd aborted,
scsi_cmnd(0xffff880003336b00), cmnd[0x2a,0x 0,0x87,0x8f,0x21,0xe0,0x
0,0x 0,0x 8,0x 0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
Jun 18 21:55:57 xentemp kernel: [514315.032389] scsi cmnd aborted,
scsi_cmnd(0xffff880003337f00), cmnd[0x8a,0x 0,0x 0,0x 0,0x 0,0x
1,0x73,0xc0,0x a,0xe0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
Jun 18 21:56:00 xentemp kernel: [514318.268404] arcmsr1: executing eh
bus reset .....num_resets = 0, num_aborts = 3
Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vkbd/5/0
Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vfb/5/0
Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/console/5/0
Jun 18 21:56:32 xentemp kernel: [514350.268311] arcmsr0: wait 'abort all
outstanding command' timeout
Jun 18 21:56:32 xentemp kernel: [514350.268326] arcmsr1: executing hw
bus reset .....
Jun 18 21:57:18 xentemp kernel: [514396.263906] Areca RAID Controller0:
F/W V1.50 2012-01-20 & Model ARC-1882
Jun 18 21:57:18 xentemp kernel: [514396.276363] arcmsr: scsi bus reset
eh returns with success

There is my domU config :

#import os, re
#arch = os.uname()[4]
#if re.search('64', arch):
# arch_libdir = 'lib64'
#else:
# arch_libdir = 'lib'

kernel = "/usr/lib/xen-4.0/boot/hvmloader"

builder='hvm'
acpi=1
apic=1
vcpus = 4

memory = 4192
shadow_memory = 8
name = "filesrv"
vif = [ 'type=ioemu, bridge=eth0' ]
disk = [
'phy:/dev/xenvg/file-hda,hda,w',
'phy:/dev/xenvg/winbu,hdb,w',
'file:/mnt/iso/fr_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617591.iso,hdc:cdrom,r',

]
device_model = '/usr/lib64/xen-4.0/bin/qemu-dm'

# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="dc"

vfb = [
'type=vnc,vncdisplay=12,vncpasswd=s3cr3t,vnclisten=0.0.0.0,keymap=fr' ]

#sdl=0
#vnc=1
#vncconsole=1
#vncpasswd=''
stdvga=0
serial='pty'
usbdevice='tablet'

I realy don't understand the problem.

Thanks for your help

Matthieu Lejeune

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xen.org/xen-users

s.seitz at netzhaut

Jun 18, 2012, 12:16 PM

Post #2 of 6 (363 views)
Permalink

Re: Gplpv drivers make reset raid controller [In reply to]

Hi,

that sound's really strange, as there's obviously no "connection" between your domU
and your controller.
I've seen similar errors with ARC18xx but - to be honest - by using Nexenta (SunOS fork)
under heavy load.
It's more a feeling than knowledge, but I would expect some potential problem inside
your dom0 which panics out on *some* workload. I'ld expect your W2k8 is only *one*
trigger.
I'ld check
- for latest Firmware on your controller (currently 1.50 / 150-20120216),
- bios settings (maybe start over with system defaults plus CPU Virtualizationflag)
and as a last option: an oldconfig vanilla 3.4.3 dom0 kernel.

Hope this helps.

Good luck!

Am Montag, den 18.06.2012, 20:22 +0200 schrieb Matthieu Lejeune:

Hi,

I have a very stange problem with my xen 4.0.1

Debian squeeze, Xen 4.0.1
Areca 1882i with 14 HDD Raid5 + 2 spare
I'm using lvm volume for my DomU

I have a Windows SBS 2K8 on this server with unsigned drivers and "test
mode" on an he is running ok !

If I make a second DomU, after installing Windows 2008R2 and Gplpv
Signed drivers, I shutdown my DomU.
When I start my DomU at the second time, my areca controler make hw
reset here is the log :

Jun 18 21:55:51 xentemp kernel: [514308.568415] scsi cmnd aborted,
scsi_cmnd(0xffff880002169400), cmnd[0x8a,0x 0,0x 0,0x 0,0x 0,0x
1,0x86,0xcf,0x30,0xc0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
Jun 18 21:55:54 xentemp kernel: [514311.800391] scsi cmnd aborted,
scsi_cmnd(0xffff880003336b00), cmnd[0x2a,0x 0,0x87,0x8f,0x21,0xe0,0x
0,0x 0,0x 8,0x 0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
Jun 18 21:55:57 xentemp kernel: [514315.032389] scsi cmnd aborted,
scsi_cmnd(0xffff880003337f00), cmnd[0x8a,0x 0,0x 0,0x 0,0x 0,0x
1,0x73,0xc0,0x a,0xe0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
Jun 18 21:56:00 xentemp kernel: [514318.268404] arcmsr1: executing eh
bus reset .....num_resets = 0, num_aborts = 3
Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vkbd/5/0
Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vfb/5/0
Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/console/5/0
Jun 18 21:56:32 xentemp kernel: [514350.268311] arcmsr0: wait 'abort all
outstanding command' timeout
Jun 18 21:56:32 xentemp kernel: [514350.268326] arcmsr1: executing hw
bus reset .....
Jun 18 21:57:18 xentemp kernel: [514396.263906] Areca RAID Controller0:
F/W V1.50 2012-01-20 & Model ARC-1882
Jun 18 21:57:18 xentemp kernel: [514396.276363] arcmsr: scsi bus reset
eh returns with success

There is my domU config :

#import os, re
#arch = os.uname()[4]
#if re.search('64', arch):
# arch_libdir = 'lib64'
#else:
# arch_libdir = 'lib'

kernel = "/usr/lib/xen-4.0/boot/hvmloader"

builder='hvm'
acpi=1
apic=1
vcpus = 4

memory = 4192
shadow_memory = 8
name = "filesrv"
vif = [ 'type=ioemu, bridge=eth0' ]
disk = [
'phy:/dev/xenvg/file-hda,hda,w',
'phy:/dev/xenvg/winbu,hdb,w',
'file:/mnt/iso/fr_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617591.iso,hdc:cdrom,r',

]
device_model = '/usr/lib64/xen-4.0/bin/qemu-dm'

# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="dc"

vfb = [
'type=vnc,vncdisplay=12,vncpasswd=s3cr3t,vnclisten=0.0.0.0,keymap=fr' ]

#sdl=0
#vnc=1
#vncconsole=1
#vncpasswd=''
stdvga=0
serial='pty'
usbdevice='tablet'

I realy don't understand the problem.

Thanks for your help

Matthieu Lejeune

_______________________________________________
Xen-users mailing list
Xen-users [at] lists<mailto:Xen-users [at] lists>
http://lists.xen.org/xen-users

--

Stephan Seitz
Senior System Administrator

netz-haut GmbH
multimediale kommunikation

Friedrich-Bergius-Ring 12
97076 Würzburg

Telefon: 0931 - 780 11 780
Telefax: 0931 - 780 11 799

Web: www.netzhaut.de

Amtsgericht Würzburg – HRB 10764

Geschäftsführer: Michael Daut, Kai Neugebauer

matthieu.lejeune at exxoss

Jul 9, 2012, 11:45 PM

Post #3 of 6 (321 views)
Permalink

Re: Gplpv drivers make reset raid controller [In reply to]

Hi,

I have make more test and on the same hardware, when I running just one
dom U with a Win 2008R2 it's ok.
After installing Gplpv drivers
(http://apt.univention.de/download/addons/gplpv-drivers/
gplpv_Vista2008x64_signed_0.11.0.356.msi) the raid controler make HW
Reset like in my last post.

We have found a workaround by exporting the volume group with
iscsi-scst. With the exported disk there is no problem.
So the problem is only when the domU write directly to the physical disk.

Thank's

Matthieu

Le 18/06/12 21:16, Stephan Seitz a écrit :
> Hi,
>
> that sound's really strange, as there's obviously no "connection"
> between your domU
> and your controller.
> I've seen similar errors with ARC18xx but - to be honest - by using
> Nexenta (SunOS fork)
> under heavy load.
> It's more a feeling than knowledge, but I would expect some potential
> problem inside
> your dom0 which panics out on *some* workload. I'ld expect your W2k8
> is only *one*
> trigger.
> I'ld check
> - for latest Firmware on your controller (currently 1.50 / 150-20120216),
> - bios settings (maybe start over with system defaults plus CPU
> Virtualizationflag)
> and as a last option: an oldconfig vanilla 3.4.3 dom0 kernel.
>
> Hope this helps.
>
> Good luck!
>
>
>
> Am Montag, den 18.06.2012, 20:22 +0200 schrieb Matthieu Lejeune:
>> Hi,
>>
>> I have a very stange problem with my xen 4.0.1
>>
>> Debian squeeze, Xen 4.0.1
>> Areca 1882i with 14 HDD Raid5 + 2 spare
>> I'm using lvm volume for my DomU
>>
>> I have a Windows SBS 2K8 on this server with unsigned drivers and "test
>> mode" on an he is running ok !
>>
>> If I make a second DomU, after installing Windows 2008R2 and Gplpv
>> Signed drivers, I shutdown my DomU.
>> When I start my DomU at the second time, my areca controler make hw
>> reset here is the log :
>>
>> Jun 18 21:55:51 xentemp kernel: [514308.568415] scsi cmnd aborted,
>> scsi_cmnd(0xffff880002169400), cmnd[0x8a,0x 0,0x 0,0x 0,0x 0,0x
>> 1,0x86,0xcf,0x30,0xc0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
>> Jun 18 21:55:54 xentemp kernel: [514311.800391] scsi cmnd aborted,
>> scsi_cmnd(0xffff880003336b00), cmnd[0x2a,0x 0,0x87,0x8f,0x21,0xe0,0x
>> 0,0x 0,0x 8,0x 0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
>> Jun 18 21:55:57 xentemp kernel: [514315.032389] scsi cmnd aborted,
>> scsi_cmnd(0xffff880003337f00), cmnd[0x8a,0x 0,0x 0,0x 0,0x 0,0x
>> 1,0x73,0xc0,0x a,0xe0,0x 0,0x 0], scsi_id = 0x 0, scsi_lun = 0x 0.
>> Jun 18 21:56:00 xentemp kernel: [514318.268404] arcmsr1: executing eh
>> bus reset .....num_resets = 0, num_aborts = 3
>> Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
>> XENBUS_PATH=backend/vkbd/5/0
>> Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
>> XENBUS_PATH=backend/vfb/5/0
>> Jun 18 21:56:23 xentemp logger: /etc/xen/scripts/xen-hotplug-cleanup:
>> XENBUS_PATH=backend/console/5/0
>> Jun 18 21:56:32 xentemp kernel: [514350.268311] arcmsr0: wait 'abort all
>> outstanding command' timeout
>> Jun 18 21:56:32 xentemp kernel: [514350.268326] arcmsr1: executing hw
>> bus reset .....
>> Jun 18 21:57:18 xentemp kernel: [514396.263906] Areca RAID Controller0:
>> F/W V1.50 2012-01-20 & Model ARC-1882
>> Jun 18 21:57:18 xentemp kernel: [514396.276363] arcmsr: scsi bus reset
>> eh returns with success
>>
>> There is my domU config :
>>
>>
>> #import os, re
>> #arch = os.uname()[4]
>> #if re.search('64', arch):
>> # arch_libdir = 'lib64'
>> #else:
>> # arch_libdir = 'lib'
>>
>> kernel = "/usr/lib/xen-4.0/boot/hvmloader"
>>
>> builder='hvm'
>> acpi=1
>> apic=1
>> vcpus = 4
>>
>> memory = 4192
>> shadow_memory = 8
>> name = "filesrv"
>> vif = [ 'type=ioemu, bridge=eth0' ]
>> disk = [
>> 'phy:/dev/xenvg/file-hda,hda,w',
>> 'phy:/dev/xenvg/winbu,hdb,w',
>> 'file:/mnt/iso/fr_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617591.iso,hdc:cdrom,r',
>>
>> ]
>> device_model = '/usr/lib64/xen-4.0/bin/qemu-dm'
>>
>> # boot on floppy (a), hard disk (c) or CD-ROM (d)
>> # default: hard disk, cd-rom, floppy
>> boot="dc"
>>
>> vfb = [
>> 'type=vnc,vncdisplay=12,vncpasswd=s3cr3t,vnclisten=0.0.0.0,keymap=fr' ]
>>
>>
>> #sdl=0
>> #vnc=1
>> #vncconsole=1
>> #vncpasswd=''
>> stdvga=0
>> serial='pty'
>> usbdevice='tablet'
>>
>>
>> I realy don't understand the problem.
>>
>> Thanks for your help
>>
>> Matthieu Lejeune
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users [at] lists <mailto:Xen-users [at] lists>
>> http://lists.xen.org/xen-users
>
> --
> Stephan Seitz
> Senior System Administrator
>
> netz-haut GmbH
> multimediale kommunikation
>
> Friedrich-Bergius-Ring 12
> 97076 Würzburg
>
> Telefon: 0931 - 780 11 780
> Telefax: 0931 - 780 11 799
>
> Web: www.netzhaut.de
>
>
> Amtsgericht Würzburg – HRB 10764
>
> Geschäftsführer: Michael Daut, Kai Neugebauer
>
>
>

james.harper at bendigoit

Jul 10, 2012, 5:34 AM

Post #4 of 6 (320 views)
Permalink

Re: Gplpv drivers make reset raid controller [In reply to]

> Hi,
>
> I have make more test and on the same hardware, when I running just one
> dom U with a Win 2008R2 it's ok.
> After installing Gplpv drivers
> (http://apt.univention.de/download/addons/gplpv-drivers/
> gplpv_Vista2008x64_signed_0.11.0.356.msi) the raid controler make HW
> Reset like in my last post.
>
> We have found a workaround by exporting the volume group with iscsi-scst.
> With the exported disk there is no problem.
> So the problem is only when the domU write directly to the physical disk.
>

One guess... with GPLPV you can probably send a lot more commands to the RAID than with an emulated IDE driver and maybe this is causing timeouts in the array... when you start up the second DomU does the scsi reset happen very quickly or just after a while? If its only after a while, can you start just one Windows 2008R2 DomU then try and generate a lot of write traffic on the RAID in Dom0?

Also, what kernel are you using? I can't find the text "scsi cmnd aborted" in any kernel source anywhere...

James

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xen.org/xen-users

s.seitz at netzhaut

Jul 10, 2012, 7:26 AM

Post #5 of 6 (324 views)
Permalink

Re: Gplpv drivers make reset raid controller [In reply to]

Am Dienstag, den 10.07.2012, 12:34 +0000 schrieb James Harper:

> Hi,
>
> I have make more test and on the same hardware, when I running just one
> dom U with a Win 2008R2 it's ok.
> After installing Gplpv drivers
> (http://apt.univention.de/download/addons/gplpv-drivers/
> gplpv_Vista2008x64_signed_0.11.0.356.msi) the raid controler make HW
> Reset like in my last post.
>
> We have found a workaround by exporting the volume group with iscsi-scst.
> With the exported disk there is no problem.
> So the problem is only when the domU write directly to the physical disk.
>

One guess... with GPLPV you can probably send a lot more commands to the RAID than with an emulated IDE driver and maybe this is causing timeouts in the array... when you start up the second DomU does the scsi reset happen very quickly or just after a while? If its only after a while, can you start just one Windows 2008R2 DomU then try and generate a lot of write traffic on the RAID in Dom0?

Also, what kernel are you using? I can't find the text "scsi cmnd aborted" in any kernel source anywhere...

That message is generated by int arcmsr_abort(struct scsi_cmnd *cmd) inside the areca arcmsr driver.
I didn't peek inside kernel-tree, it's at least at the out-of-tree driver provided by areca [1].
That driver has been added to the vanilla kernel around 2.6.16, so i'ld expect no big differences.

[1] http://www.areca.us/support/s_linux/driver/arcmsr.1.20.0X.15-111012.zip

As written a few days ago, I'm also guessing the gplpv drivers are not the source of the
problem, but a potential trigger. matthieu 's pointed out, that looping through iscsi has
solved the issue, but I assume with iscsi, it just slowed down and kept below "critical" i/o.

matthieu.lejeune at exxoss

Jul 11, 2012, 2:28 AM

Post #6 of 6 (325 views)
Permalink

Re: Gplpv drivers make reset raid controller [In reply to]

Hi,

Thanks for your help. I'll make some test.
I have this problem with one DomU at time and just on the boot when
Windows is loading drivers.

I use this version of the kernel : Linux version 2.6.32-5-xen-amd64
(Debian 2.6.32-45)
This version of the Areca Driver : arcmsr.1.20.0X.15-111012.zip
And the last firmware of the Arc-1882i

Thanks

Matthieu

I use Le 10/07/12 16:26, Stephan Seitz a �crit :
>
> Am Dienstag, den 10.07.2012, 12:34 +0000 schrieb James Harper:
>> > Hi,
>> >
>> > I have make more test and on the same hardware, when I running just one
>> > dom U with a Win 2008R2 it's ok.
>> > After installing Gplpv drivers
>> > (http://apt.univention.de/download/addons/gplpv-drivers/
>> > gplpv_Vista2008x64_signed_0.11.0.356.msi) the raid controler make HW
>> > Reset like in my last post.
>> >
>> > We have found a workaround by exporting the volume group with iscsi-scst.
>> > With the exported disk there is no problem.
>> > So the problem is only when the domU write directly to the physical disk.
>> >
>>
>> One guess... with GPLPV you can probably send a lot more commands to the RAID than with an emulated IDE driver and maybe this is causing timeouts in the array... when you start up the second DomU does the scsi reset happen very quickly or just after a while? If its only after a while, can you start just one Windows 2008R2 DomU then try and generate a lot of write traffic on the RAID in Dom0?
>>
>> Also, what kernel are you using? I can't find the text "scsi cmnd aborted" in any kernel source anywhere...
>
> That message is generated by int arcmsr_abort(struct scsi_cmnd *cmd)
> inside the areca arcmsr driver.
> I didn't peek inside kernel-tree, it's at least at the out-of-tree
> driver provided by areca [1].
> That driver has been added to the vanilla kernel around 2.6.16, so
> i'ld expect no big differences.
>
> [1]
> http://www.areca.us/support/s_linux/driver/arcmsr.1.20.0X.15-111012.zip
>
> As written a few days ago, I'm also guessing the gplpv drivers are not
> the source of the
> problem, but a potential trigger. matthieu 's pointed out, that
> looping through iscsi has
> solved the issue, but I assume with iscsi, it just slowed down and
> kept below "critical" i/o.
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users [at] lists
> http://lists.xen.org/xen-users

--

Exxoss <http://www.exxoss.com>
*Matthieu Lejeune*, System Engineer | Gsm: +32(0)491/52.70.66
*Exxoss, SPRL <http://www.exxoss.com>*
Rue de la station, 2, 4347, Fexhe-le-haut-clocher | Telephone:
+32(0)4/341.25.81 | Fax: +32(0)4/371.94.06
Twitter <http://twitter.com/exxossIT> Facebook
<http://facebook.com/exxoss> Linked In
<http://www.linkedin.com/company/exxoss>

Attachments:

exxoss.png (11.7 KB)

twittercolor.png (3.35 KB)

facebookcolor.png (1.89 KB)

linkedincolor.png (1.88 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads