Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

Issue in KVM block migration

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


Unmesh.Gurjar at nttdata

Apr 11, 2012, 5:48 AM

Post #1 of 9 (525 views)
Permalink
Issue in KVM block migration

Hi,

I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here<http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>). In addition, I have also disabled apparmor for libvirtd profile and have an entry in '/etc/hosts' of both the Compute hosts.

>From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
virsh # connect qemu+tcp://ubuntu-dev-001/system
virsh # list

The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
libvir: QEMU error : operation failed: migration job: unexpectedly failed
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
readers.get(fileno, noop).cb(fileno)
File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
recover_method(ctxt, instance_ref, dest, block_migration)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
FLAGS.live_migration_bandwidth)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
libvirtError: operation failed: migration job: unexpectedly failed
Removing descriptor: 12

I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
2012-04-11 04:14:25.971: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Domain id=15 is tainted: high-privileges
Domain id=15 is tainted: shell-scripts
char device redirected to /dev/pts/17
2012-04-11 04:14:26.406: shutting down
can't delete tapfcfa2a6c-35 from eth1: Operation not supported
SIOCSIFADDR: Permission denied
SIOCSIFFLAGS: Permission denied
SIOCSIFFLAGS: Permission denied
/etc/qemu-ifdown: could not launch network script

Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding


razique.mahroua at gmail

Apr 11, 2012, 7:02 AM

Post #2 of 9 (528 views)
Permalink
Re: Issue in KVM block migration [In reply to]

Hi,
it looks like the user under which libvirt is running doesn't have the rights to manage server's IF.
What version of libvirt are you using ?
Can I see the file /etc/libvirt/libvirtd.com ? (cat /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$

Raz

Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail



Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :

> Hi,
>
> I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here). In addition, I have also disabled apparmor for libvirtd profile and have an entry in ‘/etc/hosts’ of both the Compute hosts.
>
> From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
> virsh # connect qemu+tcp://ubuntu-dev-001/system
> virsh # list
>
> The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
> libvir: QEMU error : operation failed: migration job: unexpectedly failed
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
> readers.get(fileno, noop).cb(fileno)
> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
> result = function(*args, **kwargs)
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
> recover_method(ctxt, instance_ref, dest, block_migration)
> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> self.gen.next()
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
> FLAGS.live_migration_bandwidth)
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
> if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
> libvirtError: operation failed: migration job: unexpectedly failed
> Removing descriptor: 12
>
> I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
> 2012-04-11 04:14:25.971: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> Domain id=15 is tainted: high-privileges
> Domain id=15 is tainted: shell-scripts
> char device redirected to /dev/pts/17
> 2012-04-11 04:14:26.406: shutting down
> can't delete tapfcfa2a6c-35 from eth1: Operation not supported
> SIOCSIFADDR: Permission denied
> SIOCSIFFLAGS: Permission denied
> SIOCSIFFLAGS: Permission denied
> /etc/qemu-ifdown: could not launch network script
>
> Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.
>
> Thanks & Regards,
> Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on Twitter [at] NTTDATAAmerica
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
Attachments: NUAGECO-LOGO-Fblan_petit.jpg (9.88 KB)


Unmesh.Gurjar at nttdata

Apr 11, 2012, 7:44 AM

Post #3 of 9 (512 views)
Permalink
Re: Issue in KVM block migration [In reply to]

Thanks Razique for taking up this one.

Libvirt version on both Compute hosts:
$ libvirtd --version
libvirtd (libvirt) 0.9.2
$ virsh --version
0.9.2

Here are my libvirtd.conf details:
listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none"


Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica

From: Razique Mahroua [mailto:razique.mahroua [at] gmail]
Sent: Wednesday, April 11, 2012 7:33 PM
To: Gurjar, Unmesh
Cc: openstack [at] lists
Subject: Re: [Openstack] Issue in KVM block migration

Hi,
it looks like the user under which libvirt is running doesn't have the rights to manage server's IF.
What version of libvirt are you using ?
Can I see the file /etc/libvirt/libvirtd.com ? (cat /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$

Raz

Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail<mailto:razique.mahroua [at] gmail>

[cid:image001.jpg [at] 01CD181B]

Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :


Hi,

I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here<http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>). In addition, I have also disabled apparmor for libvirtd profile and have an entry in '/etc/hosts' of both the Compute hosts.

>From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
virsh # connect qemu+tcp://ubuntu-dev-001/system
virsh # list

The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
libvir: QEMU error : operation failed: migration job: unexpectedly failed
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
readers.get(fileno, noop).cb(fileno)
File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
recover_method(ctxt, instance_ref, dest, block_migration)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
FLAGS.live_migration_bandwidth)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
libvirtError: operation failed: migration job: unexpectedly failed
Removing descriptor: 12

I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
2012-04-11 04:14:25.971: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Domain id=15 is tainted: high-privileges
Domain id=15 is tainted: shell-scripts
char device redirected to /dev/pts/17
2012-04-11 04:14:26.406: shutting down
can't delete tapfcfa2a6c-35 from eth1: Operation not supported
SIOCSIFADDR: Permission denied
SIOCSIFFLAGS: Permission denied
SIOCSIFFLAGS: Permission denied
/etc/qemu-ifdown: could not launch network script

Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists<mailto:openstack [at] lists>
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Attachments: image001.jpg (9.88 KB)


razique.mahroua at gmail

Apr 13, 2012, 1:51 AM

Post #4 of 9 (509 views)
Permalink
Re: Issue in KVM block migration [In reply to]

Hi,
sorry for the late reply.
Does the user nova belong to the libvirt group ?
Can you try manually the migration via virsh ?
$ virsh --migrate --live --copy-storage-all $domain qemu+tcp://user [at] serve/sytem

thanks


Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail



Le 11 avr. 2012 à 16:44, Gurjar, Unmesh a écrit :

> Thanks Razique for taking up this one.
>
> Libvirt version on both Compute hosts:
> $ libvirtd --version
> libvirtd (libvirt) 0.9.2
> $ virsh --version
> 0.9.2
>
> Here are my libvirtd.conf details:
> listen_tls = 0
> listen_tcp = 1
> unix_sock_group = "libvirtd"
> unix_sock_rw_perms = "0770"
> auth_unix_ro = "none"
> auth_unix_rw = "none"
> auth_tcp = "none"
>
>
> Thanks & Regards,
> Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on Twitter [at] NTTDATAAmerica
>
> From: Razique Mahroua [mailto:razique.mahroua [at] gmail]
> Sent: Wednesday, April 11, 2012 7:33 PM
> To: Gurjar, Unmesh
> Cc: openstack [at] lists
> Subject: Re: [Openstack] Issue in KVM block migration
>
> Hi,
> it looks like the user under which libvirt is running doesn't have the rights to manage server's IF.
> What version of libvirt are you using ?
> Can I see the file /etc/libvirt/libvirtd.com ? (cat /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$
>
> Raz
>
> Nuage & Co - Razique Mahroua
> razique.mahroua [at] gmail
>
> <image001.jpg>
>
> Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :
>
>
> Hi,
>
> I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here). In addition, I have also disabled apparmor for libvirtd profile and have an entry in ‘/etc/hosts’ of both the Compute hosts.
>
> From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
> virsh # connect qemu+tcp://ubuntu-dev-001/system
> virsh # list
>
> The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
> libvir: QEMU error : operation failed: migration job: unexpectedly failed
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
> readers.get(fileno, noop).cb(fileno)
> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
> result = function(*args, **kwargs)
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
> recover_method(ctxt, instance_ref, dest, block_migration)
> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> self.gen.next()
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
> FLAGS.live_migration_bandwidth)
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
> if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
> libvirtError: operation failed: migration job: unexpectedly failed
> Removing descriptor: 12
>
> I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
> 2012-04-11 04:14:25.971: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> Domain id=15 is tainted: high-privileges
> Domain id=15 is tainted: shell-scripts
> char device redirected to /dev/pts/17
> 2012-04-11 04:14:26.406: shutting down
> can't delete tapfcfa2a6c-35 from eth1: Operation not supported
> SIOCSIFADDR: Permission denied
> SIOCSIFFLAGS: Permission denied
> SIOCSIFFLAGS: Permission denied
> /etc/qemu-ifdown: could not launch network script
>
> Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.
>
> Thanks & Regards,
> Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on Twitter [at] NTTDATAAmerica
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Attachments: NUAGECO-LOGO-Fblan_petit.jpg (9.88 KB)


Unmesh.Gurjar at nttdata

Apr 13, 2012, 5:46 AM

Post #5 of 9 (505 views)
Permalink
Re: Issue in KVM block migration [In reply to]

The user running the compute server is a member of libvirt group.
I tried the migration via virsh, however that too returns the same result. I get similar logs in the qemu log file.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica

From: Razique Mahroua [mailto:razique.mahroua [at] gmail]
Sent: Friday, April 13, 2012 2:22 PM
To: Gurjar, Unmesh
Cc: openstack [at] lists
Subject: Re: [Openstack] Issue in KVM block migration

Hi,
sorry for the late reply.
Does the user nova belong to the libvirt group ?
Can you try manually the migration via virsh ?
$ virsh --migrate --live --copy-storage-all $domain qemu+tcp://user [at] serve/sytem

thanks


Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail<mailto:razique.mahroua [at] gmail>

[cid:image001.jpg [at] 01CD199E]

Le 11 avr. 2012 à 16:44, Gurjar, Unmesh a écrit :


Thanks Razique for taking up this one.

Libvirt version on both Compute hosts:
$ libvirtd --version
libvirtd (libvirt) 0.9.2
$ virsh --version
0.9.2

Here are my libvirtd.conf details:
listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none"


Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica

From: Razique Mahroua [mailto:razique.mahroua [at] gmail]<mailto:[mailto:razique.mahroua [at] gmail]>
Sent: Wednesday, April 11, 2012 7:33 PM
To: Gurjar, Unmesh
Cc: openstack [at] lists<mailto:openstack [at] lists>
Subject: Re: [Openstack] Issue in KVM block migration

Hi,
it looks like the user under which libvirt is running doesn't have the rights to manage server's IF.
What version of libvirt are you using ?
Can I see the file /etc/libvirt/libvirtd.com ? (cat /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$

Raz

Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail<mailto:razique.mahroua [at] gmail>

<image001.jpg>

Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :



Hi,

I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here<http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>). In addition, I have also disabled apparmor for libvirtd profile and have an entry in '/etc/hosts' of both the Compute hosts.

>From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
virsh # connect qemu+tcp://ubuntu-dev-001/system
virsh # list

The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
libvir: QEMU error : operation failed: migration job: unexpectedly failed
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
readers.get(fileno, noop).cb(fileno)
File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
recover_method(ctxt, instance_ref, dest, block_migration)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
FLAGS.live_migration_bandwidth)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
libvirtError: operation failed: migration job: unexpectedly failed
Removing descriptor: 12

I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
2012-04-11 04:14:25.971: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Domain id=15 is tainted: high-privileges
Domain id=15 is tainted: shell-scripts
char device redirected to /dev/pts/17
2012-04-11 04:14:26.406: shutting down
can't delete tapfcfa2a6c-35 from eth1: Operation not supported
SIOCSIFADDR: Permission denied
SIOCSIFFLAGS: Permission denied
SIOCSIFFLAGS: Permission denied
/etc/qemu-ifdown: could not launch network script

Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists<mailto:openstack [at] lists>
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Attachments: image001.jpg (9.88 KB)


heut2008 at gmail

Apr 13, 2012, 6:53 AM

Post #6 of 9 (504 views)
Permalink
Re: Issue in KVM block migration [In reply to]

Live block migration may need to be root.through I am not sure.a few day's
ago,I tested live block migration,and config hostA and hostB to use ssh
key to access each other with nova user; but it fails to migration.when I
run nova-compute with root ,and root ssh key to both hosts;the block
migration works well. then I config libvirt to use tls to remote access .
http://libvirt.org/remote.html.it also works well.

2012/4/13 Gurjar, Unmesh <Unmesh.Gurjar [at] nttdata>

> The user running the compute server is a member of libvirt group.****
>
> I tried the migration via virsh, however that too returns the same result.
> I get similar logs in the qemu log file.****
>
> ** **
>
> Thanks & Regards,****
>
> *Unmesh Gurjar* | Lead Engineer | Vertex Software Private Ltd. | *w.* +91.20.6604.1500
> x 379 | *m.* +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on
> Twitter [at] NTTDATAAmerica**
>
> ** **
>
> *From:* Razique Mahroua [mailto:razique.mahroua [at] gmail]
> *Sent:* Friday, April 13, 2012 2:22 PM
>
> *To:* Gurjar, Unmesh
> *Cc:* openstack [at] lists
> *Subject:* Re: [Openstack] Issue in KVM block migration****
>
> ** **
>
> Hi, ****
>
> sorry for the late reply.****
>
> Does the user nova belong to the libvirt group ?****
>
> Can you try manually the migration via virsh ?****
>
> $ virsh --migrate --live --copy-storage-all $domain
> qemu+tcp://user [at] serve/sytem****
>
> ** **
>
> thanks****
>
> ** **
>
> ** **
>
> *Nuage & Co - Razique Mahroua *****
>
> razique.mahroua [at] gmail****
>
>
> ****
>
> ** **
>
> Le 11 avr. 2012 à 16:44, Gurjar, Unmesh a écrit :****
>
>
>
> ****
>
> Thanks Razique for taking up this one.****
>
> ****
>
> Libvirt version on both Compute hosts:****
>
> $ libvirtd --version****
>
> libvirtd (libvirt) 0.9.2****
>
> $ virsh --version****
>
> 0.9.2****
>
> ****
>
> Here are my libvirtd.conf details:****
>
> listen_tls = 0****
>
> listen_tcp = 1****
>
> unix_sock_group = "libvirtd"****
>
> unix_sock_rw_perms = "0770"****
>
> auth_unix_ro = "none"****
>
> auth_unix_rw = "none"****
>
> auth_tcp = "none"****
>
> ****
>
> ****
>
> Thanks & Regards,****
>
> *Unmesh Gurjar* | Lead Engineer | Vertex Software Private Ltd. | *w.* +91.20.6604.1500
> x 379 | *m.* +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on
> Twitter [at] NTTDATAAmerica****
>
> ****
>
> *From:* Razique Mahroua [mailto:razique.mahroua [at] gmail]
> *Sent:* Wednesday, April 11, 2012 7:33 PM
> *To:* Gurjar, Unmesh
> *Cc:* openstack [at] lists
> *Subject:* Re: [Openstack] Issue in KVM block migration****
>
> ****
>
> Hi,****
>
> it looks like the user under which libvirt is running doesn't have the
> rights to manage server's IF.****
>
> What version of libvirt are you using ?****
>
> Can I see the file /etc/libvirt/libvirtd.com ? (cat
> /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$****
>
> ****
>
> Raz****
>
> ****
>
> *Nuage & Co - Razique Mahroua *****
>
> razique.mahroua [at] gmail****
>
>
> <image001.jpg>****
>
> ****
>
> Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :****
>
>
>
>
> ****
>
> Hi,****
>
> ****
>
> I have setup two Compute nodes (using Openstack master branch) and
> configured libvirt for block migration (by following step #1 and #4
> mentioned here<http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>).
> In addition, I have also disabled apparmor for libvirtd profile and have an
> entry in ‘/etc/hosts’ of both the Compute hosts.****
>
> ****
>
> From both the Compute hosts, I am able to connect and fetch the list of
> running instances on the other host (using the hostname), as follows:****
>
> virsh # connect qemu+tcp://ubuntu-dev-001/system****
>
> virsh # list****
>
> ****
>
> The issue is block migrating an instance between these hosts fails with
> the following error in the source Compute host console:****
>
> libvir: QEMU error : operation failed: migration job: unexpectedly failed*
> ***
>
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on
> network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
> ****
>
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is
> 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall
> /opt/stack/nova/nova/rpc/amqp.py:321****
>
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on
> compute.ubuntu-dev-001... from (pid=18487) cast
> /opt/stack/nova/nova/rpc/amqp.py:343****
>
> Traceback (most recent call last):****
>
> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97,
> in wait****
>
> readers.get(fileno, noop).cb(fileno)****
>
> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line
> 192, in main****
>
> result = function(*args, **kwargs)****
>
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in
> _live_migration****
>
> recover_method(ctxt, instance_ref, dest, block_migration)****
>
> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__****
>
> self.gen.next()****
>
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in
> _live_migration****
>
> FLAGS.live_migration_bandwidth)****
>
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in
> migrateToURI****
>
> if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed',
> dom=self)****
>
> libvirtError: operation failed: migration job: unexpectedly failed****
>
> Removing descriptor: 12****
>
> ****
>
> I find the following in the qemu log file
> (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute
> host:****
>
> 2012-04-11 04:14:25.971: starting up****
>
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
> 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid
> f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c
> -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd
> /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda
> console=ttyS0 -drive
> file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device
> rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3
> -chardev
> file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log
> -device isa-serial,chardev=charserial0,id=serial0 -chardev
> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb
> -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus
> -incoming tcp:0.0.0.0:49166 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5****
>
> Domain id=15 is tainted: high-privileges****
>
> Domain id=15 is tainted: shell-scripts****
>
> char device redirected to /dev/pts/17****
>
> 2012-04-11 04:14:26.406: shutting down****
>
> can't delete tapfcfa2a6c-35 from eth1: Operation not supported****
>
> SIOCSIFADDR: Permission denied****
>
> SIOCSIFFLAGS: Permission denied****
>
> SIOCSIFFLAGS: Permission denied****
>
> /etc/qemu-ifdown: could not launch network script****
>
> ****
>
> Would be great if someone can point anything that I am missing here or any
> configuration changes required to resolve this issue.****
>
> ****
>
> Thanks & Regards,****
>
> *Unmesh Gurjar* | Lead Engineer | Vertex Software Private Ltd. | *w.* +91.20.6604.1500
> x 379 | *m.* +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on
> Twitter [at] NTTDATAAmerica****
>
> ****
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp****
>
> ****
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding****
>
> ** **
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


kei.masumoto at gmail

Apr 17, 2012, 4:45 PM

Post #7 of 9 (501 views)
Permalink
Re: Issue in KVM block migration [In reply to]

/etc/hosts must be exactly same between src and dest compute node. that
is your problem.
Kei

(2012/04/13 21:46), Gurjar, Unmesh wrote:
>
> The user running the compute server is a member of libvirt group.
>
> I tried the migration via virsh, however that too returns the same
> result. I get similar logs in the qemu log file.
>
> Thanks & Regards,
>
> *Unmesh Gurjar*| Lead Engineer | Vertex Software Private Ltd.| *w.*
> +91.20.6604.1500 x 379 | *m.* +91.982.324.7631
> |unmesh.gurjar [at] nttdata <mailto:unmesh.gurjar [at] nttdata> | Follow
> us on Twitter [at] NTTDATAAmerica**
>
> *From:*Razique Mahroua [mailto:razique.mahroua [at] gmail]
> *Sent:* Friday, April 13, 2012 2:22 PM
> *To:* Gurjar, Unmesh
> *Cc:* openstack [at] lists
> *Subject:* Re: [Openstack] Issue in KVM block migration
>
> Hi,
>
> sorry for the late reply.
>
> Does the user nova belong to the libvirt group ?
>
> Can you try manually the migration via virsh ?
>
> $ virsh --migrate --live --copy-storage-all $domain
> qemu+tcp://user [at] serve/sytem
>
> thanks
>
> *Nuage & Co - Razique Mahroua *
>
> razique.mahroua [at] gmail <mailto:razique.mahroua [at] gmail>
>
>
> Le 11 avr. 2012 à 16:44, Gurjar, Unmesh a écrit :
>
>
>
> Thanks Razique for taking up this one.
>
> Libvirt version on both Compute hosts:
>
> $ libvirtd --version
>
> libvirtd (libvirt) 0.9.2
>
> $ virsh --version
>
> 0.9.2
>
> Here are my libvirtd.conf details:
>
> listen_tls = 0
>
> listen_tcp = 1
>
> unix_sock_group = "libvirtd"
>
> unix_sock_rw_perms = "0770"
>
> auth_unix_ro = "none"
>
> auth_unix_rw = "none"
>
> auth_tcp = "none"
>
> Thanks & Regards,
>
> *Unmesh Gurjar*| Lead Engineer | Vertex Software Private Ltd.
> |*w.*+91.20.6604.1500 x 379
> |*m.*+91.982.324.7631|unmesh.gurjar [at] nttdata
> <mailto:unmesh.gurjar [at] nttdata>|Follow us on Twitter [at] NTTDATAAmerica
>
> *From:*Razique Mahroua [mailto:razique.mahroua [at] gmail]
> <mailto:[mailto:razique.mahroua [at] gmail]>
> *Sent:*Wednesday, April 11, 2012 7:33 PM
> *To:*Gurjar, Unmesh
> *Cc:*openstack [at] lists <mailto:openstack [at] lists>
> *Subject:*Re: [Openstack] Issue in KVM block migration
>
> Hi,
>
> it looks like the user under which libvirt is running doesn't have the
> rights to manage server's IF.
>
> What version of libvirt are you using ?
>
> Can I see the file /etc/libvirt/libvirtd.com ? (cat
> /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$
>
> Raz
>
> *Nuage & Co - Razique Mahroua *
>
> razique.mahroua [at] gmail <mailto:razique.mahroua [at] gmail>
>
>
> <image001.jpg>
>
> Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :
>
>
>
>
> Hi,
>
> I have setup two Compute nodes (using Openstack master branch) and
> configured libvirt for block migration (by following step #1 and #4
> mentionedhere
> <http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>).
> In addition, I have also disabled apparmor for libvirtd profile and
> have an entry in '/etc/hosts' of both the Compute hosts.
>
> From both the Compute hosts, I am able to connect and fetch the list
> of running instances on the other host (using the hostname), as follows:
>
> virsh # connectqemu+tcp://ubuntu-dev-001/system
>
> virsh # list
>
> The issue is block migrating an instance between these hosts fails
> with the following error in the source Compute host console:
>
> libvir: QEMU error : operation failed: migration job: unexpectedly failed
>
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call
> on network ... from (pid=18487) multicall
> /opt/stack/nova/nova/rpc/amqp.py:318
>
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is
> 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall
> /opt/stack/nova/nova/rpc/amqp.py:321
>
> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast
> on compute.ubuntu-dev-001... from (pid=18487) cast
> /opt/stack/nova/nova/rpc/amqp.py:343
>
> Traceback (most recent call last):
>
> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line
> 97, in wait
>
> readers.get(fileno, noop).cb(fileno)
>
> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py",
> line 192, in main
>
> result = function(*args, **kwargs)
>
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179,
> in _live_migration
>
> recover_method(ctxt, instance_ref, dest, block_migration)
>
> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>
> self.gen.next()
>
> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175,
> in _live_migration
>
> FLAGS.live_migration_bandwidth)
>
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in
> migrateToURI
>
> if ret == -1: raise libvirtError ('virDomainMigrateToURI()
> failed', dom=self)
>
> libvirtError: operation failed: migration job: unexpectedly failed
>
> Removing descriptor: 12
>
> I find the following in the qemu log file
> (/var/log/libvirt/qemu/instance-00000003.log ) on the destination
> Compute host:
>
> 2012-04-11 04:14:25.971: starting up
>
> LC_ALL=C
> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
> 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid
> f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
> c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd
> /opt/stack/nova/instances/instance-00000003/ramdisk -append
> root=/dev/vda console=ttyS0 -drive
> file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device
> rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3
> -chardev
> file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log
> -device isa-serial,chardev=charserial0,id=serial0 -chardev
> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1
> -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga
> cirrus -incoming tcp:0.0.0.0:49166 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
> Domain id=15 is tainted: high-privileges
>
> Domain id=15 is tainted: shell-scripts
>
> char device redirected to /dev/pts/17
>
> 2012-04-11 04:14:26.406: shutting down
>
> can't delete tapfcfa2a6c-35 from eth1: Operation not supported
>
> SIOCSIFADDR: Permission denied
>
> SIOCSIFFLAGS: Permission denied
>
> SIOCSIFFLAGS: Permission denied
>
> /etc/qemu-ifdown: could not launch network script
>
> Would be great if someone can point anything that I am missing here or
> any configuration changes required to resolve this issue.
>
> Thanks & Regards,
>
> *Unmesh Gurjar*| Lead Engineer |Vertex Software Private Ltd.
> |*w.*+91.20.6604.1500 x 379 |*m.*+91.982.324.7631
> |unmesh.gurjar [at] nttdata <mailto:unmesh.gurjar [at] nttdata>|Follow
> us on Twitter [at] NTTDATAAmerica
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest
> confidence for the sole use of the addressee and may contain legally
> privileged, confidential, and proprietary data. If you are not the
> intended recipient, please advise the sender by replying promptly to
> this email and then delete and destroy this email and any attachments
> without any further use, copying or forwarding
> _______________________________________________
> Mailing list:https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> Post to :openstack [at] lists
> <mailto:openstack [at] lists>
> Unsubscribe :https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> More help :https://help.launchpad.net/ListHelp
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest
> confidence for the sole use of the addressee and may contain legally
> privileged, confidential, and proprietary data. If you are not the
> intended recipient, please advise the sender by replying promptly to
> this email and then delete and destroy this email and any attachments
> without any further use, copying or forwarding
>
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest
> confidence for the sole use of the addressee and may contain legally
> privileged, confidential, and proprietary data. If you are not the
> intended recipient, please advise the sender by replying promptly to
> this email and then delete and destroy this email and any attachments
> without any further use, copying or forwarding
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp


razique.mahroua at gmail

Apr 17, 2012, 11:58 PM

Post #8 of 9 (506 views)
Permalink
Re: Issue in KVM block migration [In reply to]

Wouldn't libvirt complain about a non-existent host in such case ?
Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail



Le 18 avr. 2012 à 01:45, Kei.masumoto a écrit :

> /etc/hosts must be exactly same between src and dest compute node. that is your problem.
> Kei
>
> (2012/04/13 21:46), Gurjar, Unmesh wrote:
>>
>> The user running the compute server is a member of libvirt group.
>> I tried the migration via virsh, however that too returns the same result. I get similar logs in the qemu log file.
>>
>> Thanks & Regards,
>> Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 |unmesh.gurjar [at] nttdata | Follow us on Twitter [at] NTTDATAAmerica
>>
>> From: Razique Mahroua [mailto:razique.mahroua [at] gmail]
>> Sent: Friday, April 13, 2012 2:22 PM
>> To: Gurjar, Unmesh
>> Cc: openstack [at] lists
>> Subject: Re: [Openstack] Issue in KVM block migration
>>
>> Hi,
>> sorry for the late reply.
>> Does the user nova belong to the libvirt group ?
>> Can you try manually the migration via virsh ?
>> $ virsh --migrate --live --copy-storage-all $domain qemu+tcp://user [at] serve/sytem
>>
>> thanks
>>
>>
>> Nuage & Co - Razique Mahroua
>> razique.mahroua [at] gmail
>>
>> <Pièce jointe Mail.jpeg>
>>
>> Le 11 avr. 2012 à 16:44, Gurjar, Unmesh a écrit :
>>
>>
>> Thanks Razique for taking up this one.
>>
>> Libvirt version on both Compute hosts:
>> $ libvirtd --version
>> libvirtd (libvirt) 0.9.2
>> $ virsh --version
>> 0.9.2
>>
>> Here are my libvirtd.conf details:
>> listen_tls = 0
>> listen_tcp = 1
>> unix_sock_group = "libvirtd"
>> unix_sock_rw_perms = "0770"
>> auth_unix_ro = "none"
>> auth_unix_rw = "none"
>> auth_tcp = "none"
>>
>>
>> Thanks & Regards,
>> Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on Twitter [at] NTTDATAAmerica
>>
>> From: Razique Mahroua [mailto:razique.mahroua [at] gmail]
>> Sent: Wednesday, April 11, 2012 7:33 PM
>> To: Gurjar, Unmesh
>> Cc: openstack [at] lists
>> Subject: Re: [Openstack] Issue in KVM block migration
>>
>> Hi,
>> it looks like the user under which libvirt is running doesn't have the rights to manage server's IF.
>> What version of libvirt are you using ?
>> Can I see the file /etc/libvirt/libvirtd.com ? (cat /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$
>>
>> Raz
>>
>> Nuage & Co - Razique Mahroua
>> razique.mahroua [at] gmail
>>
>> <image001.jpg>
>>
>> Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :
>>
>>
>>
>> Hi,
>>
>> I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here). In addition, I have also disabled apparmor for libvirtd profile and have an entry in ‘/etc/hosts’ of both the Compute hosts.
>>
>> From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
>> virsh # connect qemu+tcp://ubuntu-dev-001/system
>> virsh # list
>>
>> The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
>> libvir: QEMU error : operation failed: migration job: unexpectedly failed
>> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
>> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
>> 2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
>> readers.get(fileno, noop).cb(fileno)
>> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
>> result = function(*args, **kwargs)
>> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
>> recover_method(ctxt, instance_ref, dest, block_migration)
>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> self.gen.next()
>> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
>> FLAGS.live_migration_bandwidth)
>> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
>> if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
>> libvirtError: operation failed: migration job: unexpectedly failed
>> Removing descriptor: 12
>>
>> I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
>> 2012-04-11 04:14:25.971: starting up
>> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> Domain id=15 is tainted: high-privileges
>> Domain id=15 is tainted: shell-scripts
>> char device redirected to /dev/pts/17
>> 2012-04-11 04:14:26.406: shutting down
>> can't delete tapfcfa2a6c-35 from eth1: Operation not supported
>> SIOCSIFADDR: Permission denied
>> SIOCSIFFLAGS: Permission denied
>> SIOCSIFFLAGS: Permission denied
>> /etc/qemu-ifdown: could not launch network script
>>
>> Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.
>>
>> Thanks & Regards,
>> Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata | Follow us on Twitter [at] NTTDATAAmerica
>>
>>
>> ______________________________________________________________________
>> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack [at] lists
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>> ______________________________________________________________________
>> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
>>
>>
>> ______________________________________________________________________
>> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack [at] lists
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
Attachments: NUAGECO-LOGO-Fblan_petit.jpg (9.88 KB)


Unmesh.Gurjar at nttdata

Apr 18, 2012, 12:23 AM

Post #9 of 9 (535 views)
Permalink
Re: Issue in KVM block migration [In reply to]

Thanks Kei. That solved the issue.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica

From: Kei.masumoto [mailto:kei.masumoto [at] gmail]
Sent: Wednesday, April 18, 2012 5:16 AM
To: Gurjar, Unmesh
Cc: openstack [at] lists
Subject: Re: [Openstack] Issue in KVM block migration

/etc/hosts must be exactly same between src and dest compute node. that is your problem.
Kei

(2012/04/13 21:46), Gurjar, Unmesh wrote:
The user running the compute server is a member of libvirt group.
I tried the migration via virsh, however that too returns the same result. I get similar logs in the qemu log file.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 |unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica

From: Razique Mahroua [mailto:razique.mahroua [at] gmail]
Sent: Friday, April 13, 2012 2:22 PM
To: Gurjar, Unmesh
Cc: openstack [at] lists<mailto:openstack [at] lists>
Subject: Re: [Openstack] Issue in KVM block migration

Hi,
sorry for the late reply.
Does the user nova belong to the libvirt group ?
Can you try manually the migration via virsh ?
$ virsh --migrate --live --copy-storage-all $domain qemu+tcp://user [at] serve/sytem

thanks


Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail<mailto:razique.mahroua [at] gmail>

[cid:image001.jpg [at] 01CD1D62]

Le 11 avr. 2012 à 16:44, Gurjar, Unmesh a écrit :



Thanks Razique for taking up this one.

Libvirt version on both Compute hosts:
$ libvirtd --version
libvirtd (libvirt) 0.9.2
$ virsh --version
0.9.2

Here are my libvirtd.conf details:
listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none"


Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica

From: Razique Mahroua [mailto:razique.mahroua [at] gmail]<mailto:[mailto:razique.mahroua [at] gmail]>
Sent: Wednesday, April 11, 2012 7:33 PM
To: Gurjar, Unmesh
Cc: openstack [at] lists<mailto:openstack [at] lists>
Subject: Re: [Openstack] Issue in KVM block migration

Hi,
it looks like the user under which libvirt is running doesn't have the rights to manage server's IF.
What version of libvirt are you using ?
Can I see the file /etc/libvirt/libvirtd.com ? (cat /etc/libvirt/libvirtd.conf | grep -v -e "#" -e "^$

Raz

Nuage & Co - Razique Mahroua
razique.mahroua [at] gmail<mailto:razique.mahroua [at] gmail>

<image001.jpg>

Le 11 avr. 2012 à 14:48, Gurjar, Unmesh a écrit :




Hi,

I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned here<http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>). In addition, I have also disabled apparmor for libvirtd profile and have an entry in '/etc/hosts' of both the Compute hosts.

>From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:
virsh # connect qemu+tcp://ubuntu-dev-001/system
virsh # list

The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:
libvir: QEMU error : operation failed: migration job: unexpectedly failed
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321
2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
readers.get(fileno, noop).cb(fileno)
File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration
recover_method(ctxt, instance_ref, dest, block_migration)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration
FLAGS.live_migration_bandwidth)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI
if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
libvirtError: operation failed: migration job: unexpectedly failed
Removing descriptor: 12

I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:
2012-04-11 04:14:25.971: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd /opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Domain id=15 is tainted: high-privileges
Domain id=15 is tainted: shell-scripts
char device redirected to /dev/pts/17
2012-04-11 04:14:26.406: shutting down
can't delete tapfcfa2a6c-35 from eth1: Operation not supported
SIOCSIFADDR: Permission denied
SIOCSIFFLAGS: Permission denied
SIOCSIFFLAGS: Permission denied
/etc/qemu-ifdown: could not launch network script

Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.

Thanks & Regards,
Unmesh Gurjar | Lead Engineer | Vertex Software Private Ltd. | w. +91.20.6604.1500 x 379 | m. +91.982.324.7631 | unmesh.gurjar [at] nttdata<mailto:unmesh.gurjar [at] nttdata> | Follow us on Twitter [at] NTTDATAAmerica


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
_______________________________________________
Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
Post to : openstack [at] lists<mailto:openstack [at] lists>
Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
More help : https://help.launchpad.net/ListHelp


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding




_______________________________________________

Mailing list: https://launchpad.net/~openstack

Post to : openstack [at] lists<mailto:openstack [at] lists>

Unsubscribe : https://launchpad.net/~openstack

More help : https://help.launchpad.net/ListHelp


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Attachments: image001.jpg (9.88 KB)

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.