Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

Improving Xen support in the libvirt driver

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


jfehlig at suse

May 9, 2012, 10:08 PM

Post #1 of 13 (855 views)
Permalink
Improving Xen support in the libvirt driver

Hi,

I've been tinkering with improving Xen support in the libvirt driver and
wanted to discuss a few issues before submitting patches.

Even the latest upstream release of Xen (4.1.x) contains a rather old
qemu, version 0.10.2, which rejects qcow2 images with cluster size >
64K. The libvirt driver creates the COW image with cluster size of 2M.
Is this for performance reasons? Any objections to removing that option
and going with 'qemu-img create' default of 64K?

In a setup with both Xen and KVM compute nodes, I've found a few options
for controlling scheduling of an instance to the correct node. One
option uses availability zones, e.g.

# nova.conf on Xen compute nodes
node_availability_zone=xen-hosts

# launching a Xen PV instance
nova boot --image <xen-pv-image> --availability_zone xen-hosts ...

The other involves a recent commit adding additional capabilities for
compute nodes [1] and the vm_mode image property [2] used by the
XenServer driver to distinguish HVM vs PV images. E.g.

# nova.conf on Xen compute nodes
additional_compute_capabilities="pv,hvm"

# Set vm_mode property on Xen image
glance update <image-uuid> vm_mode=pv

I prefer that latter approach since vm_mode will be needed in the
libvirt driver anyhow to create proper config for PV vs HVM instances.
Currently, the driver creates usable config for PV instances, but needs
some adjustments for HVM.

Regards,
Jim

[1]
https://github.com/openstack/nova/commit/bd30eb36bbf2c5164ac47256355c543f6b77e064
[2]
https://github.com/openstack/nova/commit/bd30eb36bbf2c5164ac47256355c543f6b77e064


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


berrange at redhat

May 10, 2012, 1:06 AM

Post #2 of 13 (848 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On Wed, May 09, 2012 at 11:08:13PM -0600, Jim Fehlig wrote:
> Hi,
>
> I've been tinkering with improving Xen support in the libvirt driver and
> wanted to discuss a few issues before submitting patches.
>
> Even the latest upstream release of Xen (4.1.x) contains a rather old
> qemu, version 0.10.2, which rejects qcow2 images with cluster size >
> 64K. The libvirt driver creates the COW image with cluster size of 2M.
> Is this for performance reasons? Any objections to removing that option
> and going with 'qemu-img create' default of 64K?

In general larger cluster size does improve the performance of
qcow2. I'm not sure how much of a delta we get by going from
64k to 2M though. If there's any doubt then I guess it could be
made into a configuration parameter.

> In a setup with both Xen and KVM compute nodes, I've found a few options
> for controlling scheduling of an instance to the correct node. One
> option uses availability zones, e.g.
>
> # nova.conf on Xen compute nodes
> node_availability_zone=xen-hosts
>
> # launching a Xen PV instance
> nova boot --image <xen-pv-image> --availability_zone xen-hosts ...
>
> The other involves a recent commit adding additional capabilities for
> compute nodes [1] and the vm_mode image property [2] used by the
> XenServer driver to distinguish HVM vs PV images. E.g.
>
> # nova.conf on Xen compute nodes
> additional_compute_capabilities="pv,hvm"
>
> # Set vm_mode property on Xen image
> glance update <image-uuid> vm_mode=pv
>
> I prefer that latter approach since vm_mode will be needed in the
> libvirt driver anyhow to create proper config for PV vs HVM instances.
> Currently, the driver creates usable config for PV instances, but needs
> some adjustments for HVM.

Yes, tagging the image with details of its required guest ABI does
seem like something we need to do to be able to properly support
a choice betweeen PV & HVM images. It is not very good the way we
currently just hardcode PV only for Xen usage in the libvirt driver.

Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


mucawhite at gmail

May 10, 2012, 1:41 AM

Post #3 of 13 (846 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

Il 10/05/2012 07:08, Jim Fehlig ha scritto:
> Hi,
>
> I've been tinkering with improving Xen support in the libvirt driver and
> wanted to discuss a few issues before submitting patches.
>
> Even the latest upstream release of Xen (4.1.x) contains a rather old
> qemu, version 0.10.2, which rejects qcow2 images with cluster size>
> 64K. The libvirt driver creates the COW image with cluster size of 2M.
> Is this for performance reasons? Any objections to removing that option
> and going with 'qemu-img create' default of 64K?
>
>
If I remember correctly, the qcow images are not the only problem with
xen, but I'm far from the code for too long time. In the past (diablo),
the method for counting the ram (and cpu perhaps?) did not work with xen
and this affected the choices of the scheduler. I have no idea if this
happens in essex/folsom.

Regards,
Muriel


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


berrange at redhat

May 10, 2012, 1:56 AM

Post #4 of 13 (851 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On Thu, May 10, 2012 at 09:06:58AM +0100, Daniel P. Berrange wrote:
> On Wed, May 09, 2012 at 11:08:13PM -0600, Jim Fehlig wrote:
> > Hi,
> >
> > I've been tinkering with improving Xen support in the libvirt driver and
> > wanted to discuss a few issues before submitting patches.
> >
> > Even the latest upstream release of Xen (4.1.x) contains a rather old
> > qemu, version 0.10.2, which rejects qcow2 images with cluster size >
> > 64K. The libvirt driver creates the COW image with cluster size of 2M.
> > Is this for performance reasons? Any objections to removing that option
> > and going with 'qemu-img create' default of 64K?
>
> In general larger cluster size does improve the performance of
> qcow2. I'm not sure how much of a delta we get by going from
> 64k to 2M though. If there's any doubt then I guess it could be
> made into a configuration parameter.

I had a quick chat with Kevin Wolf who's the upstream QEMU qcow2 maintainer
and he said that 64k is the current recommended cluster size for qcow2.
Above this size, the cost of COW becomes higher causing an overall
drop in performance.

Looking at GIT history, Nova has used cluster_size=2M since Vish first
added qcow2 support, and there's no mention of why in the commit message.
So unless further info comes to light, I'd say we ought to just switch
to use qemu-img's default setting of 64K for both Xen and KVM.

Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


aloga at ifca

May 10, 2012, 2:48 AM

Post #5 of 13 (840 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On Thu 10 May 2012 (10:41), Muriel wrote:
> If I remember correctly, the qcow images are not the only problem
> with xen, but I'm far from the code for too long time. In the past
> (diablo), the method for counting the ram (and cpu perhaps?) did not
> work with xen and this affected the choices of the scheduler. I have
> no idea if this happens in essex/folsom.

I've sent to review some code [1] that tries to fix this issue [2].

[1] https://review.openstack.org/#/c/7296/
[2] https://bugs.launchpad.net/nova/+bug/997014

Regards,
--
Álvaro López García aloga [at] ifca



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


mucawhite at gmail

May 10, 2012, 6:17 AM

Post #6 of 13 (842 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

Il 10/05/2012 11:48, Alvaro Lopez ha scritto:
> On Thu 10 May 2012 (10:41), Muriel wrote:
>> If I remember correctly, the qcow images are not the only problem
>> with xen, but I'm far from the code for too long time. In the past
>> (diablo), the method for counting the ram (and cpu perhaps?) did not
>> work with xen and this affected the choices of the scheduler. I have
>> no idea if this happens in essex/folsom.
> I've sent to review some code [1] that tries to fix this issue [2].
>
> [1] https://review.openstack.org/#/c/7296/
> [2] https://bugs.launchpad.net/nova/+bug/997014
>
> Regards,
Great! But there is a reason if are you using /proc/meminfo instead of
getInfo when calculating the memory used?
You know if there is a way to get, using libvirt, the reserved memory
for dom0? Or the only solution is to read the configuration file of xen?

Thanks,
Muriel


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


berrange at redhat

May 10, 2012, 6:41 AM

Post #7 of 13 (855 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On Thu, May 10, 2012 at 03:17:59PM +0200, Muriel wrote:
> Il 10/05/2012 11:48, Alvaro Lopez ha scritto:
> >On Thu 10 May 2012 (10:41), Muriel wrote:
> >>If I remember correctly, the qcow images are not the only problem
> >>with xen, but I'm far from the code for too long time. In the past
> >>(diablo), the method for counting the ram (and cpu perhaps?) did not
> >>work with xen and this affected the choices of the scheduler. I have
> >>no idea if this happens in essex/folsom.
> >I've sent to review some code [1] that tries to fix this issue [2].
> >
> >[1] https://review.openstack.org/#/c/7296/
> >[2] https://bugs.launchpad.net/nova/+bug/997014
> >
> >Regards,
> Great! But there is a reason if are you using /proc/meminfo instead
> of getInfo when calculating the memory used?
> You know if there is a way to get, using libvirt, the reserved
> memory for dom0? Or the only solution is to read the configuration
> file of xen?

Dom0 appears as just another guest in Xen/libvirt, so you can query
its memory allocation using normal libvirt APIs

Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


aloga at ifca

May 10, 2012, 7:08 AM

Post #8 of 13 (855 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On Thu 10 May 2012 (15:17), Muriel wrote:
> Great! But there is a reason if are you using /proc/meminfo instead
> of getInfo when calculating the memory used?
> You know if there is a way to get, using libvirt, the reserved
> memory for dom0? Or the only solution is to read the configuration
> file of xen?

I calculated the memory looking into /proc/meminfo because if the memory
is not limited (i.e. no dom0_mem option) the dom0 might take all the
memory available, that then will be ballooned out. For example, in a
machine with 16GB RAM you could have:

# xm li
Name ID Mem VCPUs State Time(s)
Domain-0 0 15030 8 r----- 1312.8

If you query libvirt for the dom0 mem, the free memory will be around
1GB, but you can create a machine with more RAM (since ballooning is
enabled):

# xm li
Name ID Mem VCPUs State Time(s)
Domain-0 0 9188 8 r----- 1328.6
test 4 7000 4 -b---- 3.5

If the dom0 memory is fixed and ballooning is disabled, then yes, you
can query libvirt directly.

Regards,
--
Álvaro López García aloga [at] ifca



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


mucawhite at gmail

May 10, 2012, 8:33 AM

Post #9 of 13 (864 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

Il 10/05/2012 16:08, Alvaro Lopez ha scritto:
> On Thu 10 May 2012 (15:17), Muriel wrote:
>> Great! But there is a reason if are you using /proc/meminfo instead
>> of getInfo when calculating the memory used?
>> You know if there is a way to get, using libvirt, the reserved
>> memory for dom0? Or the only solution is to read the configuration
>> file of xen?
> I calculated the memory looking into /proc/meminfo because if the memory
> is not limited (i.e. no dom0_mem option) the dom0 might take all the
> memory available, that then will be ballooned out. For example, in a
> machine with 16GB RAM you could have:
>
> # xm li
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 15030 8 r----- 1312.8
>
> If you query libvirt for the dom0 mem, the free memory will be around
> 1GB, but you can create a machine with more RAM (since ballooning is
> enabled):
>
> # xm li
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 9188 8 r----- 1328.6
> test 4 7000 4 -b---- 3.5
>
> If the dom0 memory is fixed and ballooning is disabled, then yes, you
> can query libvirt directly.
>
> Regards,
Thank you for remarking this point, sometimes I forget the balooning
system :).
I messed up the question: if you consider the case where the memory is
limited (dom0_mem) the value of MemTotal in meminfo is wrong. Do you
think it makes sense to take the total memory value from libvirt and
from meminfo the rest?
Thus it should work in both cases.

Thanks,
M.

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


aloga at ifca

May 10, 2012, 8:49 AM

Post #10 of 13 (848 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On Thu 10 May 2012 (17:33), Muriel wrote:
> I messed up the question: if you consider the case where the memory
> is limited (dom0_mem) the value of MemTotal in meminfo is wrong. Do
> you think it makes sense to take the total memory value from libvirt
> and from meminfo the rest?
> Thus it should work in both cases.

Yes, you're totally right, I missed that!

I'll fix it ASAP and submit it again.

Thank you for pointing it out,
--
Álvaro López García aloga [at] ifca



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


vishvananda at gmail

May 10, 2012, 10:17 AM

Post #11 of 13 (843 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On May 10, 2012, at 1:56 AM, Daniel P. Berrange wrote:

> On Thu, May 10, 2012 at 09:06:58AM +0100, Daniel P. Berrange wrote:
>
> I had a quick chat with Kevin Wolf who's the upstream QEMU qcow2 maintainer
> and he said that 64k is the current recommended cluster size for qcow2.
> Above this size, the cost of COW becomes higher causing an overall
> drop in performance.
>
> Looking at GIT history, Nova has used cluster_size=2M since Vish first
> added qcow2 support, and there's no mention of why in the commit message.
> So unless further info comes to light, I'd say we ought to just switch
> to use qemu-img's default setting of 64K for both Xen and KVM.
>

This is good info. Sounds like we should switch to 64K

Vish


vishvananda at gmail

May 10, 2012, 10:20 AM

Post #12 of 13 (843 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

On May 9, 2012, at 10:08 PM, Jim Fehlig wrote:

> Hi,
>
> I've been tinkering with improving Xen support in the libvirt driver and
> wanted to discuss a few issues before submitting patches.

Awesome!

>
> Even the latest upstream release of Xen (4.1.x) contains a rather old
> qemu, version 0.10.2, which rejects qcow2 images with cluster size >
> 64K. The libvirt driver creates the COW image with cluster size of 2M.
> Is this for performance reasons? Any objections to removing that option
> and going with 'qemu-img create' default of 64K?

As per other email, 64K seems correct.
>
> In a setup with both Xen and KVM compute nodes, I've found a few options
> for controlling scheduling of an instance to the correct node. One
> option uses availability zones, e.g.
>
> # nova.conf on Xen compute nodes
> node_availability_zone=xen-hosts
>
> # launching a Xen PV instance
> nova boot --image <xen-pv-image> --availability_zone xen-hosts ...
>
> The other involves a recent commit adding additional capabilities for
> compute nodes [1] and the vm_mode image property [2] used by the
> XenServer driver to distinguish HVM vs PV images. E.g.
>
> # nova.conf on Xen compute nodes
> additional_compute_capabilities="pv,hvm"
>
> # Set vm_mode property on Xen image
> glance update <image-uuid> vm_mode=pv
>
> I prefer that latter approach since vm_mode will be needed in the
> libvirt driver anyhow to create proper config for PV vs HVM instances.
> Currently, the driver creates usable config for PV instances, but needs
> some adjustments for HVM.

Agree that this is best. Once general host aggregates[1] is done, the capabilities and the availability zone will move into aggregate metadata and it will just be making sure that we have reasonable image properties to help the scheduler place the guest correctly.

Vish

[1] https://blueprints.launchpad.net/nova/+spec/general-host-aggregates


jfehlig at suse

May 10, 2012, 12:50 PM

Post #13 of 13 (847 views)
Permalink
Re: Improving Xen support in the libvirt driver [In reply to]

Vishvananda Ishaya wrote:
>
> On May 10, 2012, at 1:56 AM, Daniel P. Berrange wrote:
>
>> On Thu, May 10, 2012 at 09:06:58AM +0100, Daniel P. Berrange wrote:
>>
>> I had a quick chat with Kevin Wolf who's the upstream QEMU qcow2
>> maintainer
>> and he said that 64k is the current recommended cluster size for qcow2.
>> Above this size, the cost of COW becomes higher causing an overall
>> drop in performance.
>>
>> Looking at GIT history, Nova has used cluster_size=2M since Vish first
>> added qcow2 support, and there's no mention of why in the commit message.
>> So unless further info comes to light, I'd say we ought to just switch
>> to use qemu-img's default setting of 64K for both Xen and KVM.
>>
>
> This is good info. Sounds like we should switch to 64K

Right. I'll submit a patch for that shortly.

Regards,
Jim


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.