diego.parrilla.santamaria at gmail
Jun 9, 2012, 5:16 AM
Post #6 of 6
(541 views)
Permalink
|
|
Re: [Live Migration] shared /var/lib/nova/instances versus Performance
[In reply to]
|
|
We use NFS in private cloud deployments (for instances, volumes and image
storage) and performance compared to Gluster is better (we supported
Gluster till Red Hat's acquisition). Obviously, you need a good filer and
something (much) better than 1G.
Still, massive scaling (thousands of VMs) probably is not what a shared
filesystem is made for.
Parallel NFS is promising, and I would like to hear of the experience of
people using it.
Cheers
Diego
--
Diego Parrilla
<http://www.stackops.com/>*CEO*
*www.stackops.com | * diego.parrilla [at] stackops** | +34 649 94 43 29 |
skype:diegoparrilla*
* <http://www.stackops.com/>
*
*
On Sat, Jun 9, 2012 at 3:15 AM, Christian Parpart <trapni [at] gmail> wrote:
> On Fri, Jun 8, 2012 at 7:59 PM, Igor Laskovy <igor.laskovy [at] gmail>wrote:
>
>> Just interested to me, what the IO errors you have received? Is it IO
>> latency related?
>> Is it during peak load? What traffic and how much IOps do you have at
>> the peak when receiving log?
>>
>
> i've never measured the iops of raw nfs traffic, but the I/O errors are
> mostly EIO and a few ESTALE, AFAIR.
> the iops should not be that much, as we're just delivering static images
> via nfs, and we've a CDN in front.
>
> However, we're OpenStack's /var/lib/nova/instances is a whole different
> story :)
>
> Cheers,
> Christian.
>
>
>>
>> On Fri, Jun 8, 2012 at 4:37 PM, Christian Parpart <trapni [at] gmail>
>> wrote:
>> > On Fri, Jun 8, 2012 at 3:23 PM, Igor Laskovy <igor.laskovy [at] gmail>
>> wrote:
>> >>
>> >> Hello Christian,
>> >>
>> >> just want to clarify what do you hardly dislike of using NFS?
>> >
>> >
>> > we're currently using NFS in our legacy virtualization stack, and over
>> the
>> > day, we get
>> > quite a few logs of I/O errors wrt. NFS.
>> >
>> > maybe we can fix this, but then we're having again a single point of
>> > failure, and every
>> > I/O operation goes through a single node (the NFS master). maybe not
>> what I
>> > want
>> > to do when having 50 nodes with live VM images on it :-)
>> >
>> > Regards,
>> > Christian.
>> >>
>> >>
>> >> On Fri, Jun 8, 2012 at 2:22 PM, Christian Parpart <trapni [at] gmail>
>> >> wrote:
>> >> > Hey all,
>> >> >
>> >> > I hardly dislike the idea of using NFS as shared storage so
>> something so
>> >> > important as nova-compute,
>> >> > however, I still want to be able to perform live migrations, and so I
>> >> > found
>> >> > out, that one may want
>> >> > to use Glusterfs for that.
>> >> >
>> >> > But what's the best setup for Glusterfs volumes for use within
>> >> > nova-compute
>> >> > as shared instance dir storage?
>> >> >
>> >> > I'm asking for advice, since - in the end - we're going to have about
>> >> > 50+
>> >> > nova-compute nodes, and all
>> >> > will be running KVM instances - and all will be accessing the same
>> >> > shared
>> >> > folder? - Sounds like quite
>> >> > a lot of fun the underlying file system server, though.
>> >> >
>> >> > I first thought, Glusterfs to the rescue, it is perfect for
>> replication,
>> >> > and
>> >> > supports spreading,
>> >> > and since all compute nodes are set up with 2x1TB storage, I silently
>> >> > assumed, I could just
>> >> > setup the nova-compute's node as a backend for the shared storage,
>> too,
>> >> > a so
>> >> > called "brick",
>> >> > and then assume, that this compute node (knowing the glusterfs volume
>> >> > topology) just uses
>> >> > the local brick as first try, and thus, I should have not a big iops
>> >> > loss
>> >> > and more important, not
>> >> > that much networking I/O, since it first reads locally.
>> >> >
>> >> > Now someone in #glusterfs declined that (so intuitive to me)
>> behaviour
>> >> > of
>> >> > glusterfs, and I now fear,
>> >> > that if I am going that way, I'll end up having veeery very bad disk
>> I/O
>> >> > performance on host
>> >> > and VM side.
>> >> >
>> >> > So what are you using in order to have all instance images
>> replicated or
>> >> > high availability and
>> >> > live migrations for how many nova-compute nodes with what success?
>> >> >
>> >> > I would like to choose the right way before it is too late :)
>> >> >
>> >> > Many thanks in advance,
>> >> > Christian Parpart.
>> >> >
>> >> > _______________________________________________
>> >> > Openstack-operators mailing list
>> >> > Openstack-operators [at] lists
>> >> >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Igor Laskovy
>> >> Kiev, Ukraine
>> >
>> >
>>
>>
>>
>> --
>> Igor Laskovy
>> Kiev, Ukraine
>>
>
>
> _______________________________________________
> Openstack-operators mailing list
> Openstack-operators [at] lists
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
|
