Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Xen: Users

UnionFS + Xen

 

 

Xen users RSS feed   Index | Next | Previous | View Threaded


gledesma at gmail

May 25, 2005, 6:57 PM

Post #1 of 8 (210 views)
Permalink
UnionFS + Xen

Hi, list

I've been looking at solutions to copy UML's CoW disks for Xen, but
based on the archives and google results, this is pretty much an
active work-in-progress for Xen. I've looked at CowNFS and it seems
promising, but I'm a bit worried of relying on a machine that will run
it. If and when the daemon conks out, everything relying on it will
die as well (can't get the clients to reestablish a connection with
the server).

What I've been looking at lately is using NFS Root + UnionFS. Having
NFS root means I can use a dedicated machine to act as an NFS server
and tweak it to my heart's content, or maybe even use a NetApp for it.

The only challenge this poses is getting the guest OSs to actually do
the overlaying by mounting a copy of their work space (read/write)
over the exported NFS root (read-only). This way, each domain can
happily write to their own "disk" without having to give them their
own independent disks (which wastes space).

Has anyone gotten something similar to work? Any ideas / tips /
comments / suggestions in doing so? I'm tweaking my xen RPM spec file
to support unionfs and right now just looking at the boot up process
of getting the domain to mount something else on top of the exported
root file system (either a file-backed VBD or yet another NFS export).

gino ledesma

P.S. If there's anything waaaay simpler than this, I'd really like to
know. :-) I'm having fun playing with xen but will need to see how
this setup scales.

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


aliguori at us

May 25, 2005, 7:58 PM

Post #2 of 8 (200 views)
Permalink
Re: UnionFS + Xen [In reply to]

Gino LV. Ledesma wrote:

>Hi, list
>
>Has anyone gotten something similar to work? Any ideas / tips /
>comments / suggestions in doing so? I'm tweaking my xen RPM spec file
>to support unionfs and right now just looking at the boot up process
>of getting the domain to mount something else on top of the exported
>root file system (either a file-backed VBD or yet another NFS export).
>
>
This is actually a large part of the paper for this presentation at OLS:

http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117

There's a number of approaches to solving this problem. unionfs would
be ideal but it's a bit unstable. Another approach is to keep certain
directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
read write (/etc, /var/, etc.). This will get you pretty far.

Regards,

Anthony Liguori

>gino ledesma
>
>P.S. If there's anything waaaay simpler than this, I'd really like to
>know. :-) I'm having fun playing with xen but will need to see how
>this setup scales.
>
>_______________________________________________
>Xen-users mailing list
>Xen-users [at] lists
>http://lists.xensource.com/xen-users
>
>
>


_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


gledesma at gmail

May 25, 2005, 9:40 PM

Post #3 of 8 (200 views)
Permalink
Re: UnionFS + Xen [In reply to]

On 5/25/05, Anthony Liguori <aliguori [at] us> wrote:
> Gino LV. Ledesma wrote:
>
> >Hi, list
> >
> >Has anyone gotten something similar to work? Any ideas / tips /
> >comments / suggestions in doing so? I'm tweaking my xen RPM spec file
> >to support unionfs and right now just looking at the boot up process
> >of getting the domain to mount something else on top of the exported
> >root file system (either a file-backed VBD or yet another NFS export).
> >
> >
> This is actually a large part of the paper for this presentation at OLS:
>
> http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117
>
> There's a number of approaches to solving this problem. unionfs would
> be ideal but it's a bit unstable. Another approach is to keep certain
> directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
> read write (/etc, /var/, etc.). This will get you pretty far.

Thanks for the reply. I guess this is one of the most oftens suggested
approaches. The two goals that I'd like to meet are:
1. Storage flexibility (resize as necessary) -- LVM looks good to go,
though in our setup where we can use NetApps, NFS would be another
approach.
2. Ease of administration / maintenance -- Some of the issues I'm wary
about when going unionfs is that if we make changes to the underlying
filesystem (e.g. OS upgrade) and there are changes on the overlay,
conflicts might occur.

I'll look to doing both -- I'd like to keep things simpler, as there'd
be less chances of breaking things.

Thanks again.

gino ledesma

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


gledesma at gmail

May 28, 2005, 12:55 AM

Post #4 of 8 (200 views)
Permalink
Re: UnionFS + Xen [In reply to]

I've managed to come up with something pretty neat -- a shared base
image with unionfs providing an overlay for a "delta" filesystem that
allows each host to customize the image to their heart's content while
storing only the changes (deltas). I *love* how xen and open source
work. :-)

xen config exports to the domUs a file-backed master.img which
contains a base root filesystem as /dev/hda1. Then each domU gets
their own LVM-backed logical volumes for root (/dev/hdb1) and swap
(/dev/hdc1). This allows me to grow/shrink partitions as necessary.
And because these partitions only contain changes, they don't use as
much data. I customized the initrd process to union mount the base
root filesystem with the "delta" filesystem and overlay that as root.

This works pretty well, so far -- I'm just trying to make this work
with NFS-exported file systems for maximum flexibility.

Anyway, this may not really be a xen issue, but I thought I'd ask. I
was trying to get several domU's to share a base LVM-backed
filesystem, but when I fire up a second domain using the same fs, it
says that the vbd is already in use. I take it isn't possible to have
multiple mounts of an LVM-backed volume? That's one reason why I had
to use a file-backed root fs above.

Also... As I'm not too sure about this, is it *safe* for a file-backed
root fs to be shared across multiple domUs? I have the initrd mount it
as read-only BUT the xen config has to make it writable. Making it
read only in the config causes the mount process to spit an error
(EXT3 INFO requires write access ...).

-gino

On 5/25/05, Gino LV. Ledesma <gledesma [at] gmail> wrote:
> On 5/25/05, Anthony Liguori <aliguori [at] us> wrote:
> > Gino LV. Ledesma wrote:
> >
> > >Hi, list
> > >
> > >Has anyone gotten something similar to work? Any ideas / tips /
> > >comments / suggestions in doing so? I'm tweaking my xen RPM spec file
> > >to support unionfs and right now just looking at the boot up process
> > >of getting the domain to mount something else on top of the exported
> > >root file system (either a file-backed VBD or yet another NFS export).
> > >
> > >
> > This is actually a large part of the paper for this presentation at OLS:
> >
> > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117
> >
> > There's a number of approaches to solving this problem. unionfs would
> > be ideal but it's a bit unstable. Another approach is to keep certain
> > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
> > read write (/etc, /var/, etc.). This will get you pretty far.
>
> Thanks for the reply. I guess this is one of the most oftens suggested
> approaches. The two goals that I'd like to meet are:
> 1. Storage flexibility (resize as necessary) -- LVM looks good to go,
> though in our setup where we can use NetApps, NFS would be another
> approach.
> 2. Ease of administration / maintenance -- Some of the issues I'm wary
> about when going unionfs is that if we make changes to the underlying
> filesystem (e.g. OS upgrade) and there are changes on the overlay,
> conflicts might occur.
>
> I'll look to doing both -- I'd like to keep things simpler, as there'd
> be less chances of breaking things.
>
> Thanks again.
>
> gino ledesma
>

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


mark.williamson at cl

May 28, 2005, 4:33 AM

Post #5 of 8 (199 views)
Permalink
Re: UnionFS + Xen [In reply to]

> Anyway, this may not really be a xen issue, but I thought I'd ask. I
> was trying to get several domU's to share a base LVM-backed
> filesystem, but when I fire up a second domain using the same fs, it
> says that the vbd is already in use. I take it isn't possible to have
> multiple mounts of an LVM-backed volume? That's one reason why I had
> to use a file-backed root fs above.

The Xen tools won't let you have multiple mounts to a filesystem unless
they're all read only. This is just a "safety catch" to avoid people
shooting themselves in the foot. If you're Really Sure you want writeable
sharing, stick "w!" as the permission in the config file.

> Also... As I'm not too sure about this, is it *safe* for a file-backed
> root fs to be shared across multiple domUs?

Not if they're mounting it writeable...

> I have the initrd mount it
> as read-only BUT the xen config has to make it writable. Making it
> read only in the config causes the mount process to spit an error
> (EXT3 INFO requires write access ...).

If you mount it read only in the guest, it's probably OK. It's a bit odd that
exporting it read only doesn't work though - we export /usr read only and
then mount it read only in the guest, which seems to work fine.

HTH,
Mark

> -gino
>
> On 5/25/05, Gino LV. Ledesma <gledesma [at] gmail> wrote:
> > On 5/25/05, Anthony Liguori <aliguori [at] us> wrote:
> > > Gino LV. Ledesma wrote:
> > > >Hi, list
> > > >
> > > >Has anyone gotten something similar to work? Any ideas / tips /
> > > >comments / suggestions in doing so? I'm tweaking my xen RPM spec file
> > > >to support unionfs and right now just looking at the boot up process
> > > >of getting the domain to mount something else on top of the exported
> > > >root file system (either a file-backed VBD or yet another NFS export).
> > >
> > > This is actually a large part of the paper for this presentation at
> > > OLS:
> > >
> > > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117
> > >
> > > There's a number of approaches to solving this problem. unionfs would
> > > be ideal but it's a bit unstable. Another approach is to keep certain
> > > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others
> > > read write (/etc, /var/, etc.). This will get you pretty far.
> >
> > Thanks for the reply. I guess this is one of the most oftens suggested
> > approaches. The two goals that I'd like to meet are:
> > 1. Storage flexibility (resize as necessary) -- LVM looks good to go,
> > though in our setup where we can use NetApps, NFS would be another
> > approach.
> > 2. Ease of administration / maintenance -- Some of the issues I'm wary
> > about when going unionfs is that if we make changes to the underlying
> > filesystem (e.g. OS upgrade) and there are changes on the overlay,
> > conflicts might occur.
> >
> > I'll look to doing both -- I'd like to keep things simpler, as there'd
> > be less chances of breaking things.
> >
> > Thanks again.
> >
> > gino ledesma
>
> _______________________________________________
> Xen-users mailing list
> Xen-users [at] lists
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


lists.xensource.com at mukesh

Jun 9, 2005, 10:26 AM

Post #6 of 8 (201 views)
Permalink
Re: UnionFS + Xen [In reply to]

Hi Gino,

Mind if I ask a couple questions about your UnionFS setup?

> I've managed to come up with something pretty neat -- a shared base
> image with unionfs providing an overlay for a "delta" filesystem that
> allows each host to customize the image to their heart's content while
> storing only the changes (deltas).

Google turns up a couple projects using that name. Are you using the one
from Stony Brook (http://www.fsl.cs.sunysb.edu/project-unionfs.html), or a
different one?

> I customized the initrd process to union mount the baseroot filesystem
> with the "delta" filesystem and overlay that as root.

Are there any pitfalls to watch out for in creating this initrd, or is it
pretty straightforward?

Thanks,
mukesh


_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


gledesma at gmail

Jun 9, 2005, 2:31 PM

Post #7 of 8 (198 views)
Permalink
Re: UnionFS + Xen [In reply to]

On 6/9/05, mukesh agrawal <lists.xensource.com [at] mukesh> wrote:
> Google turns up a couple projects using that name. Are you using the one
> from Stony Brook (http://www.fsl.cs.sunysb.edu/project-unionfs.html), or a
> different one?

Yes, that's the one.

> > I customized the initrd process to union mount the baseroot filesystem
> > with the "delta" filesystem and overlay that as root.
>
> Are there any pitfalls to watch out for in creating this initrd, or is it
> pretty straightforward?

The initrd itself is pretty straightforward, especially if you're
using a file-/physical-/LVM-backed block device. Using NFS gets a
little bit tricky because of the mount procedure, but nothing too
difficult to overcome.

The only issues I'm running into are kernel stack traces during
certain I/O-heavy operations. This has more to do with unionfs I
suppose, rather than the xen kernel itself.

Another one I've looked at recently is mini_fo
<http://www.denx.de/twiki/bin/view/Know/MiniFOHome>, though this
doesn't support overlaying an NFS exported volume yet, and it has some
caching issues I'm looking into.

- gino

_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users


lists.xensource.com at mukesh

Jun 9, 2005, 4:28 PM

Post #8 of 8 (203 views)
Permalink
Re: UnionFS + Xen [In reply to]

Thanks for your replies. I've used it to start a Wiki page with
information on the various possibilities for copy-on-write functionality.

See http://wiki.xensource.com/xenwiki/COWHowTo.


_______________________________________________
Xen-users mailing list
Xen-users [at] lists
http://lists.xensource.com/xen-users

Xen users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.