Mailing List Archive: OpenStack: Dev

[Glance] Replication implementations

Index | Next | Previous | View Threaded

michael.still at canonical

May 2, 2012, 8:24 PM

Post #1 of 13 (239 views)
Permalink

[Glance] Replication implementations

Hi.

I'd be interested in hearing from people who have implemented some form
of replication with glance. I'm especially interested in how you went
about it. I attended the session at the dev summit, but that was forward
looking, and I am pretty sure that there wasn't any mention of current
implementations.

I'll get the ball rolling -- I have just implemented API level
replication for glance for an internal project. It uses the glance API
to replicate images from a master glance to slave glances. Its not very
smart at the moment, it just copies everything.

Current warts:
- no support for metadata update
- no support for filtering of what is replicated
- maintaining amazon ec2 ids across regions requires twiddling the nova
database where this mapping is stored

Advantages of this approach:
- API level replication means that I can run different versions of
glance at each end of the replication relationship
- it really wasn't that hard to do

Would other people be interested in getting this code into glance,
probably as a stand alone command line tool? Or has someone done
something much better that I should be using instead?

Cheers,
Mikal

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

brian.waldon at rackspace

May 3, 2012, 11:07 AM

Post #2 of 13 (232 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

I think the path forward is first to gather information on what people are already doing w.r.t. replication, which you have helped trigger with this email. I'm definitely interested in seeing your solution. Once we get this information out in the open, we need to explore these existing solutions can be included with Glance. I'd prefer to go with an external tool that handles replication rather than baking it directly into the code right now, but if the work has already been done, I'm more than happy to consider it.

Brian

On May 2, 2012, at 8:24 PM, Michael Still wrote:

> Hi.
>
> I'd be interested in hearing from people who have implemented some form
> of replication with glance. I'm especially interested in how you went
> about it. I attended the session at the dev summit, but that was forward
> looking, and I am pretty sure that there wasn't any mention of current
> implementations.
>
> I'll get the ball rolling -- I have just implemented API level
> replication for glance for an internal project. It uses the glance API
> to replicate images from a master glance to slave glances. Its not very
> smart at the moment, it just copies everything.
>
> Current warts:
> - no support for metadata update
> - no support for filtering of what is replicated
> - maintaining amazon ec2 ids across regions requires twiddling the nova
> database where this mapping is stored
>
> Advantages of this approach:
> - API level replication means that I can run different versions of
> glance at each end of the replication relationship
> - it really wasn't that hard to do
>
> Would other people be interested in getting this code into glance,
> probably as a stand alone command line tool? Or has someone done
> something much better that I should be using instead?
>
> Cheers,
> Mikal
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

michael.still at canonical

May 4, 2012, 12:39 AM

Post #3 of 13 (230 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

On 04/05/12 04:07, Brian Waldon wrote:
> I think the path forward is first to gather information on what
> people are already doing w.r.t. replication, which you have helped
> trigger with this email. I'm definitely interested in seeing your
> solution. Once we get this information out in the open, we need to
> explore these existing solutions can be included with Glance. I'd
> prefer to go with an external tool that handles replication rather
> than baking it directly into the code right now, but if the work has
> already been done, I'm more than happy to consider it.

My code is a separate tool, and I agree that's the way to go. It makes
it much easier for systems integrators to tweak how it works to meet
their needs, and if we find something that you can't do via the API that
probably indicates a gap in the API more than anything else.

I will try and make some time early next week to clean up the code
enough to show it to you without you laughing at me.

Mikal

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

eglynn at redhat

May 4, 2012, 3:31 AM

Post #4 of 13 (232 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

> Current warts:
> ...
> - maintaining amazon ec2 ids across regions requires twiddling the
> nova database where this mapping is stored

Hi Mikal,

We discussed that nova s3_images table earlier in the week on IRC.
Now at the time, I wasn't fully clear on the mechanics of the glance
UUID -> EC2-style image ID mapping.

I've had a dig into the code since then and it turns out that the EC2
image ID is effectively determined by an auto_increment column in the
s3_images table.

So there is nothing globally significant about that ID, in fact it's
completely determined by the insertion order into the nova s3_images
table, which occurs *on demand* the first time the image is being
referenced in an EC2-related context (for example the first call to
DescribeImages after the image has been uploaded to glance).

For example in a fresh openstack deployment, if three images are
uploaded with types aki, ari & aki respectively, the first EC2-related
reference to the images will result in the following EC2 IDs always
being generated:

aki-00000001
ari-00000002
ami-00000003

Now, if the underlying glance images are migrated to another
long-standing openstack deployment, it's likely that there are already
multiple images referenced in the local s3_images table, so that the
identifiers a[krm]i-0000000[1-3] are already taken.

So the way things currently stand, the EC2 image ID isn't really capable of
migration.

I was thinking however that we should change the EC2 image generation logic,
so that there is a reproducible glance UUID -> EC2 mapping (with a small
chance of collision). This change would allow the same EC2 ID to be generated
in multiple regions for a given glance UUID (modulo collisions).

Would that be helpful in your migration use-case?

Cheers,
Eoghan

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

eglynn at redhat

May 4, 2012, 4:09 AM

Post #5 of 13 (241 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

> > Current warts:
> > ...
> > - maintaining amazon ec2 ids across regions requires twiddling the
> > nova database where this mapping is stored
>
>
> Hi Mikal,
>
> We discussed that nova s3_images table earlier in the week on IRC.
> Now at the time, I wasn't fully clear on the mechanics of the glance
> UUID -> EC2-style image ID mapping.
>
> I've had a dig into the code since then and it turns out that the EC2
> image ID is effectively determined by an auto_increment column in the
> s3_images table.
>
> So there is nothing globally significant about that ID, in fact it's
> completely determined by the insertion order into the nova s3_images
> table, which occurs *on demand* the first time the image is being
> referenced in an EC2-related context (for example the first call to
> DescribeImages after the image has been uploaded to glance).
>
> For example in a fresh openstack deployment, if three images are
> uploaded with types aki, ari & aki respectively, the first
> EC2-related
> reference to the images will result in the following EC2 IDs always
> being generated:
>
> aki-00000001
> ari-00000002
> ami-00000003
>
> Now, if the underlying glance images are migrated to another
> long-standing openstack deployment, it's likely that there are
> already
> multiple images referenced in the local s3_images table, so that the
> identifiers a[krm]i-0000000[1-3] are already taken.
>
> So the way things currently stand, the EC2 image ID isn't really
> capable of
> migration.
>
> I was thinking however that we should change the EC2 image generation
> logic,
> so that there is a reproducible glance UUID -> EC2 mapping (with a
> small
> chance of collision). This change would allow the same EC2 ID to be
> generated
> in multiple regions for a given glance UUID (modulo collisions).
>
> Would that be helpful in your migration use-case?

Of course, the other way of looking at this would be to just follow the
lead of EC2 itself and not provide an image ID portability across regions.

Though I think we can do better than that by compressing the UUID in a
predictable way.

Cheers,
Eoghan

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

jaypipes at gmail

May 7, 2012, 6:11 AM

Post #6 of 13 (231 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

On 05/02/2012 11:24 PM, Michael Still wrote:
> Hi.
>
> I'd be interested in hearing from people who have implemented some form
> of replication with glance. I'm especially interested in how you went
> about it. I attended the session at the dev summit, but that was forward
> looking, and I am pretty sure that there wasn't any mention of current
> implementations.
>
> I'll get the ball rolling -- I have just implemented API level
> replication for glance for an internal project. It uses the glance API
> to replicate images from a master glance to slave glances. Its not very
> smart at the moment, it just copies everything.
>
> Current warts:
> - no support for metadata update
> - no support for filtering of what is replicated
> - maintaining amazon ec2 ids across regions requires twiddling the nova
> database where this mapping is stored
>
> Advantages of this approach:
> - API level replication means that I can run different versions of
> glance at each end of the replication relationship
> - it really wasn't that hard to do
>
> Would other people be interested in getting this code into glance,
> probably as a stand alone command line tool? Or has someone done
> something much better that I should be using instead?

To make your solution look incredibly complex, in TryStack, we do image
replication with a Jenkins job that runs a two line bash script every 20
minutes or so that rsyncs the image_data_dir from our active Glance API
server (with filesystem backend) to our passive failover Glance API
server. We use a simple DRBD HA setup for the MySQL server so we aren't
concerned about the Glance registry server.

So there! I have officially one-upped your simple solution ;)

-jay

p.s. Yes, I know it's not cross-region replication, but figured I would
get in on the conversation anyway ;)

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

michael.still at canonical

May 8, 2012, 12:24 AM

Post #7 of 13 (224 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

On 04/05/12 20:31, Eoghan Glynn wrote:

Sorry for the slow reply, I've been trapped in meetings.

[snip]

> So the way things currently stand, the EC2 image ID isn't really capable of
> migration.
>
> I was thinking however that we should change the EC2 image generation logic,
> so that there is a reproducible glance UUID -> EC2 mapping (with a small
> chance of collision). This change would allow the same EC2 ID to be generated
> in multiple regions for a given glance UUID (modulo collisions).
>
> Would that be helpful in your migration use-case?

I do think this is a good idea. Or even if the column wasn't
auto-increment, but just picked a random number or something (because
that would be marginally less likely to clash). Without somehow making
these ec2 ids more global, replication between regions is going to
suffer from ec2 api users having to somehow perform a lookup out of band.

Now, my use case is a bit special, because I can enforce that images are
only ever uploaded to one master region, and then copied to all others.
I think that's probably not true for other users though.

Mikal

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

vishvananda at gmail

May 8, 2012, 11:34 AM

Post #8 of 13 (224 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

Alternatively, we could just consider the ec2 mapping layer to be global data that must be replicated somehow across the system. I don't think we can really ensure no collisions mapping from uuid -> ec2_id deterministically, and I don't see a clear path forward when we do get a collision.

Vish

On May 8, 2012, at 12:24 AM, Michael Still wrote:

> On 04/05/12 20:31, Eoghan Glynn wrote:
>
> Sorry for the slow reply, I've been trapped in meetings.
>
> [snip]
>
>> So the way things currently stand, the EC2 image ID isn't really capable of
>> migration.
>>
>> I was thinking however that we should change the EC2 image generation logic,
>> so that there is a reproducible glance UUID -> EC2 mapping (with a small
>> chance of collision). This change would allow the same EC2 ID to be generated
>> in multiple regions for a given glance UUID (modulo collisions).
>>
>> Would that be helpful in your migration use-case?
>
> I do think this is a good idea. Or even if the column wasn't
> auto-increment, but just picked a random number or something (because
> that would be marginally less likely to clash). Without somehow making
> these ec2 ids more global, replication between regions is going to
> suffer from ec2 api users having to somehow perform a lookup out of band.
>
> Now, my use case is a bit special, because I can enforce that images are
> only ever uploaded to one master region, and then copied to all others.
> I think that's probably not true for other users though.
>
> Mikal
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

eglynn at redhat

May 8, 2012, 2:29 PM

Post #9 of 13 (217 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

I'm working on a patch to at least make the glance UUID -> S3 image ID
not totally depend on an on-demand insertion order as it does now.

Agreed, collisions are inevitable given the relative domain and range
sizes (122 unique bit UUID versus 32-bit hex string) - in testing,
the first colliding UUID tends to occur after ~75k-80k images IDs
have been generated.

So at least it would be useful for smaller deployments to have a
semi-predictable ID mapping (modulo collisions).

For larger deployments, the mapping data to be replicated could be
much reduced by limiting it to the colliding IDs.

Cheers,
Eoghan

> Alternatively, we could just consider the ec2 mapping layer to be
> global data that must be replicated somehow across the system. I
> don't think we can really ensure no collisions mapping from uuid ->
> ec2_id deterministically, and I don't see a clear path forward when
> we do get a collision.
>
> Vish
>
> On May 8, 2012, at 12:24 AM, Michael Still wrote:
>
> > On 04/05/12 20:31, Eoghan Glynn wrote:
> >
> > Sorry for the slow reply, I've been trapped in meetings.
> >
> > [snip]
> >
> >> So the way things currently stand, the EC2 image ID isn't really
> >> capable of
> >> migration.
> >>
> >> I was thinking however that we should change the EC2 image
> >> generation logic,
> >> so that there is a reproducible glance UUID -> EC2 mapping (with a
> >> small
> >> chance of collision). This change would allow the same EC2 ID to
> >> be generated
> >> in multiple regions for a given glance UUID (modulo collisions).
> >>
> >> Would that be helpful in your migration use-case?
> >
> > I do think this is a good idea. Or even if the column wasn't
> > auto-increment, but just picked a random number or something
> > (because
> > that would be marginally less likely to clash). Without somehow
> > making
> > these ec2 ids more global, replication between regions is going to
> > suffer from ec2 api users having to somehow perform a lookup out of
> > band.
> >
> > Now, my use case is a bit special, because I can enforce that
> > images are
> > only ever uploaded to one master region, and then copied to all
> > others.
> > I think that's probably not true for other users though.
> >
> > Mikal
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

gabe.westmaas at RACKSPACE

May 8, 2012, 4:01 PM

Post #10 of 13 (217 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

This actually sounds like it covers most of the first run needs in this
etherpad:

http://etherpad.openstack.org/FolsomGlanceImageReplication

Definitely excited to see the tool, and then I wonder if we can just add a
few things to it to get metadata and filtering support. Sounds like ec2
stuff is being covered.

Would it make sense to use image metadata to communicate where (and
therefore if) an image should be replicated to? I'm imagining a list of
endpoints that the tool could read and then copy the image to (unless the
tool was running in copy all to endpoint X mode as I imagine it does now).
Auth is going to be an issue here if a glance endpoint is specified that
doesn't share the operator's auth. This isn't terribly transparent to the
customer either since there isn't really a way to set any sort of fault
for a customer to see.

I guess one other thing to note is that we definitely have different use
cases: many operators probably want full control of what gets replicated,
whereas we are happy to let customers choose which specific images to
replicate and then charge for that service. Hopefully we can come to
something that supports both.

This isn't required for the first run, but we could add a property that
indicates the "replicated at" time on the destination so that we can just
compare the updated at of the master image with the replicated at on the
slave, and if they match, there's no need to re-replicate. Possibly
that's just as expensive as replicating all the metadata though.

Gabe

On 5/2/12 10:24 PM, "Michael Still" <michael.still [at] canonical> wrote:

>Hi.
>
>I'd be interested in hearing from people who have implemented some form
>of replication with glance. I'm especially interested in how you went
>about it. I attended the session at the dev summit, but that was forward
>looking, and I am pretty sure that there wasn't any mention of current
>implementations.
>
>I'll get the ball rolling -- I have just implemented API level
>replication for glance for an internal project. It uses the glance API
>to replicate images from a master glance to slave glances. Its not very
>smart at the moment, it just copies everything.
>
>Current warts:
> - no support for metadata update
> - no support for filtering of what is replicated
> - maintaining amazon ec2 ids across regions requires twiddling the nova
>database where this mapping is stored
>
>Advantages of this approach:
> - API level replication means that I can run different versions of
>glance at each end of the replication relationship
> - it really wasn't that hard to do
>
>Would other people be interested in getting this code into glance,
>probably as a stand alone command line tool? Or has someone done
>something much better that I should be using instead?
>
>Cheers,
>Mikal
>
>_______________________________________________
>Mailing list: https://launchpad.net/~openstack
>Post to : openstack [at] lists
>Unsubscribe : https://launchpad.net/~openstack
>More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

eglynn at redhat

May 10, 2012, 6:17 AM

Post #11 of 13 (203 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

BTW that patch is up for review as:

https://review.openstack.org/7302

Cheers,
eoghan

> I'm working on a patch to at least make the glance UUID -> S3 image
> ID
> not totally depend on an on-demand insertion order as it does now.
>
> Agreed, collisions are inevitable given the relative domain and range
> sizes (122 unique bit UUID versus 32-bit hex string) - in testing,
> the first colliding UUID tends to occur after ~75k-80k images IDs
> have been generated.
>
> So at least it would be useful for smaller deployments to have a
> semi-predictable ID mapping (modulo collisions).
>
> For larger deployments, the mapping data to be replicated could be
> much reduced by limiting it to the colliding IDs.
>
> Cheers,
> Eoghan
>
> > Alternatively, we could just consider the ec2 mapping layer to be
> > global data that must be replicated somehow across the system. I
> > don't think we can really ensure no collisions mapping from uuid ->
> > ec2_id deterministically, and I don't see a clear path forward when
> > we do get a collision.
> >
> > Vish
> >
> > On May 8, 2012, at 12:24 AM, Michael Still wrote:
> >
> > > On 04/05/12 20:31, Eoghan Glynn wrote:
> > >
> > > Sorry for the slow reply, I've been trapped in meetings.
> > >
> > > [snip]
> > >
> > >> So the way things currently stand, the EC2 image ID isn't really
> > >> capable of
> > >> migration.
> > >>
> > >> I was thinking however that we should change the EC2 image
> > >> generation logic,
> > >> so that there is a reproducible glance UUID -> EC2 mapping (with
> > >> a
> > >> small
> > >> chance of collision). This change would allow the same EC2 ID to
> > >> be generated
> > >> in multiple regions for a given glance UUID (modulo collisions).
> > >>
> > >> Would that be helpful in your migration use-case?
> > >
> > > I do think this is a good idea. Or even if the column wasn't
> > > auto-increment, but just picked a random number or something
> > > (because
> > > that would be marginally less likely to clash). Without somehow
> > > making
> > > these ec2 ids more global, replication between regions is going
> > > to
> > > suffer from ec2 api users having to somehow perform a lookup out
> > > of
> > > band.
> > >
> > > Now, my use case is a bit special, because I can enforce that
> > > images are
> > > only ever uploaded to one master region, and then copied to all
> > > others.
> > > I think that's probably not true for other users though.
> > >
> > > Mikal
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~openstack
> > > Post to : openstack [at] lists
> > > Unsubscribe : https://launchpad.net/~openstack
> > > More help : https://help.launchpad.net/ListHelp
> >
> >
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

michael.still at canonical

May 20, 2012, 7:43 PM

Post #12 of 13 (165 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

On 09/05/12 09:01, Gabe Westmaas wrote:
> This actually sounds like it covers most of the first run needs in this
> etherpad:
>
> http://etherpad.openstack.org/FolsomGlanceImageReplication
>
>
> Definitely excited to see the tool, and then I wonder if we can just add a
> few things to it to get metadata and filtering support. Sounds like ec2
> stuff is being covered.

Ok, I _finally_ got around to uploading this change. Sorry for the
delay. Its at https://review.openstack.org/#/c/7615/ .

I've dropped the ec2 id replication for now, as it seems like we haven't
come up with a good plan for that yet. I note that the ec2 api spec says
that the id is just a string, so we could just use the uuid here,
although we don't know what tools and scripts that might break.

The next step I think is some sort of configuration language which lets
you filter which images get replicated. I don't think that's too hard to
do, but I haven't had a chance to give it a try yet.

Cheers,
Mikal

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

jaypipes at gmail

May 21, 2012, 9:23 AM

Post #13 of 13 (166 views)
Permalink

Re: [Glance] Replication implementations [In reply to]

Nice job, Mikal. I've added some review comments along with Eoghan.

Best,
-jay

On 05/20/2012 10:43 PM, Michael Still wrote:
> On 09/05/12 09:01, Gabe Westmaas wrote:
>> This actually sounds like it covers most of the first run needs in this
>> etherpad:
>>
>> http://etherpad.openstack.org/FolsomGlanceImageReplication
>>
>>
>> Definitely excited to see the tool, and then I wonder if we can just add a
>> few things to it to get metadata and filtering support. Sounds like ec2
>> stuff is being covered.
>
> Ok, I _finally_ got around to uploading this change. Sorry for the
> delay. Its at https://review.openstack.org/#/c/7615/ .
>
> I've dropped the ec2 id replication for now, as it seems like we haven't
> come up with a good plan for that yet. I note that the ec2 api spec says
> that the id is just a string, so we could just use the uuid here,
> although we don't know what tools and scripts that might break.
>
> The next step I think is some sort of configuration language which lets
> you filter which images get replicated. I don't think that's too hard to
> do, but I haven't had a chance to give it a try yet.
>
> Cheers,
> Mikal
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads