Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

EC2 compat.

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


harlowja at yahoo-inc

Apr 10, 2012, 4:39 PM

Post #1 of 7 (256 views)
Permalink
EC2 compat.

Hi all,

I've started gathering tools/docs and possibly a mock ec2 server (wip) that can allow openstack to figure out exactly what is broken with there ec2 implementation.

The process of course starts with figuring out what is there currently, what is broken and what needs fixing.

I've started this @ https://github.com/yahoo/Openstack-EC2/wiki

This includes XSD's grabbed from amazon (converted from there wsdls), https://github.com/yahoo/Openstack-EC2/tree/master/data/xsds

I've started to fill in what might be a good template for the run instances call @ https://github.com/yahoo/Openstack-EC2/wiki/RunInstances

Do people think this would be useful? Or any suggestions on how to do it better...

Since EC2 is so key to openstack, the only way to make it better is to have a very detailed level of docs on how compatible it is, hopefully this is a start.

Of course contributions are welcome, since documenting what is in amazon EC2 responses/requests and what is in openstack responses/requests/code is a very laborious task (but it has to be done).

-Josh


eric at cloudscaling

Apr 10, 2012, 5:30 PM

Post #2 of 7 (250 views)
Permalink
Re: EC2 compat. [In reply to]

I agree that it is important to access the limitations of the OpenStack EC2 API implementation.

To that end, make sure to take a look at https://github.com/cloudscaling/aws-compat

--
Eric Windisch


On Tuesday, April 10, 2012 at 7:39 PM, Joshua Harlow wrote:

> EC2 compat. Hi all,
>
> I’ve started gathering tools/docs and possibly a mock ec2 server (wip) that can allow openstack to figure out exactly what is broken with there ec2 implementation.
>
> The process of course starts with figuring out what is there currently, what is broken and what needs fixing.
>
> I’ve started this @ https://github.com/yahoo/Openstack-EC2/wiki
>
> This includes XSD’s grabbed from amazon (converted from there wsdls), https://github.com/yahoo/Openstack-EC2/tree/master/data/xsds
>
> I’ve started to fill in what might be a good template for the run instances call @ https://github.com/yahoo/Openstack-EC2/wiki/RunInstances
>
> Do people think this would be useful? Or any suggestions on how to do it better...
>
> Since EC2 is so key to openstack, the only way to make it better is to have a very detailed level of docs on how compatible it is, hopefully this is a start.
>
> Of course contributions are welcome, since documenting what is in amazon EC2 responses/requests and what is in openstack responses/requests/code is a very laborious task (but it has to be done).
>
> -Josh
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists (mailto:openstack [at] lists)
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


eric at cloudscaling

Apr 10, 2012, 7:02 PM

Post #3 of 7 (251 views)
Permalink
Re: EC2 compat. [In reply to]

Josh, as a follow-up, it would be good to keep an open dialogue on this. When/if you get a chance to review the aws-compat branch, I'd like to get your feedback as well.

PS I meant to write "assess", not "access". I only noticed when I read back my email. I'm too pedantic to not correct myself.

--
Eric Windisch


On Tuesday, April 10, 2012 at 8:30 PM, Eric Windisch wrote:

> I agree that it is important to access the limitations of the OpenStack EC2 API implementation.
>
> To that end, make sure to take a look at https://github.com/cloudscaling/aws-compat
>
> --
> Eric Windisch
>
>
> On Tuesday, April 10, 2012 at 7:39 PM, Joshua Harlow wrote:
>
> > EC2 compat. Hi all,
> >
> > I’ve started gathering tools/docs and possibly a mock ec2 server (wip) that can allow openstack to figure out exactly what is broken with there ec2 implementation.
> >
> > The process of course starts with figuring out what is there currently, what is broken and what needs fixing.
> >
> > I’ve started this @ https://github.com/yahoo/Openstack-EC2/wiki
> >
> > This includes XSD’s grabbed from amazon (converted from there wsdls), https://github.com/yahoo/Openstack-EC2/tree/master/data/xsds
> >
> > I’ve started to fill in what might be a good template for the run instances call @ https://github.com/yahoo/Openstack-EC2/wiki/RunInstances
> >
> > Do people think this would be useful? Or any suggestions on how to do it better...
> >
> > Since EC2 is so key to openstack, the only way to make it better is to have a very detailed level of docs on how compatible it is, hopefully this is a start.
> >
> > Of course contributions are welcome, since documenting what is in amazon EC2 responses/requests and what is in openstack responses/requests/code is a very laborious task (but it has to be done).
> >
> > -Josh
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists (mailto:openstack [at] lists)
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
> >
> >
> >
>
>


harlowja at yahoo-inc

Apr 10, 2012, 7:43 PM

Post #4 of 7 (250 views)
Permalink
Re: EC2 compat. [In reply to]

Very cool, glad to see that is being worked on, it looks pretty similar to what I was thinking of.
I'm all for open dialogues.
In fact.
I was thinking of what is needed to make this work better.

Open questions/thoughts/brainstorm (at least that I was thinking of):


1. How strict do we want to be with the XSD? (there aren't a lot of tolerant xsd validators out there, which sux)
* Should we use something like jaxb for python, that should be more tolerant (unsure as what the best solution here is)
2. How do we continuously measure the compatibility level?
* # of test cases passing, # of xml differences, # of xsd issues
3. Should we use boto as a intermediate layer? (it is very tolerant)
* From what I understand there XML code is basically selecting certain attributes out of the XML using SAX, then adding any unknown attributes dynamically on to a object
4. How do we make it repeatable?
* For a given test X, if there is a problem with test X and its response Y, how do we easily recreate that test X and response Y (so that dev's can fix it)?
* Do we have a "golden set" of responses that when test X is called it should match golden response Z (otherwise there is an issue)
* This is where the mock server maybe useful, in that we can point test X at the mock server; get the expected responses Z,
* Then point the test X at the real openstack server and get responses Y that should match Z (exactly, minus the request id?)
* EC2 seems to also already have some type of mocking, but I haven't used it... (http://bit.ly/HJkdh7)

I like how there is a tests folder that u guys have, that seems like it could be a good location for the "content checking tests" which actually require code/logic to dig into the XML response. It might make sense to use another tool to verify the XSD's (how tolerant we want to be is an open question) and another tool that will show u the xml differences (some of which might be ok, some not). I have used in java xmlunit to do those kind of xml difference comparisons, it provides some nifty ways of ignoring certain differences and such. If say we had 3 levels of tests I think that would make sense (starting say from XSD validation, to difference comparisons to content comparisons), and would make a hell of a EC2 cool validation toolkit.

The other usage of the site I was making was to list all the known error conditions, and any other incompatibilities that I am noticing with EC2 (error conditions, features, parameters...). That seems really needed to allow for anyone to actually use the EC2 apis and handle all the cases which could be thrown at them.

-Josh



On 4/10/12 7:02 PM, "Eric Windisch" <eric [at] cloudscaling> wrote:


Josh, as a follow-up, it would be good to keep an open dialogue on this. When/if you get a chance to review the aws-compat branch, I'd like to get your feedback as well.

PS I meant to write "assess", not "access". I only noticed when I read back my email. I'm too pedantic to not correct myself.


duncan at dreamhost

Apr 11, 2012, 8:06 AM

Post #5 of 7 (251 views)
Permalink
Re: EC2 compat. [In reply to]

Lurking on the thread, but love what I'm seeing :-)

Nice work, guys!

d

On Tue, Apr 10, 2012 at 10:43 PM, Joshua Harlow <harlowja [at] yahoo-inc> wrote:
> Very cool, glad to see that is being worked on, it looks pretty similar to
> what I was thinking of.
> Im all for open dialogues.
> In fact.
> I was thinking of what is needed to make this work better.
>
> Open questions/thoughts/brainstorm (at least that I was thinking of):
>
> How strict do we want to be with the XSD? (there arent a lot of tolerant
> xsd validators out there, which sux)
>
> Should we use something like jaxb for python, that should be more tolerant
> (unsure as what the best solution here is)
>
> How do we continuously measure the compatibility level?
>
> # of test cases passing, # of xml differences, # of xsd issues
>
> Should we use boto as a intermediate layer? (it is very tolerant)
>
> From what I understand there XML code is basically selecting certain
> attributes out of the XML using SAX, then adding any unknown attributes
> dynamically on to a object
>
> How do we make it repeatable?
>
> For a given test X, if there is a problem with test X and its response Y,
> how do we easily recreate that test X and response Y (so that devs can fix
> it)?
> Do we have a golden set of responses that when test X is called it should
> match golden response Z (otherwise there is an issue)
>
> This is where the mock server maybe useful, in that we can point test X at
> the mock server; get the expected responses Z,
> Then point the test X at the real openstack server and get responses Y that
> should match Z (exactly, minus the request id?)
> EC2 seems to also already have some type of mocking, but I havent used
> it... (http://bit.ly/HJkdh7)
>
>
> I like how there is a tests folder that u guys have, that seems like it
> could be a good location for the content checking tests which actually
> require code/logic to dig into the XML response. It might make sense to use
> another tool to verify the XSDs (how tolerant we want to be is an open
> question) and another tool that will show u the xml differences (some of
> which might be ok, some not). I have used in java xmlunit to do those kind
> of xml difference comparisons, it provides some nifty ways of ignoring
> certain differences and such. If say we had 3 levels of tests I think that
> would make sense (starting say from XSD validation, to difference
> comparisons to content comparisons), and would make a hell of a EC2 cool
> validation toolkit.
>
> The other usage of the site I was making was to list all the known error
> conditions, and any other incompatibilities that I am noticing with EC2
> (error conditions, features, parameters...). That seems really needed to
> allow for anyone to actually use the EC2 apis and handle all the cases which
> could be thrown at them.
>
> -Josh
>
>
>
>
> On 4/10/12 7:02 PM, "Eric Windisch" <eric [at] cloudscaling> wrote:
>
>
> Josh, as a follow-up, it would be good to keep an open dialogue on this.
> When/if you get a chance to review the aws-compat branch, I'd like to get
> your feedback as well.
>
> PS I meant to write "assess", not "access". I only noticed when I read back
> my email. I'm too pedantic to not correct myself.
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


harlowja at yahoo-inc

Apr 11, 2012, 10:42 AM

Post #6 of 7 (251 views)
Permalink
Re: EC2 compat. [In reply to]

Sweet,

Ideas and thoughts are welcome.

There is a lot of cleanup/testing/validation that needs to happen, but someones got to do it :-P

On 4/11/12 8:06 AM, "Duncan McGreggor" <duncan [at] dreamhost> wrote:

Lurking on the thread, but love what I'm seeing :-)

Nice work, guys!

d

On Tue, Apr 10, 2012 at 10:43 PM, Joshua Harlow <harlowja [at] yahoo-inc> wrote:
> Very cool, glad to see that is being worked on, it looks pretty similar to
> what I was thinking of.
> I'm all for open dialogues.
> In fact.
> I was thinking of what is needed to make this work better.
>
> Open questions/thoughts/brainstorm (at least that I was thinking of):
>
> How strict do we want to be with the XSD? (there aren't a lot of tolerant
> xsd validators out there, which sux)
>
> Should we use something like jaxb for python, that should be more tolerant
> (unsure as what the best solution here is)
>
> How do we continuously measure the compatibility level?
>
> # of test cases passing, # of xml differences, # of xsd issues
>
> Should we use boto as a intermediate layer? (it is very tolerant)
>
> From what I understand there XML code is basically selecting certain
> attributes out of the XML using SAX, then adding any unknown attributes
> dynamically on to a object
>
> How do we make it repeatable?
>
> For a given test X, if there is a problem with test X and its response Y,
> how do we easily recreate that test X and response Y (so that dev's can fix
> it)?
> Do we have a "golden set" of responses that when test X is called it should
> match golden response Z (otherwise there is an issue)
>
> This is where the mock server maybe useful, in that we can point test X at
> the mock server; get the expected responses Z,
> Then point the test X at the real openstack server and get responses Y that
> should match Z (exactly, minus the request id?)
> EC2 seems to also already have some type of mocking, but I haven't used
> it... (http://bit.ly/HJkdh7)
>
>
> I like how there is a tests folder that u guys have, that seems like it
> could be a good location for the "content checking tests" which actually
> require code/logic to dig into the XML response. It might make sense to use
> another tool to verify the XSD's (how tolerant we want to be is an open
> question) and another tool that will show u the xml differences (some of
> which might be ok, some not). I have used in java xmlunit to do those kind
> of xml difference comparisons, it provides some nifty ways of ignoring
> certain differences and such. If say we had 3 levels of tests I think that
> would make sense (starting say from XSD validation, to difference
> comparisons to content comparisons), and would make a hell of a EC2 cool
> validation toolkit.
>
> The other usage of the site I was making was to list all the known error
> conditions, and any other incompatibilities that I am noticing with EC2
> (error conditions, features, parameters...). That seems really needed to
> allow for anyone to actually use the EC2 apis and handle all the cases which
> could be thrown at them.
>
> -Josh
>
>
>
>
> On 4/10/12 7:02 PM, "Eric Windisch" <eric [at] cloudscaling> wrote:
>
>
> Josh, as a follow-up, it would be good to keep an open dialogue on this.
> When/if you get a chance to review the aws-compat branch, I'd like to get
> your feedback as well.
>
> PS I meant to write "assess", not "access". I only noticed when I read back
> my email. I'm too pedantic to not correct myself.
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>


John.Garbutt at citrix

Apr 13, 2012, 12:37 PM

Post #7 of 7 (256 views)
Permalink
Re: EC2 compat. [In reply to]

I got asked by E2C support by RealStatus who have a cool 3D modeling tool that works against EC2 and would love better APIs to make that work well with OpenStack:
http://www.youtube.com/watch?feature=player_embedded&v=SOkWRxDwTNI

Hopefully I should be able to lend a hand with some of this EC2 work. See you all at the summit session on EC2!

Cheers,
John

From: openstack-bounces+john.garbutt=eu.citrix.com [at] lists [mailto:openstack-bounces+john.garbutt=eu.citrix.com [at] lists] On Behalf Of Joshua Harlow
Sent: 11 April 2012 10:42
To: Duncan McGreggor
Cc: openstack
Subject: Re: [Openstack] EC2 compat.

Sweet,

Ideas and thoughts are welcome.

There is a lot of cleanup/testing/validation that needs to happen, but someones got to do it :-P

On 4/11/12 8:06 AM, "Duncan McGreggor" <duncan [at] dreamhost> wrote:
Lurking on the thread, but love what I'm seeing :-)

Nice work, guys!

d

On Tue, Apr 10, 2012 at 10:43 PM, Joshua Harlow <harlowja [at] yahoo-inc> wrote:
> Very cool, glad to see that is being worked on, it looks pretty similar to
> what I was thinking of.
> I'm all for open dialogues.
> In fact.
> I was thinking of what is needed to make this work better.
>
> Open questions/thoughts/brainstorm (at least that I was thinking of):
>
> How strict do we want to be with the XSD? (there aren't a lot of tolerant
> xsd validators out there, which sux)
>
> Should we use something like jaxb for python, that should be more tolerant
> (unsure as what the best solution here is)
>
> How do we continuously measure the compatibility level?
>
> # of test cases passing, # of xml differences, # of xsd issues
>
> Should we use boto as a intermediate layer? (it is very tolerant)
>
> From what I understand there XML code is basically selecting certain
> attributes out of the XML using SAX, then adding any unknown attributes
> dynamically on to a object
>
> How do we make it repeatable?
>
> For a given test X, if there is a problem with test X and its response Y,
> how do we easily recreate that test X and response Y (so that dev's can fix
> it)?
> Do we have a "golden set" of responses that when test X is called it should
> match golden response Z (otherwise there is an issue)
>
> This is where the mock server maybe useful, in that we can point test X at
> the mock server; get the expected responses Z,
> Then point the test X at the real openstack server and get responses Y that
> should match Z (exactly, minus the request id?)
> EC2 seems to also already have some type of mocking, but I haven't used
> it... (http://bit.ly/HJkdh7)
>
>
> I like how there is a tests folder that u guys have, that seems like it
> could be a good location for the "content checking tests" which actually
> require code/logic to dig into the XML response. It might make sense to use
> another tool to verify the XSD's (how tolerant we want to be is an open
> question) and another tool that will show u the xml differences (some of
> which might be ok, some not). I have used in java xmlunit to do those kind
> of xml difference comparisons, it provides some nifty ways of ignoring
> certain differences and such. If say we had 3 levels of tests I think that
> would make sense (starting say from XSD validation, to difference
> comparisons to content comparisons), and would make a hell of a EC2 cool
> validation toolkit.
>
> The other usage of the site I was making was to list all the known error
> conditions, and any other incompatibilities that I am noticing with EC2
> (error conditions, features, parameters...). That seems really needed to
> allow for anyone to actually use the EC2 apis and handle all the cases which
> could be thrown at them.
>
> -Josh
>
>
>
>
> On 4/10/12 7:02 PM, "Eric Windisch" <eric [at] cloudscaling> wrote:
>
>
> Josh, as a follow-up, it would be good to keep an open dialogue on this.
> When/if you get a chance to review the aws-compat branch, I'd like to get
> your feedback as well.
>
> PS I meant to write "assess", not "access". I only noticed when I read back
> my email. I'm too pedantic to not correct myself.
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.