Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

Just JSON, and extensibility

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


mnot at mnot

Apr 13, 2012, 6:47 AM

Post #1 of 15 (593 views)
Permalink
Just JSON, and extensibility

[. Full disclosure -- I'm using my personal address with Launchpad, etc., but I work for Rackspace. ]

On 12/04/2012, at 7:28 PM, Jorge Williams wrote:

> Generally, I agree with a lot of what you're saying, but I want to point out a couple of things:
>
> 1. Static language folks gravitate to XML, not simply because they're invested in it, but because it solves a real problem:
> […]
> and I should see those errors at compile time or while I'm authoring my code if a have a good editor or IDE....I shouldn't have to wait until my program is run to catch those errors.
> […]
> but at that point, there's very little benefit to having a static language, because I don't have the assurances and guarantees that the language provides. So I don't see a lot of Java folks dealing with JSON in that manner. Most devs will need to build a class before hand. So, you decrease barriers for static language clients because there's a set of tools that can extract the relevant info from XML schema languages and generate a set of class files at compile time. There's nothing saying you can't do something similar with JSON, but those sort of tools aren't there yet.

Great -- I think it's good to level set here. We're talking about supporting an entire serialisation, all of its supporting code, the QA cycle, schema authoring, etc., and the #1 benefit is making the programming experience for a subset of our audience more comfortable.


> 2. Then there's the issue of extensibility....especially distributed extensibility. XML has that notion built in

Heh; you mean bolted on with Namespaces in XML, and then completely messed up by XML Schema 1.0, then only partially fixed by 1.1. But yes.

> JSON has no concept of it...and we are building extensible APIs. There are no standard way in JSON to introduce a new property without guaranteeing that there won't be clash. You've mention the need for namespaces in JSON precisely to deal with this sort of issue (http://www.mnot.net/blog/2011/10/12/thinking_about_namespaces_in_json).

First of all, as we've discussed many times, I don't think "extensibility" == "good" in all cases; if we allow/encourage too much extensibility, the platform we're building will fragment, and people won't see its full value. Extensions should be allowed where the make sense, not everywhere, and extensions should be encouraged to eventually converge.

> In the absence of a standard method, we're been using prefixes, which has worked out well, but most JSON tools don't know how to deal with them

What does that *mean*? The prefix is an opaque part of the name -- what should a tool do with it?

> and they seem alien to folk that are used to using JSON day to day.

Perhaps, but doing something like RAX_FOO isn't that onerous. Or using a registry. Or just leaving it to the community to coordinate and document; try opening a shell, typing "set" and pondering how *that* name space is managed...

> This is a big deal because dynamic language folks are more likely to deal with the JSON directly...Static language folks are generally not dealing with XML in the same way. In XML, the notion of extensibility is build into parsers and data binding tools directly.

Not really; it's a syntactic artefact of namespaces. I'd say that the tools manage it really, really badly, given how hard it is to author extensible schemas (and don't get me started on XML Schema).

> Most folks don't have to worry too much about it. In fact extensible protocols like XMPP and Atom Pub generally benefit from the extensibility that's already built in: http://metajack.im/2010/02/01/json-versus-xml-not-as-simple-as-you-think/

You bring up some interesting examples; XMPP and Atom are both products of the fashion for XML in the IETF in the early part of the 00's… we've learned a LOT since then, and there's a clear move towards more no-nonsense JSON in the APPS area more recently. The extensibility of both XMPP and Atom turned out to not really need that much coordination.

Is extensibility important? Sure, but we've learned a LOT about it since XML and PEP; it shouldn't be slathered over your protocols and formats like peanut butter, but instead only exposed where it's genuinely needed. Is that a judgement call? Absolutely.

> Given that, if we're going to go the route of just picking one format, I think the fact that our API is extensible means that we might want to ask ourselves whether XML isn't a better fit :-)

I'm guessing you know I disagree with that by now.

> Having said all of that, I realize that our devs are working in a dynamic language, and don't see a lot of value in XML. It's important to take that into consideration, but we should also be asking whether there's value to our current clients and potential clients. Like it or not, there's a lot of folks out there using static languages.

Sure, but it's less clear they're spinning up VMs with them, or writing apps that talk to Swift. I'd really like to get some logging of User-Agents into Nova et al to help get these metrics (e.g., lib cloud produces a UA); see <https://bugs.launchpad.net/nova/+bug/975589>.

> You're right in stating that if we had really good language bindings for Java and .Net, then issue #1 would essentially go away -- especially if we had language binding that was flexible enough to remove the need to go down to the HTTP level. Also, if the language binding itself was extensible we could also deal with issue 2. As things stand today though, I don't thing that we are even remotely there yet.


Sure. The question I'd ask, then, is it worth making our APIs seriously more complex, hard to develop, understand, maintain, test, document, etc. in the meantime, just to allow static language users to have their IDEs help them. Something that the dynamic folks have gotten pretty used to living without.

Cheers,

--
Mark Nottingham
http://www.mnot.net/





_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


jorge.williams at rackspace

Apr 13, 2012, 9:29 AM

Post #2 of 15 (583 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

On Apr 13, 2012, at 8:47 AM, Mark Nottingham wrote:

> [. Full disclosure -- I'm using my personal address with Launchpad, etc., but I work for Rackspace. ]
>
> On 12/04/2012, at 7:28 PM, Jorge Williams wrote:
>
>> Generally, I agree with a lot of what you're saying, but I want to point out a couple of things:
>>
>> 1. Static language folks gravitate to XML, not simply because they're invested in it, but because it solves a real problem:
>> […]
>> and I should see those errors at compile time or while I'm authoring my code if a have a good editor or IDE....I shouldn't have to wait until my program is run to catch those errors.
>> […]
>> but at that point, there's very little benefit to having a static language, because I don't have the assurances and guarantees that the language provides. So I don't see a lot of Java folks dealing with JSON in that manner. Most devs will need to build a class before hand. So, you decrease barriers for static language clients because there's a set of tools that can extract the relevant info from XML schema languages and generate a set of class files at compile time. There's nothing saying you can't do something similar with JSON, but those sort of tools aren't there yet.
>
> Great -- I think it's good to level set here. We're talking about supporting an entire serialisation, all of its supporting code, the QA cycle, schema authoring, etc., and the #1 benefit is making the programming experience for a subset of our audience more comfortable.
>

There are a lot more clients than servers. You have to weigh the cost of lowering barriers for those clients at the server side vs the cost of getting those client to successfully integrate with the system. This is typically the a argument I make against SOAP and towards REST. In static language circles creating a SOAP service is really, really easy...but using soap typically introduces high barriers to dynamic language folks. Making the move from SOAP to REST definitely introduces some complexity on the dev side of the service in dev cycles, but you have to compare this cost against the cost that's saved on the client side and multiply that by the number of clients that benefit.

Having said that, I understand the cost of supporting a different media type is not insignificant, but it's also not rocket science. I reckon that the sate of XML support today may have to do more with the fact that our dev community doesn't see much value in XML, than in does with what it will actually takes to get it right....and like I said before, I can understand that perspective.

>
>> 2. Then there's the issue of extensibility....especially distributed extensibility. XML has that notion built in
>
> Heh; you mean bolted on with Namespaces in XML, and then completely messed up by XML Schema 1.0, then only partially fixed by 1.1. But yes.
>
>> JSON has no concept of it...and we are building extensible APIs. There are no standard way in JSON to introduce a new property without guaranteeing that there won't be clash. You've mention the need for namespaces in JSON precisely to deal with this sort of issue (http://www.mnot.net/blog/2011/10/12/thinking_about_namespaces_in_json).
>
> First of all, as we've discussed many times, I don't think "extensibility" == "good" in all cases; if we allow/encourage too much extensibility, the platform we're building will fragment, and people won't see its full value. Extensions should be allowed where the make sense, not everywhere, and extensions should be encouraged to eventually converge.

I totally agree, encouraging convergence is important.

>
>> In the absence of a standard method, we're been using prefixes, which has worked out well, but most JSON tools don't know how to deal with them
>
> What does that *mean*? The prefix is an opaque part of the name -- what should a tool do with it?
>

It messes up some of the syntactic sugar that folks are used to using:

server.name vs server["foo:name"]


>> and they seem alien to folk that are used to using JSON day to day.
>
> Perhaps, but doing something like RAX_FOO isn't that onerous. Or using a registry. Or just leaving it to the community to coordinate and document; try opening a shell, typing "set" and pondering how *that* name space is managed...
>
>> This is a big deal because dynamic language folks are more likely to deal with the JSON directly...Static language folks are generally not dealing with XML in the same way. In XML, the notion of extensibility is build into parsers and data binding tools directly.
>
> Not really; it's a syntactic artefact of namespaces. I'd say that the tools manage it really, really badly, given how hard it is to author extensible schemas (and don't get me started on XML Schema).
>

That hasn't entirely been my experience. For the most part you can set things up so you don't have to think much about namespaces.

>> Most folks don't have to worry too much about it. In fact extensible protocols like XMPP and Atom Pub generally benefit from the extensibility that's already built in: http://metajack.im/2010/02/01/json-versus-xml-not-as-simple-as-you-think/
>
> You bring up some interesting examples; XMPP and Atom are both products of the fashion for XML in the IETF in the early part of the 00's… we've learned a LOT since then, and there's a clear move towards more no-nonsense JSON in the APPS area more recently. The extensibility of both XMPP and Atom turned out to not really need that much coordination.

I reckon that not a lot of coordination was needed precisely because XML has built in extensibility. In it's absence, you have to adopt the namespace model that XML uses in JSON...that's generally perceived to be very unJSONic. Or you have to build a system of coordination in. I'm all for it though BTW.

>
> Is extensibility important? Sure, but we've learned a LOT about it since XML and PEP; it shouldn't be slathered over your protocols and formats like peanut butter, but instead only exposed where it's genuinely needed. Is that a judgement call? Absolutely.


Sure.

>
>> Given that, if we're going to go the route of just picking one format, I think the fact that our API is extensible means that we might want to ask ourselves whether XML isn't a better fit :-)
>
> I'm guessing you know I disagree with that by now.

:-)

>
>> Having said all of that, I realize that our devs are working in a dynamic language, and don't see a lot of value in XML. It's important to take that into consideration, but we should also be asking whether there's value to our current clients and potential clients. Like it or not, there's a lot of folks out there using static languages.
>
> Sure, but it's less clear they're spinning up VMs with them, or writing apps that talk to Swift. I'd really like to get some logging of User-Agents into Nova et al to help get these metrics (e.g., lib cloud produces a UA); see <https://bugs.launchpad.net/nova/+bug/975589>.
>
>> You're right in stating that if we had really good language bindings for Java and .Net, then issue #1 would essentially go away -- especially if we had language binding that was flexible enough to remove the need to go down to the HTTP level. Also, if the language binding itself was extensible we could also deal with issue 2. As things stand today though, I don't thing that we are even remotely there yet.
>
>
> Sure. The question I'd ask, then, is it worth making our APIs seriously more complex, hard to develop, understand, maintain, test, document, etc. in the meantime, just to allow static language users to have their IDEs help them. Something that the dynamic folks have gotten pretty used to living without.
>

Asking static language folks to just get used to not having compile time checks is silly. That said, I don't want to turn this into a static vs dynamic language debate. I advocate just as hard for good JSON support when I'm dealing with static language folks....go talk to the atlas guys.

Again, I'm generally agreeing with what your article is saying. If we had great/nontrivial static language binding support...that's extensible and flexible enough to remove the need to go down to the HTTP level, then there wouldn't be a need for this debate.

-jOrGe W.


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


mnot at mnot

Apr 13, 2012, 9:56 AM

Post #3 of 15 (578 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

On 13/04/2012, at 11:29 AM, Jorge Williams wrote:
>
> There are a lot more clients than servers. You have to weigh the cost of lowering barriers for those clients at the server side vs the cost of getting those client to successfully integrate with the system. This is typically the a argument I make against SOAP and towards REST. In static language circles creating a SOAP service is really, really easy…

As long as the server and client are using the same tools, yes. SOAP doesn't provide any interoperability, it's just an envelope for a proprietary messaging protocol. Background: <http://www.mnot.net/blog/2006/05/10/vendors>

But I digress...


> but using soap typically introduces high barriers to dynamic language folks. Making the move from SOAP to REST definitely introduces some complexity on the dev side of the service in dev cycles, but you have to compare this cost against the cost that's saved on the client side and multiply that by the number of clients that benefit.
>
> Having said that, I understand the cost of supporting a different media type is not insignificant, but it's also not rocket science. I reckon that the sate of XML support today may have to do more with the fact that our dev community doesn't see much value in XML, than in does with what it will actually takes to get it right....and like I said before, I can understand that perspective.

OK, we're going to have to disagree here. It's not that just "our dev community" isn't seeing much value in XML, it's a good portion of the world. The question here is how far we bend over to cater to those who are using these tools, and how we go about it. I'm suggesting that doing it in the protocol, rather than with tools, is buying us more long-term trouble than it's solving for them.

In particular, if people are actually using these tools to do data binding, it's going to lead them to place dependencies upon the structure of our interfaces, and unless the scheme is constructed *exactly* right, we'll get lots of bug reports in the future when these tools base their "contracts" on their interpretation of a Schema that got published on the Web site at some point in the past.

However, I question whether it's even being used in this fashion. Looking at our docs for Nova <http://docs.openstack.org/api/openstack-compute/2/content/>, the schema isn't locatable, and the XML Namespace <http://docs.openstack.org/compute/api/v1.1> is a 404. How are they finding the schema, and how is it getting into these tools?


> It messes up some of the syntactic sugar that folks are used to using:
>
> server.name vs server["foo:name"]

If you use colons, sure, but let's not try to reinvent Namespaces in XML in JSON.


>> Sure. The question I'd ask, then, is it worth making our APIs seriously more complex, hard to develop, understand, maintain, test, document, etc. in the meantime, just to allow static language users to have their IDEs help them. Something that the dynamic folks have gotten pretty used to living without.
>
> Asking static language folks to just get used to not having compile time checks is silly. That said, I don't want to turn this into a static vs dynamic language debate. I advocate just as hard for good JSON support when I'm dealing with static language folks....go talk to the atlas guys.
>
> Again, I'm generally agreeing with what your article is saying. If we had great/nontrivial static language binding support...that's extensible and flexible enough to remove the need to go down to the HTTP level, then there wouldn't be a need for this debate.


Let's get there, then!

We won't have them tomorrow, so what I think we need to think very carefully about is whether offering this audience XML at this point in time is doing more harm than good, both in terms of opportunity cost and in terms of causing problems down the road.

I'm not talking about *removing* XML from the current APIs that ship it, BTW; at most I'd like to see us:

1. Not require XML for new APIs, and
2. Talk about deprecating XML support in current ones (i.e., still support it, but de-emphasise it in documentation, don't expend new effort on it), and
3. Put some considered caveats around our current use of WADL and XML Schema.

Cheers,


--
Mark Nottingham
http://www.mnot.net/





_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


Caitlin.Bestler at nexenta

Apr 13, 2012, 11:53 AM

Post #4 of 15 (577 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

The argument that XML has better extensibility than JSON isn't very convincing to my ears.

I'm an old war horse, and recall extending message formats in ANSI C so as to maintain backwards
compatibility with existing clients (by versioning the struct name itself and always keeping the same
fields up front).

If this problem could be solved in ANSI C, the solutions available to Python with JSON can only be better.


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


anne at openstack

Apr 13, 2012, 12:04 PM

Post #5 of 15 (575 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

Hi Mark,


> However, I question whether it's even being used in this fashion. Looking
> at our docs for Nova <
> http://docs.openstack.org/api/openstack-compute/2/content/>, the schema
> isn't locatable, and the XML Namespace <
> http://docs.openstack.org/compute/api/v1.1> is a 404. How are they
> finding the schema, and how is it getting into these tools?
>
>
>
The schema is available at
http://docs.openstack.org/api/openstack-compute/2/xsd/

I am told that we don't need to redirect to the XML Namespace as it is just
a string not an actual URL. If you disagree, please log a doc bug at
http://bugs.launchpad.net/openstack-manuals/ and tag it with compute-api
with specifics about how you think it should work on the docs site.


I'm not talking about *removing* XML from the current APIs that ship it,
> BTW; at most I'd like to see us:
>
> 1. Not require XML for new APIs, and
> 2. Talk about deprecating XML support in current ones (i.e., still support
> it, but de-emphasise it in documentation, don't expend new effort on it),
> and
> 3. Put some considered caveats around our current use of WADL and XML
> Schema.
>
>
One additional comment about WADL - basically one of those caveats. We had
little community traction documenting Compute API extensions with various
templates. A working solution now is to add extensions to
api.openstack.orgusing WADL and JSON samples. It's one of those
examples of elbow grease and
a small group adding great value. Next week I'll be doing a "Show-N-Tell"
of the api.openstack.org site first thing Monday morning as well as on a
panel about TryStack Thursday afternoon and would love feedback.

Thanks,
Anne




> Cheers,
>
>
> --
> Mark Nottingham
> http://www.mnot.net/
>
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>


mnot at mnot

Apr 13, 2012, 12:18 PM

Post #6 of 15 (578 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

On 13/04/2012, at 2:04 PM, Anne Gentle wrote:

> Hi Mark,
>
> However, I question whether it's even being used in this fashion. Looking at our docs for Nova <http://docs.openstack.org/api/openstack-compute/2/content/>, the schema isn't locatable, and the XML Namespace <http://docs.openstack.org/compute/api/v1.1> is a 404. How are they finding the schema, and how is it getting into these tools?
>
>
> The schema is available at
> http://docs.openstack.org/api/openstack-compute/2/xsd/
>
> I am told that we don't need to redirect to the XML Namespace as it is just a string not an actual URL. If you disagree, please log a doc bug at http://bugs.launchpad.net/openstack-manuals/ and tag it with compute-api with specifics about how you think it should work on the docs site.

OK.

> I'm not talking about *removing* XML from the current APIs that ship it, BTW; at most I'd like to see us:
>
> 1. Not require XML for new APIs, and
> 2. Talk about deprecating XML support in current ones (i.e., still support it, but de-emphasise it in documentation, don't expend new effort on it), and
> 3. Put some considered caveats around our current use of WADL and XML Schema.
>
>
> One additional comment about WADL - basically one of those caveats. We had little community traction documenting Compute API extensions with various templates. A working solution now is to add extensions to api.openstack.org using WADL and JSON samples. It's one of those examples of elbow grease and a small group adding great value. Next week I'll be doing a "Show-N-Tell" of the api.openstack.org site first thing Monday morning as well as on a panel about TryStack Thursday afternoon and would love feedback.

Yes. I think the api.openstack.org is great and definitely moving in the right direction; should be an interesting discussion.

Cheers,

--
Mark Nottingham
http://www.mnot.net/





_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


justin at fathomdb

Apr 13, 2012, 1:20 PM

Post #7 of 15 (576 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

I don't think that works for non-linear extensibility...

I would be very happy if we could agree out how we're going to deal with
extensibility in JSON. It is easy to support XML & JSON & any future
formats, and have them all be nice if there's willingness to do so, but
there's not, so let's drop it and focus on "stuff that matters"!

I believe we have a solution now for JSON extensibility (thanks Jorge),
it's not as "complete" as XML (read into that what you will) but it seems
to work fine.

My understanding is that the solution we have now is that any extension
goes into its own namespace; we assign a prefix to the namespace and have a
way to map that prefix to the full namespace. (Similar to XML schemas).
Currently prefixes are hard-coded, but we may not be able to keep doing
this forever (XML has pre-document prefixes to avoid the need for a central
authority).

I see 3 questions:
1) Is my summary correct?
2) Are there any problems with the solution?
3) Are there any other problems we're not addressing?


As one of the two authors of the Java binding, I can tell you how I plan on
dealing with extensions:


- Map the JSON/XML/HPSTR to a strongly-typed model (so the
representation is irrelevant to the consumer of the Java library).
- Each (known) extension has its own strongly-typed model object.
- These are stored in a registry.
- When we come across an extension, we look it up in the registry and
either bind or ignore it.
- Every model object has an "Extensions" collection, which can be
queried by type, to see if that extension data was present.


(Note: this has mostly been tested with the XML)

The nice thing about this is that a consumer of the library can write a
binding for an extension, and register it with the client library, and "it
just works". So, even if you have a private extension, which you tell
nobody about and run only on your private cloud, you can use it with the
stock Java library.

Now, how you would do something that awesome in Python, I don't know ;-)

Justin



On Fri, Apr 13, 2012 at 11:53 AM, Caitlin Bestler <
Caitlin.Bestler [at] nexenta> wrote:

> The argument that XML has better extensibility than JSON isn't very
> convincing to my ears.
>
> I'm an old war horse, and recall extending message formats in ANSI C so as
> to maintain backwards
> compatibility with existing clients (by versioning the struct name itself
> and always keeping the same
> fields up front).
>
> If this problem could be solved in ANSI C, the solutions available to
> Python with JSON can only be better.
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>


Caitlin.Bestler at nexenta

Apr 13, 2012, 2:24 PM

Post #8 of 15 (578 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

Exactly what do you see as the required "non-linear extensibility"?

These are ultimately requests to a server. Each new extension is coded in that server.
There is no value in a client making up its own extensions that are not understood by the server.

What is relevant is a server continuing to support clients that have not yet been updated to
understand a new format.

As I stated in my first post, that problem was solved in ANSI C. Python/JSON is trivial.


dug at us

Apr 13, 2012, 3:05 PM

Post #9 of 15 (582 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

Mark Nottingham wrote on 04/13/2012 12:56:46 PM:

> In particular, if people are actually using these tools to do data
> binding, it's going to lead them to place dependencies upon the
> structure of our interfaces, and unless the scheme is constructed
> *exactly* right, we'll get lots of bug reports in the future when
> these tools base their "contracts" on their interpretation of a
> Schema that got published on the Web site at some point in the past.

Be a bit careful here. If you're suggesting that JSON is better because
it doesn't force (or even give you the option) of being precise in what
your interfaces looks like, implying that you won't get bug reports when
the JSON changes, I think this is probably wrong. IMO, the stuff around
XML like XSDs (and I include in this topic things like IDLs) just makes
it easier for people to write tooling if they want. But in the end,
changing an interface is just as brittle regardless of whether its in
XML or JSON - both have extensibility points and both will have clients
that will break with the 'expected' data goes missing.

IMO, both JSON and XML have their places. And I do agree with Mark
on one key thing... don't add all of the stuff to JSON that people
claimed was too heavy in XML. And, to me, this includes namespaces.
In the end, once you start down that road you'll just end up with
a curl-braced version of XML and won't we look silly at that point. :-)

Earlier you said:
> It's not that just "our dev community" isn't seeing much value in
> XML, it's a good portion of the world.

Again, this needs to be weighed very carefully. With emerging technologies
its easy to assume that the entire world agrees with you while things
are still too new to be part of the mainstream. This xml vs json fight has
been going on for a while, and I've heard a number of times from a number
of different companies that they will not touch JSON
simply because its too new, too unstructured, doesn't have the tooling
and validation capabilities of XML, blah blah blah. Now, ideally these
same folks should be speaking up in the community to express their concerns
and desires, but for whatever reason they may not choose to - or can't.

Now, I'm not suggesting that we MUST keep XML just to satisfy
people who choose not to participate in OS, rather I want the decision to
be
made with all of the implications considered. And to me, knowing that
Openstack
might not be as easy a sell to some folks because their shop is so heavily
invested in XML and not JSON (when, IMO, support for both isn't that hard -
its just a matter of making it part of the normal code development/approval
process) seems like it might be a mistake - at least given the current
state of the world. After all, if XML were really as dead as some people
imply I doubt we would even be having this conversation.

-Doug


justin at fathomdb

Apr 13, 2012, 4:40 PM

Post #10 of 15 (579 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

It's easy when each new version is defined by a central body.

The problem we face is that we want to allow HP, Rackspace, Nexenta etc to
define their own extensions, without serializing through a central body.
Some extensions may only apply to private clouds and never be shared
publicly.

This is a bit like ZFS feature flags, to use an example that should be near
and dear to your heart!



On Fri, Apr 13, 2012 at 2:24 PM, Caitlin Bestler <
Caitlin.Bestler [at] nexenta> wrote:

> Exactly what do you see as the required “non-linear extensibility”?****
>
> ** **
>
> These are ultimately requests to a server. Each new extension is coded in
> that server.****
>
> There is no value in a client making up its own extensions that are not
> understood by the server.****
>
> ** **
>
> What is relevant is a server continuing to support clients that have not
> yet been updated to
> understand a new format.****
>
> ** **
>
> As I stated in my first post, that problem was solved in ANSI C.
> Python/JSON is trivial.****
>
> ** **
>


jorge.williams at rackspace

Apr 13, 2012, 9:23 PM

Post #11 of 15 (576 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

On Apr 13, 2012, at 3:20 PM, Justin Santa Barbara wrote:

My understanding is that the solution we have now is that any extension goes into its own namespace; we assign a prefix to the namespace and have a way to map that prefix to the full namespace. (Similar to XML schemas). Currently prefixes are hard-coded, but we may not be able to keep doing this forever (XML has pre-document prefixes to avoid the need for a central authority).

I see 3 questions:
1) Is my summary correct?

It's pretty close. I did propose that we maintain a registry of prefixes, but that's never taken off.

I did a write up a while back on extensions, you can find it here: https://github.com/RackerWilliams/OpenStack-Extensions/blob/master/apix-intro.pdf

Take the document that's in GitHub with a grain of salt, it doesn't entirely reflect the reality of things as they stand now and I feel, after working with extensions for a while, that we need to make some slight modifications.
It wasn't clear in the beginning where extension would be most useful so we added extensibility to everything. As Mark mentioned, it's clear now that we need to scale down the points of extensibility. In some cases we may introduce barriers to devs if we make absolutely everything extensible. In other cases, defining our own extensibility doesn't make sense. For example, there's no need define a way of extending HTTP headers -- first because, no one is writing those kinds of extensions and also because there already exists a method of extending headers in HTTP so there's no need us to reinvent the wheel. Stuff like that.


2) Are there any problems with the solution?

Yes a couple. Especially when you consider what happens when an extension gets promoted to a full feature in a new version of the API. I'm now leaning towards keeping prefixes forever...and providing a way for folks to write extension without a prefix, provided that they register the extension with the PTL of the project -- lets face it this is the sort of stuff that's happening now anyway. Mind you these are all just thoughts at the moment we should have a larger discussion.

3) Are there any other problems we're not addressing?

Probably :-)

We're having a panel on extensions at the summit. We should discuss in detail then.

As one of the two authors of the Java binding, I can tell you how I plan on dealing with extensions:


* Map the JSON/XML/HPSTR to a strongly-typed model (so the representation is irrelevant to the consumer of the Java library).
* Each (known) extension has its own strongly-typed model object.
* These are stored in a registry.
* When we come across an extension, we look it up in the registry and either bind or ignore it.
* Every model object has an "Extensions" collection, which can be queried by type, to see if that extension data was present.

(Note: this has mostly been tested with the XML)


When you say registry, do you mean maven?


The nice thing about this is that a consumer of the library can write a binding for an extension, and register it with the client library, and "it just works". So, even if you have a private extension, which you tell nobody about and run only on your private cloud, you can use it with the stock Java library.

That sounds like a really great approach! I'd love to check out the nitty-gritty details...Are you documenting this somewhere?


Now, how you would do something that awesome in Python, I don't know ;-)


I'm sure there are Pythonistas out there working to figure it out.

-jOrGe W.


doug.hellmann at dreamhost

Apr 14, 2012, 5:16 AM

Post #12 of 15 (577 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

On Fri, Apr 13, 2012 at 4:20 PM, Justin Santa Barbara
<justin [at] fathomdb>wrote:

> I don't think that works for non-linear extensibility...
>
> I would be very happy if we could agree out how we're going to deal with
> extensibility in JSON. It is easy to support XML & JSON & any future
> formats, and have them all be nice if there's willingness to do so, but
> there's not, so let's drop it and focus on "stuff that matters"!
>
> I believe we have a solution now for JSON extensibility (thanks Jorge),
> it's not as "complete" as XML (read into that what you will) but it seems
> to work fine.
>
> My understanding is that the solution we have now is that any extension
> goes into its own namespace; we assign a prefix to the namespace and have a
> way to map that prefix to the full namespace. (Similar to XML schemas).
> Currently prefixes are hard-coded, but we may not be able to keep doing
> this forever (XML has pre-document prefixes to avoid the need for a central
> authority).
>

> I see 3 questions:
> 1) Is my summary correct?
> 2) Are there any problems with the solution?
> 3) Are there any other problems we're not addressing?
>
>
> As one of the two authors of the Java binding, I can tell you how I plan
> on dealing with extensions:
>
>
> - Map the JSON/XML/HPSTR to a strongly-typed model (so the
> representation is irrelevant to the consumer of the Java library).
> - Each (known) extension has its own strongly-typed model object.
>
>
Does that mean that an extension cannot add properties to an existing
object (such as adding a new attribute an Image), or just that all of those
properties will be an a nested object (such as
Image.my_extension.my_attribute)?


>
> - These are stored in a registry.
> - When we come across an extension, we look it up in the registry and
> either bind or ignore it.
> - Every model object has an "Extensions" collection, which can be
> queried by type, to see if that extension data was present.
>
>
> (Note: this has mostly been tested with the XML)
>
> The nice thing about this is that a consumer of the library can write a
> binding for an extension, and register it with the client library, and "it
> just works". So, even if you have a private extension, which you tell
> nobody about and run only on your private cloud, you can use it with the
> stock Java library.
>
> Now, how you would do something that awesome in Python, I don't know ;-)
>

Probably the same way. I would use setuptools entry points to map namespace
prefixes to the class, so the client could load the handler automatically
when it encounters data it does not recognize.


>
> Justin
>
>
>
>
> On Fri, Apr 13, 2012 at 11:53 AM, Caitlin Bestler <
> Caitlin.Bestler [at] nexenta> wrote:
>
>> The argument that XML has better extensibility than JSON isn't very
>> convincing to my ears.
>>
>> I'm an old war horse, and recall extending message formats in ANSI C so
>> as to maintain backwards
>> compatibility with existing clients (by versioning the struct name itself
>> and always keeping the same
>> fields up front).
>>
>> If this problem could be solved in ANSI C, the solutions available to
>> Python with JSON can only be better.
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack [at] lists
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


justin at fathomdb

Apr 14, 2012, 7:46 AM

Post #13 of 15 (576 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

>
>
>> - Each (known) extension has its own strongly-typed model object.
>>
>>
> Does that mean that an extension cannot add properties to an existing
> object (such as adding a new attribute an Image), or just that all of those
> properties will be an a nested object (such as
> Image.my_extension.my_attribute)?
>

Good point - I should clarify... I'm not using the standard Java XML
binding code here, because they would require a wrapping object. It's
using custom code to do the binding whenever data from a different
namespace is encountered, although it uses the standard JAXB annotations to
guide it. From the user's point of view, "it just works"; the
implementation is not as nice :-)

Because we're not wrapping our extensions in their own elements, I didn't
know how to get the standard tooling to work with it anyway. Interesting
data-point for the "standard tools" argument...


> Now, how you would do something that awesome in Python, I don't know ;-)
>>
>
> Probably the same way. I would use setuptools entry points to map
> namespace prefixes to the class, so the client could load the handler
> automatically when it encounters data it does not recognize.
>

Yes - I should have said "dictionary" rather than "registry"! Are you
doing this in python-novaclient? Awesome!


doug.hellmann at dreamhost

Apr 14, 2012, 1:46 PM

Post #14 of 15 (575 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

On Sat, Apr 14, 2012 at 10:46 AM, Justin Santa Barbara
<justin [at] fathomdb>wrote:

>
>>> - Each (known) extension has its own strongly-typed model object.
>>>
>>>
>> Does that mean that an extension cannot add properties to an existing
>> object (such as adding a new attribute an Image), or just that all of those
>> properties will be an a nested object (such as
>> Image.my_extension.my_attribute)?
>>
>
> Good point - I should clarify... I'm not using the standard Java XML
> binding code here, because they would require a wrapping object. It's
> using custom code to do the binding whenever data from a different
> namespace is encountered, although it uses the standard JAXB annotations to
> guide it. From the user's point of view, "it just works"; the
> implementation is not as nice :-)
>

OK. I was asking from a lack-of-knowledge of how this sort of thing is
usually done in Java. If an extension adds a piece of data to an object,
what does a client need to do to access it?


>
> Because we're not wrapping our extensions in their own elements, I didn't
> know how to get the standard tooling to work with it anyway. Interesting
> data-point for the "standard tools" argument...
>
>
>> Now, how you would do something that awesome in Python, I don't know ;-)
>>>
>>
>> Probably the same way. I would use setuptools entry points to map
>> namespace prefixes to the class, so the client could load the handler
>> automatically when it encounters data it does not recognize.
>>
>
> Yes - I should have said "dictionary" rather than "registry"! Are you
> doing this in python-novaclient? Awesome!
>

I've done it before in other projects, but not in OpenStack, yet. I was
going to bring up the idea of using setuptools for plugins and extensions
like this at the summit during the session(s) on moving code into common
and revamping the CLI tools.

Doug


Caitlin.Bestler at nexenta

Apr 16, 2012, 3:12 PM

Post #15 of 15 (571 views)
Permalink
Re: Just JSON, and extensibility [In reply to]

ZFS feature flags are different on two important fronts.

First, they are binary. There is an intrinsic limit to the number claimable and no reason not to think that two project would not claim the same binary value.

More importantly, ZFS has a very specific challenge caused by Oracle maintaining their own closed source repository. There is a specific challenge in trying
To import a file system created under a closed repository to a file system built on the open repository (or vise versa).

Maybe I'm missing something, but where is the scenario for an openstack deployment that is built from mixed repositories?


From: Justin Santa Barbara [mailto:justin [at] fathomdb]
Sent: Friday, April 13, 2012 4:40 PM
To: Caitlin Bestler
Cc: Jorge Williams; Mark Nottingham; Thierry Carrez; <openstack [at] lists>
Subject: Re: [Openstack] Just JSON, and extensibility

It's easy when each new version is defined by a central body.

The problem we face is that we want to allow HP, Rackspace, Nexenta etc to define their own extensions, without serializing through a central body. Some extensions may only apply to private clouds and never be shared publicly.

This is a bit like ZFS feature flags, to use an example that should be near and dear to your heart!


On Fri, Apr 13, 2012 at 2:24 PM, Caitlin Bestler <Caitlin.Bestler [at] nexenta<mailto:Caitlin.Bestler [at] nexenta>> wrote:
Exactly what do you see as the required "non-linear extensibility"?

These are ultimately requests to a server. Each new extension is coded in that server.
There is no value in a client making up its own extensions that are not understood by the server.

What is relevant is a server continuing to support clients that have not yet been updated to
understand a new format.

As I stated in my first post, that problem was solved in ANSI C. Python/JSON is trivial.

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.