Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

[metering] Do we need an API and storage?

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


francis.lacoste at canonical

May 16, 2012, 2:00 PM

Post #1 of 11 (322 views)
Permalink
[metering] Do we need an API and storage?

Hi,

The whole API discussion made me wondered if this part of the
architecture is worth keeping.

The main use case for the metering API is so that billing systems can be
integrated in OpenStack. We have the assumption that any billing system
will need an integation layer. I think this is a fair assumption.

But at the same time, we are forcing the integration to be made around a
polling model. From time to time, poll the metering API to create
billing artefacts.

I'm now of the opinion that we exclude storage and API from the metering
project scope. Let's just focus on defining a metering message format,
bus, and maybe a client-library to make it easy to write metering consumers.

That way we avoid building a lot of things that we only _think will be
useful_ for potential billing integration. Only writing/delivering such
an integration component would prove that we built at least something
that is useful.

Cheers

--
Francis J. Lacoste
francis.lacoste [at] canonical
Attachments: signature.asc (0.26 KB)


acs at parvuscaptus

May 16, 2012, 4:22 PM

Post #2 of 11 (303 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

+1

On Wed, May 16, 2012 at 2:00 PM, Francis J. Lacoste <
francis.lacoste [at] canonical> wrote:

> Hi,
>
> The whole API discussion made me wondered if this part of the
> architecture is worth keeping.
>
> The main use case for the metering API is so that billing systems can be
> integrated in OpenStack. We have the assumption that any billing system
> will need an integation layer. I think this is a fair assumption.
>
> But at the same time, we are forcing the integration to be made around a
> polling model. From time to time, poll the metering API to create
> billing artefacts.
>
> I'm now of the opinion that we exclude storage and API from the metering
> project scope. Let's just focus on defining a metering message format,
> bus, and maybe a client-library to make it easy to write metering
> consumers.
>
> That way we avoid building a lot of things that we only _think will be
> useful_ for potential billing integration. Only writing/delivering such
> an integration component would prove that we built at least something
> that is useful.
>
> Cheers
>
> --
> Francis J. Lacoste
> francis.lacoste [at] canonical
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


loic at enovance

May 17, 2012, 2:13 AM

Post #3 of 11 (309 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
>
> I'm now of the opinion that we exclude storage and API from the metering
> project scope. Let's just focus on defining a metering message format,
> bus, and maybe a client-library to make it easy to write metering consumers.
>
> That way we avoid building a lot of things that we only _think will be
> useful_ for potential billing integration. Only writing/delivering such
> an integration component would prove that we built at least something
> that is useful.
>
>

Hi,

I'm a little reluctant to question the whole approach because I'm engaged in it :-) It's scary to contemplate the idea of throwing away some of the work done but I welcome the challenge. Better lose a few days work than keep going in the wrong direction.

Are you proposing that such a library would then be integrated in whatever billing system someone has in mind ? For instance

Dough https://github.com/lzyeval/dough
trystack.org billing https://github.com/trystack/dash_billing
nova-billing https://github.com/griddynamics/nova-billing

Would depend on this library and rely on its API to collect what they need. The same would be done by proprietary billing systems ?

Regarding the relevance of the metrics exposed by the API, I made sure they match the need of the eNovance sales. I'm quite sure Nicolas checked for Canonical and Doug did the same regarding the needs of Dreamhost. I'm confident that the information is actualy useful, at least in these practical cases.

Getting rid of the storage imposes a constraint on the billing system : it must make 100% sure that once a message is consumed it will be reliably archived. It also adds a constraint on the chosen bus : it must be able to retain all messages for as long as a consumer needs, which may be days or weeks. Or it adds a constraint on the billing system which must make 100% sure it will consume all relevant messages the bus at all times before they expire.

I have no strong feelings about exposing a bus for everyone to use instead of a REST API. I tend to prefer the REST API because it is an established standard for OpenStack. Could you expand on why a bus would be significantly better than a REST API in this specific case ?

Cheers

--
Loïc Dachary Chief Research Officer
// eNovance labs http://labs.enovance.com
// ✉ loic [at] enovance ☎ +33 1 49 70 99 82


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


nick.barcet at canonical

May 17, 2012, 2:47 AM

Post #4 of 11 (306 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On 05/17/2012 11:13 AM, Loic Dachary wrote:
> On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
>>
>> I'm now of the opinion that we exclude storage and API from the
>> metering project scope. Let's just focus on defining a metering
>> message format, bus, and maybe a client-library to make it easy to
>> write metering consumers.
>>
>> That way we avoid building a lot of things that we only _think will
>> be useful_ for potential billing integration. Only
>> writing/delivering such an integration component would prove that
>> we built at least something that is useful.
>>
>>
>
> Hi,
>
> I'm a little reluctant to question the whole approach because I'm
> engaged in it :-) It's scary to contemplate the idea of throwing away
> some of the work done but I welcome the challenge. Better lose a few
> days work than keep going in the wrong direction.
>
> Are you proposing that such a library would then be integrated in
> whatever billing system someone has in mind ? For instance
>
> Dough https://github.com/lzyeval/dough trystack.org billing
> https://github.com/trystack/dash_billing nova-billing
> https://github.com/griddynamics/nova-billing
>
> Would depend on this library and rely on its API to collect what they
> need. The same would be done by proprietary billing systems ?
>
> Regarding the relevance of the metrics exposed by the API, I made
> sure they match the need of the eNovance sales. I'm quite sure
> Nicolas checked for Canonical and Doug did the same regarding the
> needs of Dreamhost. I'm confident that the information is actualy
> useful, at least in these practical cases.
>
> Getting rid of the storage imposes a constraint on the billing system
> : it must make 100% sure that once a message is consumed it will be
> reliably archived. It also adds a constraint on the chosen bus : it
> must be able to retain all messages for as long as a consumer needs,
> which may be days or weeks. Or it adds a constraint on the billing
> system which must make 100% sure it will consume all relevant
> messages the bus at all times before they expire.
>
> I have no strong feelings about exposing a bus for everyone to use
> instead of a REST API. I tend to prefer the REST API because it is an
> established standard for OpenStack. Could you expand on why a bus
> would be significantly better than a REST API in this specific case
> ?

Thanks for challenging our thought process, Francis, it is a great
sanity check :)

I do have a few use cases where a REST API is better than a bus:

* Private clouds: Users are unlikely to want to activate a complete
billing system but still want to be able top provide to their users some
metrics of their uses. The REST API plus some scripts would allow them
to do this without too much pain.

* Metric of usage on GUI: It might be usefull to provide a quick way to
assess usage in almost real time to users as part of an extension of
Horizon for example. The REST API would allow for such data to be
extracted dynamically without having to run a full billing tool in real
time.

* In house billing tools: about half of the ISPs I surveyed are running
some form of a customized ERP system to handle the billing for their
customers. They need to be able to produce CSVs on a weekly basis to
feed their custom billing solutions. Integrating a bus would be much
more complex than the script that would issue request to the rest API.

* Auditability/Non repudiability: if the messages go from the queue to
some unknown DB, how do you solve, in a generic way, coherent audit
check and ensure non-repudiability?

I do not mean to totally discard Francis' bus idea though, and think
that we should allow for (a not necessarily db-less integration but)
direct queue attachment model for billing systems. I do think that for
all the above reasons, plus the simplification of testing of the overall
system, the REST API must remain part of this project.

Nick
Attachments: signature.asc (0.88 KB)


doug.hellmann at dreamhost

May 17, 2012, 5:14 AM

Post #5 of 11 (306 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On Thu, May 17, 2012 at 5:47 AM, Nick Barcet <nick.barcet [at] canonical>wrote:

> On 05/17/2012 11:13 AM, Loic Dachary wrote:
> > On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
> >>
> >> I'm now of the opinion that we exclude storage and API from the
> >> metering project scope. Let's just focus on defining a metering
> >> message format, bus, and maybe a client-library to make it easy to
> >> write metering consumers.
>

The plan, as I understand it, is to ensure that all metering messages
appear on a common bus using a documented format. Deployers who do not want
the storage system and REST API will not need to use it, and can set up
their own clients to listen on that bus. I'm not sure how much of a client
library is needed, since the bus is AMQP and the messages are JSON, both of
which have standard libraries in most common languages.


> >>
> >> That way we avoid building a lot of things that we only _think will
> >> be useful_ for potential billing integration. Only
> >> writing/delivering such an integration component would prove that
> >> we built at least something that is useful.
> >>
> >>
> >
> > Hi,
> >
> > I'm a little reluctant to question the whole approach because I'm
> > engaged in it :-) It's scary to contemplate the idea of throwing away
> > some of the work done but I welcome the challenge. Better lose a few
> > days work than keep going in the wrong direction.
> >
> > Are you proposing that such a library would then be integrated in
> > whatever billing system someone has in mind ? For instance
> >
> > Dough https://github.com/lzyeval/dough trystack.org billing
> > https://github.com/trystack/dash_billing nova-billing
> > https://github.com/griddynamics/nova-billing
> >
> > Would depend on this library and rely on its API to collect what they
> > need. The same would be done by proprietary billing systems ?
> >
> > Regarding the relevance of the metrics exposed by the API, I made
> > sure they match the need of the eNovance sales. I'm quite sure
> > Nicolas checked for Canonical and Doug did the same regarding the
> > needs of Dreamhost. I'm confident that the information is actualy
> > useful, at least in these practical cases.
>

Yes, all of the feedback I have provided has been directly related to needs
we have here at DreamHost. I expect us to use the API service to pull data
into our legacy billing system, which does not have an API of its own.


> >
> > Getting rid of the storage imposes a constraint on the billing system
> > : it must make 100% sure that once a message is consumed it will be
> > reliably archived. It also adds a constraint on the chosen bus : it
> > must be able to retain all messages for as long as a consumer needs,
> > which may be days or weeks. Or it adds a constraint on the billing
> > system which must make 100% sure it will consume all relevant
> > messages the bus at all times before they expire.
>

That's exactly right. It will be easier for me to bridge between our two
systems by pulling a day's worth of details from the ceilometer API and
storing them in the billing system using a batch job, rather than trying to
ensure that the billing database performs well enough to record the
information in real time. My goal is to not have to change the billing
system at all.


> >
> > I have no strong feelings about exposing a bus for everyone to use
> > instead of a REST API. I tend to prefer the REST API because it is an
> > established standard for OpenStack. Could you expand on why a bus
> > would be significantly better than a REST API in this specific case
> > ?
>
> Thanks for challenging our thought process, Francis, it is a great
> sanity check :)
>
> I do have a few use cases where a REST API is better than a bus:
>
> * Private clouds: Users are unlikely to want to activate a complete
> billing system but still want to be able top provide to their users some
> metrics of their uses. The REST API plus some scripts would allow them
> to do this without too much pain.
>
> * Metric of usage on GUI: It might be usefull to provide a quick way to
> assess usage in almost real time to users as part of an extension of
> Horizon for example. The REST API would allow for such data to be
> extracted dynamically without having to run a full billing tool in real
> time.
>

I don't know about real time, but I do know we eventually plan to provide
usage information like this to our users. So, change "might be useful" to
"will be useful." :-)


>
> * In house billing tools: about half of the ISPs I surveyed are running
> some form of a customized ERP system to handle the billing for their
> customers. They need to be able to produce CSVs on a weekly basis to
> feed their custom billing solutions. Integrating a bus would be much
> more complex than the script that would issue request to the rest API.
>

Right. A lot of accounting systems run in a batch model like this.


>
> * Auditability/Non repudiability: if the messages go from the queue to
> some unknown DB, how do you solve, in a generic way, coherent audit
> check and ensure non-repudiability?
>
> I do not mean to totally discard Francis' bus idea though, and think
> that we should allow for (a not necessarily db-less integration but)
> direct queue attachment model for billing systems. I do think that for
> all the above reasons, plus the simplification of testing of the overall
> system, the REST API must remain part of this project.
>

As I mentioned above, we will have the bus available. I expect to have
several routing keys, much as is done with notifications. For example, we
should have an easy way to receive all messages ("metering.all") as well as
a way to select only messages for counters the consumer is interested in
("metering.c1", "metering.n2", etc.). That should allow someone to replace
the ceilometer collector daemon with their own client and storage backend,
or to simply watch the messages for monitoring purposes.

Doug


thierry at openstack

May 18, 2012, 2:27 AM

Post #6 of 11 (302 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

Doug Hellmann wrote:
> On Thu, May 17, 2012 at 5:47 AM, Nick Barcet <nick.barcet [at] canonical
> <mailto:nick.barcet [at] canonical>> wrote:
>
> On 05/17/2012 11:13 AM, Loic Dachary wrote:
> > On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
> >>
> >> I'm now of the opinion that we exclude storage and API from the
> >> metering project scope. Let's just focus on defining a metering
> >> message format, bus, and maybe a client-library to make it easy to
> >> write metering consumers.
>
>
> The plan, as I understand it, is to ensure that all metering messages
> appear on a common bus using a documented format. Deployers who do not
> want the storage system and REST API will not need to use it, and can
> set up their own clients to listen on that bus. I'm not sure how much of
> a client library is needed, since the bus is AMQP and the messages are
> JSON, both of which have standard libraries in most common languages.
> [...]

You can certainly architect it in a way so that storage and API are
optional: expose metering messages on the bus, and provide an
optionally-run aggregation component that exposes a REST API (and that
would use the metering-consumer client library). That would give
deployers the option to poll via REST or implement their own alternate
aggregation using the metering-consumer client lib, depending on the
system they need to integrate with.

Having the aggregation component clearly separate and optional will
serve as a great example of how it could be done (and what are the
responsibilities of the aggregation component). I would still do a
(minimal) client library to facilitate integration, but maybe that's
just me :)

--
Thierry Carrez (ttx)
Release Manager, OpenStack

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


doug.hellmann at dreamhost

May 18, 2012, 7:27 AM

Post #7 of 11 (302 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On Fri, May 18, 2012 at 5:27 AM, Thierry Carrez <thierry [at] openstack>wrote:

> Doug Hellmann wrote:
> > On Thu, May 17, 2012 at 5:47 AM, Nick Barcet <nick.barcet [at] canonical
> > <mailto:nick.barcet [at] canonical>> wrote:
> >
> > On 05/17/2012 11:13 AM, Loic Dachary wrote:
> > > On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
> > >>
> > >> I'm now of the opinion that we exclude storage and API from the
> > >> metering project scope. Let's just focus on defining a metering
> > >> message format, bus, and maybe a client-library to make it easy to
> > >> write metering consumers.
> >
> >
> > The plan, as I understand it, is to ensure that all metering messages
> > appear on a common bus using a documented format. Deployers who do not
> > want the storage system and REST API will not need to use it, and can
> > set up their own clients to listen on that bus. I'm not sure how much of
> > a client library is needed, since the bus is AMQP and the messages are
> > JSON, both of which have standard libraries in most common languages.
> > [...]
>
> You can certainly architect it in a way so that storage and API are
> optional: expose metering messages on the bus, and provide an
> optionally-run aggregation component that exposes a REST API (and that
> would use the metering-consumer client library). That would give
> deployers the option to poll via REST or implement their own alternate
> aggregation using the metering-consumer client lib, depending on the
> system they need to integrate with.
>
> Having the aggregation component clearly separate and optional will
> serve as a great example of how it could be done (and what are the
> responsibilities of the aggregation component). I would still do a
> (minimal) client library to facilitate integration, but maybe that's
> just me :)


I can see some benefit to that, especially when it comes to validating the
message signatures.

Doug


francis.lacoste at canonical

May 18, 2012, 1:49 PM

Post #8 of 11 (305 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On 12-05-17 08:14 AM, Doug Hellmann wrote:
>
>
> On Thu, May 17, 2012 at 5:47 AM, Nick Barcet <nick.barcet [at] canonical
> <mailto:nick.barcet [at] canonical>> wrote:
>
> On 05/17/2012 11:13 AM, Loic Dachary wrote:
> > On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
> >>
> >> I'm now of the opinion that we exclude storage and API from the
> >> metering project scope. Let's just focus on defining a metering
> >> message format, bus, and maybe a client-library to make it easy to
> >> write metering consumers.
>
>
> The plan, as I understand it, is to ensure that all metering messages
> appear on a common bus using a documented format. Deployers who do not
> want the storage system and REST API will not need to use it, and can
> set up their own clients to listen on that bus. I'm not sure how much of
> a client library is needed, since the bus is AMQP and the messages are
> JSON, both of which have standard libraries in most common languages.

Like Thierry Carrez mentioned, the main use for a library was to handle
validation of message signature in a handy fashion. Bug I agree that
this client library would just be a thin convenience wrapper around the
bus protocol.


>
> >
> > Getting rid of the storage imposes a constraint on the billing system
> > : it must make 100% sure that once a message is consumed it will be
> > reliably archived. It also adds a constraint on the chosen bus : it
> > must be able to retain all messages for as long as a consumer needs,
> > which may be days or weeks. Or it adds a constraint on the billing
> > system which must make 100% sure it will consume all relevant
> > messages the bus at all times before they expire.
>
>
> That's exactly right. It will be easier for me to bridge between our two
> systems by pulling a day's worth of details from the ceilometer API and
> storing them in the billing system using a batch job, rather than trying
> to ensure that the billing database performs well enough to record the
> information in real time. My goal is to not have to change the billing
> system at all.

That's good information to have. Which means that a REST API + storage
component definitively has some values for some integration cases.

--
Francis J. Lacoste
francis.lacoste [at] canonical
Attachments: signature.asc (0.26 KB)


francis.lacoste at canonical

May 18, 2012, 1:53 PM

Post #9 of 11 (305 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On 12-05-18 05:27 AM, Thierry Carrez wrote:
> You can certainly architect it in a way so that storage and API are
> optional: expose metering messages on the bus, and provide an
> optionally-run aggregation component that exposes a REST API (and that
> would use the metering-consumer client library). That would give
> deployers the option to poll via REST or implement their own alternate
> aggregation using the metering-consumer client lib, depending on the
> system they need to integrate with.
>
> Having the aggregation component clearly separate and optional will
> serve as a great example of how it could be done (and what are the
> responsibilities of the aggregation component). I would still do a
> (minimal) client library to facilitate integration, but maybe that's
> just me :)

Right, I like this approach very much.

The main thing I'm worried about is that we are building a system that
has no use in _itself_. It's all about enabling integration in
third-party billing system, but we aren't building such an integration
as part of this project.

We could easily implement something that lacks our focus. Maybe, that's
an argument for building a simple billing app as part of OpenStack as a
proof of concept that the metering system can be integrated.

Sure, interested parties will try to integrate it once we have early
versions of it, but that still increase the feedback cycle on our
API/architecture hypotheses.

Cheers

--
Francis J. Lacoste
francis.lacoste [at] canonical
Attachments: signature.asc (0.26 KB)


francis.lacoste at canonical

May 18, 2012, 1:55 PM

Post #10 of 11 (306 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On 12-05-18 04:49 PM, Francis J. Lacoste wrote:
>> On 12-05-17 08:14 AM, Doug Hellmann wrote:
>> > The plan, as I understand it, is to ensure that all metering messages
>> > appear on a common bus using a documented format. Deployers who do not
>> > want the storage system and REST API will not need to use it, and can
>> > set up their own clients to listen on that bus. I'm not sure how much of
>> > a client library is needed, since the bus is AMQP and the messages are
>> > JSON, both of which have standard libraries in most common languages.

> Like Thierry Carrez mentioned, the main use for a library was to handle
> validation of message signature in a handy fashion. Bug I agree that
> this client library would just be a thin convenience wrapper around the
> bus protocol.
>

Of course, it was you in your reply to Thierry that mentioned this :-)


--
Francis J. Lacoste
francis.lacoste [at] canonical
Attachments: signature.asc (0.26 KB)


doug.hellmann at dreamhost

May 18, 2012, 2:33 PM

Post #11 of 11 (306 views)
Permalink
Re: [metering] Do we need an API and storage? [In reply to]

On Fri, May 18, 2012 at 4:53 PM, Francis J. Lacoste <
francis.lacoste [at] canonical> wrote:

> On 12-05-18 05:27 AM, Thierry Carrez wrote:
> > You can certainly architect it in a way so that storage and API are
> > optional: expose metering messages on the bus, and provide an
> > optionally-run aggregation component that exposes a REST API (and that
> > would use the metering-consumer client library). That would give
> > deployers the option to poll via REST or implement their own alternate
> > aggregation using the metering-consumer client lib, depending on the
> > system they need to integrate with.
> >
> > Having the aggregation component clearly separate and optional will
> > serve as a great example of how it could be done (and what are the
> > responsibilities of the aggregation component). I would still do a
> > (minimal) client library to facilitate integration, but maybe that's
> > just me :)
>
> Right, I like this approach very much.
>
> The main thing I'm worried about is that we are building a system that
> has no use in _itself_. It's all about enabling integration in
> third-party billing system, but we aren't building such an integration
> as part of this project.
>

Well, several of us actually *are* building such integration systems at the
same time that we are building ceilometer. That's where these requirements
are coming from! :-) I don't expect to be releasing all of the code for
that integration, but I will release what I can and I am happy to talk
about the general requirements and constraints for the rest on the list to
help with the design of ceilometer.


> We could easily implement something that lacks our focus. Maybe, that's
> an argument for building a simple billing app as part of OpenStack as a
> proof of concept that the metering system can be integrated.
>

I would not object if you wanted to do that, but I have a high degree of
confidence that we can produce something usable and useful without going
that far.


>
> Sure, interested parties will try to integrate it once we have early
> versions of it, but that still increase the feedback cycle on our
> API/architecture hypotheses.
>

I could shorten that feedback cycle if folks would do code reviews for the
outstanding items at
https://review.stackforge.org/#/q/status:open+project:stackforge/ceilometer,n,zso
I could stand up a copy of what has already been implemented. ;-)

In all seriousness, it seems reasonable for us to concentrate on the
front-end pieces (collecting and storing) for this release, and build a
"good enough" API service to retrieve data. Even if that means I end up
having to retrieve all of the raw records and process them myself, I can
still get my project done as a proof of concept and we can refine the API
as we go along using the experience I (and others) gain this time around.

Doug

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.