Mailing List Archive: OpenStack: Dev

Distributed rate-limiting

Index | Next | Previous | View Threaded

kevin.mitchell at rackspace

Mar 16, 2012, 2:45 PM

Post #1 of 11 (186 views)
Permalink

Distributed rate-limiting

Howdy, folks. I've been working on a replacement for nova's
rate-limiting middleware that will handle the multiple-node case, and
I've developed a fairly generic rate-limiting package, along with a
second package that adapts it to nova. (This means you could also use
this rate-limiting setup with, say, glance, or with any other project
that uses Python middleware.) Here is some information:

* Turnstile
Turnstile is a piece of WSGI middleware that performs true distributed
rate-limiting. System administrators can run an API on multiple
nodes, then place this middleware in the pipeline prior to the
application. Turnstile uses a Redis database to track the rate at
which users are hitting the API, and can then apply configured rate
limits, even if each request was made against a different API node.

- https://github.com/klmitch/turnstile
- http://pypi.python.org/pypi/turnstile

* nova_limits
This package provides the ``nova_limits`` Python module, which
contains the ``nova_preprocess()`` preprocessor, the
``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware``
replacement middleware class, all for use with Turnstile. These
pieces work together to provide class-based rate limiting integration
with nova.

- https://github.com/klmitch/nova_limits
- http://pypi.python.org/pypi/nova_limits

Both packages should be fairly well documented (start with README.rst),
and please feel free to log issues or make pull requests.
--
Kevin L. Mitchell <kevin.mitchell [at] rackspace>

jaypipes at gmail

Mar 17, 2012, 9:31 AM

Post #2 of 11 (189 views)
Permalink

Re: Distributed rate-limiting [In reply to]

Rock. On.

Kevin, you've really impressed me. Well documented, well thought-out code.

I hope you won't mind if I contribute a REST-ful interface for
configuration management and status reporting?

Best,
-jay

On 03/16/2012 05:45 PM, Kevin L. Mitchell wrote:
> Howdy, folks. I've been working on a replacement for nova's
> rate-limiting middleware that will handle the multiple-node case, and
> I've developed a fairly generic rate-limiting package, along with a
> second package that adapts it to nova. (This means you could also use
> this rate-limiting setup with, say, glance, or with any other project
> that uses Python middleware.) Here is some information:
>
> * Turnstile
> Turnstile is a piece of WSGI middleware that performs true distributed
> rate-limiting. System administrators can run an API on multiple
> nodes, then place this middleware in the pipeline prior to the
> application. Turnstile uses a Redis database to track the rate at
> which users are hitting the API, and can then apply configured rate
> limits, even if each request was made against a different API node.
>
> - https://github.com/klmitch/turnstile
> - http://pypi.python.org/pypi/turnstile
>
> * nova_limits
> This package provides the ``nova_limits`` Python module, which
> contains the ``nova_preprocess()`` preprocessor, the
> ``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware``
> replacement middleware class, all for use with Turnstile. These
> pieces work together to provide class-based rate limiting integration
> with nova.
>
> - https://github.com/klmitch/nova_limits
> - http://pypi.python.org/pypi/nova_limits
>
> Both packages should be fairly well documented (start with README.rst),
> and please feel free to log issues or make pull requests.

me at not

Mar 17, 2012, 9:57 AM

Post #3 of 11 (198 views)
Permalink

Re: Distributed rate-limiting [In reply to]

If you're looking for wsgi rate limiting, take a look at swift's rate limiting middleware. It's distributed, simple (uses memcached to keep track of what's going on), supports multiple tiers of rate limiting (with interpolation between them), and white/black lists.

While swift's rate limiting is tuned for swift, its technique could be pretty simply adapted to other applications. Swift's rate limiting is swift-specific in what it checks to rate limit on. Those checks should be able to be adapted to other use cases.

source: https://github.com/openstack/swift/blob/master/swift/common/middleware/ratelimit.py
docs: https://github.com/openstack/swift/blob/master/doc/source/ratelimit.rst
sample config: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L118

--John

On Mar 16, 2012, at 4:45 PM, Kevin L. Mitchell wrote:

> Howdy, folks. I've been working on a replacement for nova's
> rate-limiting middleware that will handle the multiple-node case, and
> I've developed a fairly generic rate-limiting package, along with a
> second package that adapts it to nova. (This means you could also use
> this rate-limiting setup with, say, glance, or with any other project
> that uses Python middleware.) Here is some information:
>
> * Turnstile
> Turnstile is a piece of WSGI middleware that performs true distributed
> rate-limiting. System administrators can run an API on multiple
> nodes, then place this middleware in the pipeline prior to the
> application. Turnstile uses a Redis database to track the rate at
> which users are hitting the API, and can then apply configured rate
> limits, even if each request was made against a different API node.
>
> - https://github.com/klmitch/turnstile
> - http://pypi.python.org/pypi/turnstile
>
> * nova_limits
> This package provides the ``nova_limits`` Python module, which
> contains the ``nova_preprocess()`` preprocessor, the
> ``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware``
> replacement middleware class, all for use with Turnstile. These
> pieces work together to provide class-based rate limiting integration
> with nova.
>
> - https://github.com/klmitch/nova_limits
> - http://pypi.python.org/pypi/nova_limits
>
> Both packages should be fairly well documented (start with README.rst),
> and please feel free to log issues or make pull requests.
> --
> Kevin L. Mitchell <kevin.mitchell [at] rackspace>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

Attachments:

smime.p7s (4.23 KB)

kiall at managedit

Mar 17, 2012, 10:09 AM

Post #4 of 11 (192 views)
Permalink

Re: Distributed rate-limiting [In reply to]

I was just thinking about why redis was chosen over memcache..

Ignoring any capability differences, service and technology sprawl was the
first thing that crossed my mind.

Is redis providing more than the, more then likely preexisting, memcache
and rabbitmq combination?

Thanks,
Kiall

Sent from my phone.
On Mar 17, 2012 4:58 p.m., "John Dickinson" <me [at] not> wrote:

> If you're looking for wsgi rate limiting, take a look at swift's rate
> limiting middleware. It's distributed, simple (uses memcached to keep track
> of what's going on), supports multiple tiers of rate limiting (with
> interpolation between them), and white/black lists.
>
> While swift's rate limiting is tuned for swift, its technique could be
> pretty simply adapted to other applications. Swift's rate limiting is
> swift-specific in what it checks to rate limit on. Those checks should be
> able to be adapted to other use cases.
>
> source:
> https://github.com/openstack/swift/blob/master/swift/common/middleware/ratelimit.py
> docs:
> https://github.com/openstack/swift/blob/master/doc/source/ratelimit.rst
> sample config:
> https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L118
>
> --John
>
> On Mar 16, 2012, at 4:45 PM, Kevin L. Mitchell wrote:
>
> > Howdy, folks. I've been working on a replacement for nova's
> > rate-limiting middleware that will handle the multiple-node case, and
> > I've developed a fairly generic rate-limiting package, along with a
> > second package that adapts it to nova. (This means you could also use
> > this rate-limiting setup with, say, glance, or with any other project
> > that uses Python middleware.) Here is some information:
> >
> > * Turnstile
> > Turnstile is a piece of WSGI middleware that performs true
> distributed
> > rate-limiting. System administrators can run an API on multiple
> > nodes, then place this middleware in the pipeline prior to the
> > application. Turnstile uses a Redis database to track the rate at
> > which users are hitting the API, and can then apply configured
> rate
> > limits, even if each request was made against a different API
> node.
> >
> > - https://github.com/klmitch/turnstile
> > - http://pypi.python.org/pypi/turnstile
> >
> > * nova_limits
> > This package provides the ``nova_limits`` Python module, which
> > contains the ``nova_preprocess()`` preprocessor, the
> > ``NovaClassLimit`` limit class, and the
> ``NovaTurnstileMiddleware``
> > replacement middleware class, all for use with Turnstile. These
> > pieces work together to provide class-based rate limiting
> integration
> > with nova.
> >
> > - https://github.com/klmitch/nova_limits
> > - http://pypi.python.org/pypi/nova_limits
> >
> > Both packages should be fairly well documented (start with README.rst),
> > and please feel free to log issues or make pull requests.
> > --
> > Kevin L. Mitchell <kevin.mitchell [at] rackspace>
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>

kevin.mitchell at rackspace

Mar 19, 2012, 8:19 AM

Post #5 of 11 (188 views)
Permalink

Re: Distributed rate-limiting [In reply to]

On Sat, 2012-03-17 at 12:31 -0400, Jay Pipes wrote:
> Kevin, you've really impressed me. Well documented, well thought-out code.

Yeah, well…I got into the habit of documenting my code well when I wrote
a very large project and discovered I was forgetting how to use its
pieces :)

> I hope you won't mind if I contribute a REST-ful interface for
> configuration management and status reporting?

Not at all; I designed the limit classes—specifically the limit
attributes—to be easily introspectable for exactly that sort of purpose.
--
Kevin L. Mitchell <kevin.mitchell [at] rackspace>

philip.day at hp

Mar 29, 2012, 2:58 PM

Post #6 of 11 (186 views)
Permalink

Re: Distributed rate-limiting [In reply to]

Hi Kevin,

A couple of quick questions:

- As you get the tenant id from the context I assume this module has to come after the authentication in the pipeline. Have you thought about using the tenant_id in the URL instead ? (I'm thinking of the case where you want rate limit requests into the authentication system as well as Nova itself).

- Does this work for EC2 as well as OSAPI ?

Cheers,
Phil

-----Original Message-----
From: openstack-bounces+philip.day=hp.com [at] lists [mailto:openstack-bounces+philip.day=hp.com [at] lists] On Behalf Of Kevin L. Mitchell
Sent: 16 March 2012 21:45
To: openstack [at] lists
Subject: [Openstack] Distributed rate-limiting

Howdy, folks. I've been working on a replacement for nova's rate-limiting middleware that will handle the multiple-node case, and I've developed a fairly generic rate-limiting package, along with a second package that adapts it to nova. (This means you could also use this rate-limiting setup with, say, glance, or with any other project that uses Python middleware.) Here is some information:

* Turnstile
Turnstile is a piece of WSGI middleware that performs true distributed
rate-limiting. System administrators can run an API on multiple
nodes, then place this middleware in the pipeline prior to the
application. Turnstile uses a Redis database to track the rate at
which users are hitting the API, and can then apply configured rate
limits, even if each request was made against a different API node.

- https://github.com/klmitch/turnstile
- http://pypi.python.org/pypi/turnstile

* nova_limits
This package provides the ``nova_limits`` Python module, which
contains the ``nova_preprocess()`` preprocessor, the
``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware``
replacement middleware class, all for use with Turnstile. These
pieces work together to provide class-based rate limiting integration
with nova.

- https://github.com/klmitch/nova_limits
- http://pypi.python.org/pypi/nova_limits

Both packages should be fairly well documented (start with README.rst), and please feel free to log issues or make pull requests.
--
Kevin L. Mitchell <kevin.mitchell [at] rackspace>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

kevin.mitchell at rackspace

Mar 29, 2012, 3:26 PM

Post #7 of 11 (185 views)
Permalink

Re: Distributed rate-limiting [In reply to]

On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote:
> - As you get the tenant id from the context I assume this module has
> to come after the authentication in the pipeline.

Yes, I have made that assumption. It seems reasonable, given that the
existing rate-limit middleware is right after authentication as well.

> Have you thought about using the tenant_id in the URL instead ? (I'm
> thinking of the case where you want rate limit requests into the
> authentication system as well as Nova itself).

No, I haven't. I don't trust the user, which is where the tenant_id in
the URL comes from. I do trust the auth system, which is why I want to
use the tenant ID from the context. (And yes, you could argue that
authz would prevent their access to other tenants anyway, but why make
nova have to check authz if rate limiting would stop them in the first
place?)

As for rate limiting requests into the authentication system, I'd
suggest using a Limit subclass which uses the remote IP address in place
of a tenant ID, at least for the user endpoint. I don't think we want
any rate limiting at all on the service side of Keystone; our current
architecture means that Keystone is going to be hit a *lot*: at least
once for each request that hits Nova, and more in certain cases (i.e.,
instance boot, where we'll have to hit quantum and glance as well).

> - Does this work for EC2 as well as OSAPI ?

Actually, it didn't occur to me to test, given that I don't really use
the EC2 API. I don't think there's anything in the basic architecture
which would be incompatible with EC2; the only possible sticking point
that occurs to me is the URL construction in
nova_limits:NovaClassLimit.route(): if the URL specified is prefixed
with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise
the route wouldn't match). That would be easy to work around; simply
extend NovaClassLimit and override route() to do the appropriate
transformation for EC2. Any EC2 experts want to weigh in?
--
Kevin L. Mitchell <kevin.mitchell [at] rackspace>

cbehrens at codestud

Mar 29, 2012, 4:30 PM

Post #8 of 11 (191 views)
Permalink

Re: Distributed rate-limiting [In reply to]

My issue with using the URL is someone could easily DoS any tenant. Maybe you said that below. I only have a brief moment to scan email ATM. :)

On Mar 29, 2012, at 5:26 PM, "Kevin L. Mitchell" <kevin.mitchell [at] rackspace> wrote:

> On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote:
>> - As you get the tenant id from the context I assume this module has
>> to come after the authentication in the pipeline.
>
> Yes, I have made that assumption. It seems reasonable, given that the
> existing rate-limit middleware is right after authentication as well.
>
>> Have you thought about using the tenant_id in the URL instead ? (I'm
>> thinking of the case where you want rate limit requests into the
>> authentication system as well as Nova itself).
>
> No, I haven't. I don't trust the user, which is where the tenant_id in
> the URL comes from. I do trust the auth system, which is why I want to
> use the tenant ID from the context. (And yes, you could argue that
> authz would prevent their access to other tenants anyway, but why make
> nova have to check authz if rate limiting would stop them in the first
> place?)
>
> As for rate limiting requests into the authentication system, I'd
> suggest using a Limit subclass which uses the remote IP address in place
> of a tenant ID, at least for the user endpoint. I don't think we want
> any rate limiting at all on the service side of Keystone; our current
> architecture means that Keystone is going to be hit a *lot*: at least
> once for each request that hits Nova, and more in certain cases (i.e.,
> instance boot, where we'll have to hit quantum and glance as well).
>
>> - Does this work for EC2 as well as OSAPI ?
>
> Actually, it didn't occur to me to test, given that I don't really use
> the EC2 API. I don't think there's anything in the basic architecture
> which would be incompatible with EC2; the only possible sticking point
> that occurs to me is the URL construction in
> nova_limits:NovaClassLimit.route(): if the URL specified is prefixed
> with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise
> the route wouldn't match). That would be easy to work around; simply
> extend NovaClassLimit and override route() to do the appropriate
> transformation for EC2. Any EC2 experts want to weigh in?
> --
> Kevin L. Mitchell <kevin.mitchell [at] rackspace>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

philip.day at hp

Mar 30, 2012, 2:06 AM

Post #9 of 11 (190 views)
Permalink

Re: Distributed rate-limiting [In reply to]

Yep - good point Chris & Kevin. I hadn't thought of it that way.

-----Original Message-----
From: openstack-bounces+philip.day=hp.com [at] lists [mailto:openstack-bounces+philip.day=hp.com [at] lists] On Behalf Of Chris Behrens
Sent: 30 March 2012 00:30
To: Kevin L. Mitchell
Cc: openstack [at] lists
Subject: Re: [Openstack] Distributed rate-limiting

My issue with using the URL is someone could easily DoS any tenant. Maybe you said that below. I only have a brief moment to scan email ATM. :)

On Mar 29, 2012, at 5:26 PM, "Kevin L. Mitchell" <kevin.mitchell [at] rackspace> wrote:

> On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote:
>> - As you get the tenant id from the context I assume this module has
>> to come after the authentication in the pipeline.
>
> Yes, I have made that assumption. It seems reasonable, given that the
> existing rate-limit middleware is right after authentication as well.
>
>> Have you thought about using the tenant_id in the URL instead ? (I'm
>> thinking of the case where you want rate limit requests into the
>> authentication system as well as Nova itself).
>
> No, I haven't. I don't trust the user, which is where the tenant_id
> in the URL comes from. I do trust the auth system, which is why I
> want to use the tenant ID from the context. (And yes, you could argue
> that authz would prevent their access to other tenants anyway, but why
> make nova have to check authz if rate limiting would stop them in the
> first
> place?)
>
> As for rate limiting requests into the authentication system, I'd
> suggest using a Limit subclass which uses the remote IP address in
> place of a tenant ID, at least for the user endpoint. I don't think
> we want any rate limiting at all on the service side of Keystone; our
> current architecture means that Keystone is going to be hit a *lot*:
> at least once for each request that hits Nova, and more in certain
> cases (i.e., instance boot, where we'll have to hit quantum and glance as well).
>
>> - Does this work for EC2 as well as OSAPI ?
>
> Actually, it didn't occur to me to test, given that I don't really use
> the EC2 API. I don't think there's anything in the basic architecture
> which would be incompatible with EC2; the only possible sticking point
> that occurs to me is the URL construction in
> nova_limits:NovaClassLimit.route(): if the URL specified is prefixed
> with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise
> the route wouldn't match). That would be easy to work around; simply
> extend NovaClassLimit and override route() to do the appropriate
> transformation for EC2. Any EC2 experts want to weigh in?
> --
> Kevin L. Mitchell <kevin.mitchell [at] rackspace>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

Rohit.Karajgi at nttdata

Dec 5, 2012, 6:12 AM

Post #10 of 11 (100 views)
Permalink

Re: Distributed rate-limiting [In reply to]

Hi,

Sorry to bring alive a fairly old thread, but I had a few questions on Nova's rate limiting in a distributed/ load balanced Openstack environment.

My understanding is Turnstile manages the situation where, the in-memory rate limits that are configured on load balanced API servers
are imposed properly on the incoming requests, so each API server is correctly updated/synced with the used rate limits.
Can you please confirm this understanding?

Also, I don't think this is part of the Openstack trunk code, and if so, is there any reason why it's not part of Nova, as it was meant to be a replacement?

Regards,
Rohit

-----Original Message-----
From: openstack-bounces+rohit.karajgi=vertex.co.in [at] lists [mailto:openstack-bounces+rohit.karajgi=vertex.co.in [at] lists] On Behalf Of Kevin L. Mitchell
Sent: Saturday, March 17, 2012 3:15 AM
To: openstack [at] lists
Subject: [Openstack] Distributed rate-limiting

Howdy, folks. I've been working on a replacement for nova's rate-limiting middleware that will handle the multiple-node case, and I've developed a fairly generic rate-limiting package, along with a second package that adapts it to nova. (This means you could also use this rate-limiting setup with, say, glance, or with any other project that uses Python middleware.) Here is some information:

* Turnstile
Turnstile is a piece of WSGI middleware that performs true distributed
rate-limiting. System administrators can run an API on multiple
nodes, then place this middleware in the pipeline prior to the
application. Turnstile uses a Redis database to track the rate at
which users are hitting the API, and can then apply configured rate
limits, even if each request was made against a different API node.

- https://github.com/klmitch/turnstile
- http://pypi.python.org/pypi/turnstile

* nova_limits
This package provides the ``nova_limits`` Python module, which
contains the ``nova_preprocess()`` preprocessor, the
``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware``
replacement middleware class, all for use with Turnstile. These
pieces work together to provide class-based rate limiting integration
with nova.

- https://github.com/klmitch/nova_limits
- http://pypi.python.org/pypi/nova_limits

Both packages should be fairly well documented (start with README.rst), and please feel free to log issues or make pull requests.
--
Kevin L. Mitchell <kevin.mitchell [at] rackspace>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

kevin.mitchell at rackspace

Dec 5, 2012, 8:15 AM

Post #11 of 11 (94 views)
Permalink

Re: Distributed rate-limiting [In reply to]

On Wed, 2012-12-05 at 14:12 +0000, Karajgi, Rohit wrote:
> My understanding is Turnstile manages the situation where, the
> in-memory rate limits that are configured on load balanced API servers
> are imposed properly on the incoming requests, so each API server is
> correctly updated/synced with the used rate limits.
> Can you please confirm this understanding?

Yes. Turnstile uses Redis to coordinate rate limit configuration and
bucket data, in order to provide rate limiting.

> Also, I don't think this is part of the Openstack trunk code, and if
> so, is there any reason why it's not part of Nova, as it was meant to
> be a replacement?

I wrote Turnstile to be general; it can be used for Nova, Keystone, or
any other system for which rate limiting is desired. (I in fact
designed it with a goal of being able to use it for some personal
projects which are not OpenStack-related.) This is the primary reason
it's not a direct part of any OpenStack repository. That said, it is
hosted on github and I welcome pull-requests…and I'm not at all adverse
to the suggestion that it become an OpenStack project; I'm just not
convinced that that would be generally desired, or that it would be
generally beneficial…
--
Kevin L. Mitchell <kevin.mitchell [at] rackspace>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads