Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Misc

High CPU usage when using ban with HTTP PURGE

 

 

Varnish misc RSS feed   Index | Next | Previous | View Threaded


connor.walls at skillpages

Jul 26, 2012, 1:54 AM

Post #1 of 4 (998 views)
Permalink
High CPU usage when using ban with HTTP PURGE

Hi all,

So, I've been having an issue with CPU usage recently. We run a cluster of 3 varnish boxes, running 3.0.2, serving content on a particular port, and accepting HTTP PURGE request on another (:82). This is used so whenever a user on the site changes something on their profile page, one of the backend servers will send a HTTP PURGE request to each box in the cluster, containing a regex in the header, which is then used to ban based on a particular URL - the relevant VCL is as follows:

//to allow purging
if (req.request == "PURGE") {
if (!server.port == 82) {
error 405 "Not Allowed.";
return (lookup);
} else {
if(req.http.Purge-Regex)
{
ban("req.url ~ " + req.http.Purge-Regex);
error 200 "Purged";
}
else
{
error 200 "Not Purged";
}
}
}

Now what we've been seeing is very high CPU usage with this functionality, which has been slowly growing over time with increased traffic on the website, and is now averaging above 70%, peaking at 100% at times and causing some user requests to fail. The CPU usage correlates very closely with timeout exceptions we get when making the HTTP PURGE requests. This became an issue yesterday to the point where we disabled the HTTP PURGE - CPU usage has now fallen from averaging > 70% to averaging around 8%, so clearly the purges are the issue. What I'm wondering is whether or not banning like this is going to be inherently heavy and I need to find a way to throttle the purge requests, or if I could rework the VCL so it will be less CPU heavy to process these requests? Will the CPU usage depend on what the actual regex is?

Thanks,
Connor Walls


lasse.karstensen at gmail

Jul 26, 2012, 2:36 AM

Post #2 of 4 (966 views)
Permalink
Re: High CPU usage when using ban with HTTP PURGE [In reply to]

Connor Walls:
> So, I've been having an issue with CPU usage recently. We run a cluster of
> 3 varnish boxes, running 3.0.2, serving content on a particular port, and
> accepting HTTP PURGE request on another (:82). This is used so whenever a
> user on the site changes something on their profile page, one of the
> backend servers will send a HTTP PURGE request to each box in the cluster,
> containing a regex in the header, which is then used to ban based on a
> particular URL - the relevant VCL is as follows:
[..]
> Now what we've been seeing is very high CPU usage with this functionality,
> which has been slowly growing over time with increased traffic on the
> website, and is now averaging above 70%, peaking at 100% at times and
> causing some user requests to fail. The CPU usage correlates very closely
> with timeout exceptions we get when making the HTTP PURGE requests. This
> became an issue yesterday to the point where we disabled the HTTP PURGE -
> CPU usage has now fallen from averaging > 70% to averaging around 8%, so
> clearly the purges are the issue. What I'm wondering is whether or not
> banning like this is going to be inherently heavy and I need to find a way
> to throttle the purge requests, or if I could rework the VCL so it will be
> less CPU heavy to process these requests? Will the CPU usage depend on what
> the actual regex is?

There are two methods for cache invalidation in Varnish 3.0: Purges and bans.
Bans are flexible and cool, and purges are less flexible but very fast. Use
purges if you can.

If you are using a high amount of bans, you should use what Kristian calls
"Smart bans" in the Varnish Book:

https://www.varnish-software.com/static/book/Cache_invalidation.html#smart-bans


--
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


connor.walls at skillpages

Jul 30, 2012, 4:04 AM

Post #3 of 4 (979 views)
Permalink
RE: High CPU usage when using ban with HTTP PURGE [In reply to]

Hi Lasse, thanks for the prompt reply. I've done a little bit more investigation on this issue.

1. I tried using smart bans but I saw no change in performance. I tried change the ban from a regex to a straight match on the URL (i.e. from ban("obj.http.x-url ~ " + [header containing regex]) to ban("obj.http.x-url == " + [header containing url]) ) and I still observed the same performance issues. I also tried changing from ban() to ban_url() - no performance improvement. Removing that line from the VCL brings the CPU right back down, so I've concluded the problem is adding the ban to the ban list. Is this an inherently CPU intensive operation, is there going to be no way around this?

2. I've also gone down the route of trying purges because they are significantly faster and look like they can handle the load okay, but my only issue is matching variations of a page. My vcl_hash function looks like this:

sub vcl_hash {
hash_data(req.http.Accept);
hash_data(req.http.X-Forwarded-Proto);
}

So the problem I'm having is purging all variations of the file based on "Accept" and "X-Forwarded-Proto" headers, as well as the host. Is this something that is even possible? I personally wouldn't have thought so but after seeing in the documentation (https://www.varnish-cache.org/docs/trunk/tutorial/purging.html): "The purge in vcl_miss is necessary to purge all variants in the cases where you hit an object, but miss a particular variant." Seems to imply that this functionality is present but I can't find any more documentation on how it works. Do I need to remove the hash_data() calls in vcl_hash and instead include "Accept" and "X-Forwarded-Proto" in the "Vary" header coming from the backend server?

Thanks,
Connor Walls

-----Original Message-----
From: varnish-misc-bounces [at] varnish-cache [mailto:varnish-misc-bounces [at] varnish-cache] On Behalf Of Lasse Karstensen
Sent: 26 July 2012 10:36
To: varnish-misc [at] varnish-cache
Subject: Re: High CPU usage when using ban with HTTP PURGE

Connor Walls:
> So, I've been having an issue with CPU usage recently. We run a
> cluster of
> 3 varnish boxes, running 3.0.2, serving content on a particular port,
> and accepting HTTP PURGE request on another (:82). This is used so
> whenever a user on the site changes something on their profile page,
> one of the backend servers will send a HTTP PURGE request to each box
> in the cluster, containing a regex in the header, which is then used
> to ban based on a particular URL - the relevant VCL is as follows:
[..]
> Now what we've been seeing is very high CPU usage with this
> functionality, which has been slowly growing over time with increased
> traffic on the website, and is now averaging above 70%, peaking at
> 100% at times and causing some user requests to fail. The CPU usage
> correlates very closely with timeout exceptions we get when making the
> HTTP PURGE requests. This became an issue yesterday to the point where
> we disabled the HTTP PURGE - CPU usage has now fallen from averaging >
> 70% to averaging around 8%, so clearly the purges are the issue. What
> I'm wondering is whether or not banning like this is going to be
> inherently heavy and I need to find a way to throttle the purge
> requests, or if I could rework the VCL so it will be less CPU heavy to
> process these requests? Will the CPU usage depend on what the actual regex is?

There are two methods for cache invalidation in Varnish 3.0: Purges and bans.
Bans are flexible and cool, and purges are less flexible but very fast. Use purges if you can.

If you are using a high amount of bans, you should use what Kristian calls "Smart bans" in the Varnish Book:

https://www.varnish-software.com/static/book/Cache_invalidation.html#smart-bans


--
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


perbu at varnish-software

Aug 1, 2012, 12:06 AM

Post #4 of 4 (952 views)
Permalink
Re: High CPU usage when using ban with HTTP PURGE [In reply to]

Hi,

On Mon, Jul 30, 2012 at 1:04 PM, Connor Walls
<connor.walls [at] skillpages>wrote:


> So the problem I'm having is purging all variations of the file based on
> "Accept" and "X-Forwarded-Proto" headers, as well as the host. Is this
> something that is even possible? I personally wouldn't have thought so but
> after seeing in the documentation (
> https://www.varnish-cache.org/docs/trunk/tutorial/purging.html): "The
> purge in vcl_miss is necessary to purge all variants in the cases where you
> hit an object, but miss a particular variant." Seems to imply that this
> functionality is present but I can't find any more documentation on how it
> works. Do I need to remove the hash_data() calls in vcl_hash and instead
> include "Accept" and "X-Forwarded-Proto" in the "Vary" header coming from
> the backend server?
>

Yes. That is the way it is suppose to work.

--
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>

Varnish misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.