Mailing List Archive: Varnish: Misc

retry on 503 -- but not right away please

Index | Next | Previous | View Threaded

janfrode at tanso

Apr 25, 2012, 4:57 AM

Post #1 of 6 (442 views)
Permalink

retry on 503 -- but not right away please

We have a backend apache server that will generate 503 errors on some files
because of a TOCTOU problems while replacing the files. Unfortunately we
see no way of fixing the updates of these files to be atomic, and avoid
the problem -- so I was hoping varnish might be able to solve it for us.

I tried to have varnish retry on 503 using:

=========================================================
sub vcl_error {
# retry on errors
if (obj.status == 503) {
if ( req.restarts < 12 ) {
restart;
}
}
}
=========================================================

but it didn't seem to have the desired effect. We still get the 503's. Maybe
varnish is trying again too quickly.. Is it possible to either insert a
short delay before restarting here? Or possibly send the client a HTTP 307
pointing at the same URL to cause a delay that way?

Any other ideas?

-jf

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

anders at comoyo

Apr 25, 2012, 5:44 AM

Post #2 of 6 (431 views)
Permalink

Re: retry on 503 -- but not right away please [In reply to]

I think you want https://www.varnish-cache.org/trac/wiki/VCLExampleSaintMode
=)

See also
https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html

-@M
--
"Sooner or later we're going to run out of things to go wrong, and
then it'll work great." - Jamie Hyneman, The Mythbusters

janfrode at tanso

Apr 25, 2012, 5:58 AM

Post #3 of 6 (434 views)
Permalink

Re: retry on 503 -- but not right away please [In reply to]

On Wed, Apr 25, 2012 at 02:44:37PM +0200, Anders Daljord Morken wrote:
> I think you want https://www.varnish-cache.org/trac/wiki/VCLExampleSaintMode
> =)

Unfortunately I don't think that helps. I don't have a stale version of
these files when it happens (or the stale version would have been
outdated) and I only have one backend server.

> See also
> https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html

The "Known limitations on grace- and saint mode" on that page says:

"If your request fails while it is being fetched you're thrown into
vcl_error. vcl_error has access to a rather limited set of data so
you can't enable saint mode or grace mode here."

-jf

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Apr 25, 2012, 6:08 AM

Post #4 of 6 (432 views)
Permalink

Re: retry on 503 -- but not right away please [In reply to]

On Wed, Apr 25, 2012 at 01:57:53PM +0200, Jan-Frode Myklebust wrote:
>
> if (obj.status == 503) {
> if ( req.restarts < 12 ) {
> restart;
> }
>
> but it didn't seem to have the desired effect. We still get the 503's. Maybe
> varnish is trying again too quickly.. Is it possible to either insert a
> short delay before restarting here? Or possibly send the client a HTTP 307
> pointing at the same URL to cause a delay that way?

The only effect of the above is probably that you're DoSing the backend :)
And unless you changed the default, max_restarts will kick in before you reach 12.

> Any other ideas?

I don't understand why you're seeing this 503 in vcl_error. A backend
responding with a 503 would just hit the regular fetch->deliver path, not
vcl_error, unless you explicitly return error somewhere in your vcl, so maybe
your initial analysis of the problem is wrong?

I think my preferred solution would be to return some synthetic page with meta
refresh (and that would require you to use error, since that's the only place
you can do synthetic currently).

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

janfrode at tanso

Apr 25, 2012, 6:38 AM

Post #5 of 6 (427 views)
Permalink

Re: retry on 503 -- but not right away please [In reply to]

On Wed, Apr 25, 2012 at 03:08:35PM +0200, Andreas Plesner Jacobsen wrote:
> >
>
> The only effect of the above is probably that you're DoSing the backend :)

It's been happening only a few times a day (eight 503's on a cache that
server 1.7 million requests yesterday), so I think that small DoS might
be acceptable :-)

> And unless you changed the default, max_restarts will kick in before you reach 12.

Oh.. didn't know about that one...

>
> > Any other ideas?
>
> I don't understand why you're seeing this 503 in vcl_error. A backend
> responding with a 503 would just hit the regular fetch->deliver path, not
> vcl_error, unless you explicitly return error somewhere in your vcl, so maybe
> your initial analysis of the problem is wrong?

I'm uncertain where the error is coming from (backend or varnish), but
I think I understand why we get it. The file served is PUT on the backend apache
server using webdav (mod_dav), so every now and then varnish will
request a file that is in the process of being written. Then f.ex. the
file length will change between apache stat'ing it, and reading it -- so
apache or varnish will bail out with a 503..

> I think my preferred solution would be to return some synthetic page with meta
> refresh (and that would require you to use error, since that's the only place
> you can do synthetic currently).

It's not html-files being served, so I don't know if the client will
understand meta refresh. I'm wondering if maybe we need to solve this on
the backend then.. write the files to a temp-name and use incron to move
them to the right place (we don't have much control over the client
writing the files).

-jf

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

apj at mutt

Apr 25, 2012, 7:11 AM

Post #6 of 6 (437 views)
Permalink

Re: retry on 503 -- but not right away please [In reply to]

On Wed, Apr 25, 2012 at 03:38:13PM +0200, Jan-Frode Myklebust wrote:

> > I don't understand why you're seeing this 503 in vcl_error. A backend
> > responding with a 503 would just hit the regular fetch->deliver path, not
> > vcl_error, unless you explicitly return error somewhere in your vcl, so maybe
> > your initial analysis of the problem is wrong?
>
> I'm uncertain where the error is coming from (backend or varnish), but

Please investigate using varnishlog.
Something like varnishlog -m TxStatus:503

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads