Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Misc

"Large" Binary Files triggering 503 Response

 

 

Varnish misc RSS feed   Index | Next | Previous | View Threaded


drtaber at northcarolina

Jan 12, 2012, 2:48 PM

Post #1 of 3 (1230 views)
Permalink
"Large" Binary Files triggering 503 Response

Hi all-

I'm relatively new to varnish, but i haven't seen/stumbled upon a similar situation in the last few months of mailing list archives. I'm having a weird issue where below approximately 200k, varnish returns binary files just fine, but above that it returns a 503 page. According to the log, apache is returning a 200 status code from the backend. I'm not really sure what to make of it, but it looks like the exact error being thrown back is:

11 FetchError c straight read_error: 0 0 (Remote closed connection)

I've included my configs and the associated log segment. Hopefully someone can shed some light on it. The server version information is:
Varnish Server:
RHEL6.2
Varnish 3.0.2 (from the EL5 RPM)

Apache Server:
RHEL6.2
Apache 2.2.x

The varnish command is(from /etc/sysconfig/varnish):
DAEMON_OPTS="-a 80\
-f /etc/varnish/default.vcl \
-T 127.0.0.1:6082 \
-t 120 \
-w 10,5000,120
-u varnish -g varnish \
-S /etc/varnish/secret \
-p thread_pool_add_delay=2 \
-p thread_pools=4 \
-p thread_pool_min=200 \
-p thread_pool_max=4000 \
-p cli_timeout=25 \
-p session_linger=60 \
-p sess_timeout=60 \
-s malloc,10G \
-s $file,/var/lib/varnish/varnish_storage.bin,15G"

The default config is:
backend default {
.host = "10.11.12.1";
.port = "80";
}
include "/etc/varnish/federation.vcl";
include "/etc/varnish/mailwatch.vcl";
include "/etc/varnish/devcollab.vcl";
include "/etc/varnish/www.vcl";
include "/etc/varnish/collab.vcl";
include "/etc/varnish/uncdm.vcl";
include "/etc/varnish/services.vcl";
include "/etc/varnish/devfederation.vcl";
include "/etc/varnish/devservices.vcl";
include "/etc/varnish/devuncdm.vcl";
include "/etc/varnish/devwww.vcl";
sub vcl_recv {
# if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|css|js|gz|tgz|bz2|tbz|mp3|ogg|mp4|flv|f4v|pdf|swf)(\?[a-z0-9]+)?$") {
# unset req.http.Cookie;
# return(lookup);
# }
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
return (lookup);
}

sub vcl_error {
set obj.http.Content-Type = "text/html; charset=utf-8";
synthetic {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<style type='text/css'>

#msg
{
text-align: center;
}
{
text-align: center;
}

</style>
</head>
<body>

<div id='msg'>
<h3>The University of North Carolina General Administration</h3>
<h3>Our websites are currently down for maintenance.<br />
We apologize for the inconvenience.</h3>
</div>

</body>
</html>

"};
return (deliver);
}

The error log is:

11 SessionOpen c 71.77.0.73 49448 :80
11 ReqStart c 71.77.0.73 49448 570029255
11 RxRequest c GET
11 RxURL c /QA/exams/standards_and_guidelines.pdf
11 RxProtocol c HTTP/1.1
11 RxHeader c Host: 152.4.21.240
11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7
11 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
11 RxHeader c Accept-Language: en-us
11 RxHeader c Accept-Encoding: gzip, deflate
11 RxHeader c Cookie: __utma=67627729.194036926.1325818231.1325818231.1326406657.2; __utmb=67627729.5.10.1326406657; __utmc=67627729; __utmz=67627729.1325818231.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=4mvdfvuv1gd9h59m8nqg5gsgs1
11 RxHeader c Connection: keep-alive
11 VCL_call c recv lookup
11 VCL_call c hash
11 Hash c /QA/exams/standards_and_guidelines.pdf
11 Hash c 152.4.21.240
11 VCL_return c hash
11 VCL_call c miss fetch
11 Backend c 4 devservices devservices2
11 TTL c 570029255 RFC 0 -1 -1 1326407133 0 1326407118 375007920 0
11 VCL_call c fetch
11 TTL c 570029255 VCL 302 -1 -1 1326407131 -2
11 VCL_return c deliver
11 ObjProtocol c HTTP/1.1
11 ObjResponse c OK
11 ObjHeader c Date: Thu, 12 Jan 2012 22:25:18 GMT
11 ObjHeader c Server: Apache/2.2.15 (Red Hat)
11 ObjHeader c X-Powered-By: PHP/5.3.3
11 ObjHeader c Set-Cookie: PHPSESSID=af0ivec0rh864s6rifiirs0gu4; path=/
11 ObjHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT
11 ObjHeader c Cache-Control: public, must-revalidate
11 ObjHeader c Pragma: hack
11 ObjHeader c X-Pad: avoid browser bug
11 ObjHeader c Content-Encoding: gzip
11 ObjHeader c Vary: Accept-Encoding
11 ObjHeader c Content-Type: application/pdf
11 FetchError c straight read_error: 0 0 (Remote closed connection)
11 Gzip c u F - 90795 96895 80 726280 726290
11 VCL_call c error deliver
11 VCL_call c deliver deliver
11 TxProtocol c HTTP/1.1
11 TxStatus c 503
11 TxResponse c Service Unavailable
11 TxHeader c Server: Varnish
11 TxHeader c Content-Type: text/html; charset=utf-8
11 TxHeader c Content-Length: 690
11 TxHeader c Accept-Ranges: bytes
11 TxHeader c Date: Thu, 12 Jan 2012 22:25:33 GMT
11 TxHeader c X-Varnish: 570029255
11 TxHeader c Age: 2
11 TxHeader c Via: 1.1 varnish
11 TxHeader c Connection: close
11 Length c 690
11 ReqEnd c 570029255 1326407130.792037964 1326407133.193643808 0.000042439 2.401573896 0.000031948
11 SessionClose c error
11 StatSess c 71.77.0.73 49448 2 1 1 0 0 0 240 690


Thanks in advance if anyone has any insight into this.
_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


esbjorn at bazooka

Apr 11, 2012, 4:26 AM

Post #2 of 3 (1089 views)
Permalink
Re: "Large" Binary Files triggering 503 Response [In reply to]

I got this exact same error. Solved it by doing a check on req.url for the troublesome url and piping the response.

if (req.url ~ "/troublesome/") {
return (pipe);
}

--
Esbjörn Eriksson | http://bazooka.se | 08 31 70 00


_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


drtaber at northcarolina

Apr 13, 2012, 7:08 AM

Post #3 of 3 (1086 views)
Permalink
Re: "Large" Binary Files triggering 503 Response [In reply to]

On Apr 11, 2012, at 7:26 AM, Esbjörn Eriksson wrote:

> I got this exact same error. Solved it by doing a check on req.url for the troublesome url and piping the response.
>
> if (req.url ~ "/troublesome/") {
> return (pipe);
> }
>
> --
> Esbjörn Eriksson | http://bazooka.se | 08 31 70 00
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc [at] varnish-cache
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

We found that our developers were manually setting a value for the content-length header that was causing varnish to 503 on us. They were calculating in code based on the file system size causing it to be incorrect. Removing that manually set header solved the issue.
_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Varnish misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.