Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Misc
"not cacheable" even on static files
 

Index | Next | Previous | View Flat


sufehmi at gmail

Jan 12, 2012, 1:37 PM


Views: 590
Permalink
"not cacheable" even on static files

Hi folks, for the past few days I've been puzzled by this problem -
Varnish have refused to cache even static files. While it was fine
before, and will cache even dynamic HTML files (with proper cache
headers, of course)

The problem begun when my client reported that his website was
constantly going up & down. When I checked, it was under some sort of
DDoS attack. This is not news, because his website was pretty much
under 24x7 syn flood attack.
However, this attack is now able to bypass HAproxy & Varnish - and hit
Apache directly, right where it hurts most.

To cut the long story short - basically now Varnish refuses to cache
almost everything.
The X-Cacheable header that I enabled contains the dreaded "NO:Not
Cacheable" status. And I've not been able to find out why.

This is where I stuck, if I can find out the cause, the I'd be able to
rectify it.

Anyway, the DDoS attack may not be related at all to the current
problem (varnish not caching), I included the story just for the sake
of completeness.

Let's proceed to some facts that I've gathered :

Same GIF file - but different sizes (note the "Received" column) :
http://minus.com/mbawzSZUxJ#3

The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy.

HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4

HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2

List of cookies sent by browser & received from Varnish :
http://minus.com/mbawzSZUxJ#1
(stripped clean by varnish, basically)

Please find the result of varnishstat -1 & content of
/etc/varnish/default.vcl attached to the end of this email.

Varnish is running with the following parameters :

$ ps aux|grep varnish
myuser 3327 0.0 0.0 3324 796 pts/6 S+ 04:36 0:00 grep varnish
root 8441 0.0 1.0 86024 83532 pts/2 SL+ Jan12 0:01
/usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f
/etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s
malloc,2G -d


Hopefully someone will be able to point / give me a hint to the right direction.


Thanks,
Harry

===================

$ telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
200 199
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit

Type 'help' for command list.
Type 'quit' to close CLI session.

stats
200 1978
16564302 Client connections accepted
16514647 Client requests received
2378064 Cache hits
988836 Cache misses
592750 Backend conn. success
462880 Backend conn. failures
13096234 Backend conn. reuses
63305 Backend conn. was closed
13160380 Backend conn. recycles
82 Fetch head
9937334 Fetch with Length
1856 Fetch chunked
245 Fetch wanted close
66 Fetch failed
26436 N struct sess_mem
26091 N struct sess
18773 N struct object
18862 N struct objectcore
6827 N struct objecthead
3 N struct vbe_conn
107 N worker threads
26496 N worker threads created
144188 N overflowed work requests
1 N backends
932755 N expired objects
1851315 N LRU moved objects
12437230 Objects sent with write
16564302 Total Sessions
16514647 Total Requests
870 Total pipe
13146877 Total pass
13630039 Total fetch
6464791313 Total header bytes
122615308544 Total body bytes
16564289 Session Closed
11 Session Linger
17 Session herd
1364697125 SHM records
93867958 SHM writes
2574 SHM flushes due to overflow
446883 SHM MTX contention
540 SHM cycles through buffer
10815252 SMA allocator requests
37554 SMA outstanding allocations
45497812 SMA outstanding bytes
114575871831 SMA bytes allocated
114530374019 SMA bytes free
505674 SMS allocator requests
211877406 SMS bytes allocated
211877406 SMS bytes freed
13688813 Backend requests made
1 N vcl total
1 N vcl available
1 N total active purges
1 N new purges added
3365986 HCB Lookups without lock
122890 HCB Lookups with lock
122890 HCB Inserts
99485 Client uptime
16426 Backend conn. retry
3690588 Fetch no body (304)

=================== default.vcl ===============

$ cat /etc/varnish/default.vcl

# Default backend definition. Set this to point to your content
# server.
backend default {
.host = "127.0.0.1";
.port = "81";
}


sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}

# Below is a commented-out copy of the default VCL logic. If you
# redefine any of these subroutines, the built-in logic will be
# appended to your code.
#
sub vcl_recv {

if (req.backend.healthy) {
set req.grace = 180s;
} else {
set req.grace = 1h;
}
}



sub vcl_fetch {

set beresp.grace = 1h;

# Varnish determined the object was not cacheable
if (!beresp.cacheable) {
set beresp.http.X-Cacheable = "NO:Not Cacheable";

# You don't wish to cache content for logged in users
} elsif (req.http.Cookie ~ "(UserID|_session)") {
set beresp.http.X-Cacheable = "NO:Got Session";
return(pass);

# You are respecting the Cache-Control=private header from the backend
} elsif (beresp.http.Cache-Control ~ "private") {
set beresp.http.X-Cacheable = "NO:Cache-Control=private";
return(pass);

# You are extending the lifetime of the object artificially
} elsif (beresp.ttl < 1s) {
set beresp.ttl = 5s;
set beresp.grace = 5s;
set beresp.http.X-Cacheable = "YES:FORCED";

# Varnish determined the object was cacheable
} else {
set beresp.http.X-Cacheable = "YES";
}

if (req.url ~
"\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$")
{
unset beresp.http.set-cookie;
set beresp.ttl = 24h;
}

return(deliver);

# if (!beresp.cacheable) {
# return (pass);
# }
# if (beresp.http.Set-Cookie) {
# return (pass);
# }
# return (deliver);
}

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Subject User Time
"not cacheable" even on static files sufehmi at gmail Jan 12, 2012, 1:37 PM
    Re: "not cacheable" even on static files bedis9 at gmail Jan 13, 2012, 12:07 AM

  Index | Next | Previous | View Flat
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.