Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Dev

varnishncsa outage

 

 

Varnish dev RSS feed   Index | Next | Previous | View Threaded


jbq at caraldi

Sep 2, 2011, 5:25 AM

Post #1 of 11 (522 views)
Permalink
varnishncsa outage

Hello list,

We are running varnish 3.0.1 rc1 in production since a few days, and
notice that varnishncsa sometimes disappears! It runs for hours and
suddenly exits without an explanation. The only exit() calls I can
find in the code are when the format string is invalid, which would
only occur on start.

No core dump, no kernel message. I don't know if varnishncsa printed
an error because when run as a daemon there is no error log (maybe
using syslog for this would be more appropriate BTW).

Your help will be appreciated,
--
Jean-Baptiste Quenot

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


thierry.magnien at sfr

Sep 2, 2011, 5:39 AM

Post #2 of 11 (523 views)
Permalink
RE: varnishncsa outage [In reply to]

Hi,

We experience the same thing with varnish 3.0.1. I left a varnishncsa running with gdb attached to it and I'm still waiting for it to stop. I hope I'll see something when it does.

Regards,
Thierry

-----Message d'origine-----
De : varnish-dev-bounces [at] varnish-cache [mailto:varnish-dev-bounces [at] varnish-cache] De la part de Jean-Baptiste Quenot
Envoyé : vendredi 2 septembre 2011 14:25
À : varnish-dev [at] varnish-cache
Objet : varnishncsa outage

Hello list,

We are running varnish 3.0.1 rc1 in production since a few days, and
notice that varnishncsa sometimes disappears! It runs for hours and
suddenly exits without an explanation. The only exit() calls I can
find in the code are when the format string is invalid, which would
only occur on start.

No core dump, no kernel message. I don't know if varnishncsa printed
an error because when run as a daemon there is no error log (maybe
using syslog for this would be more appropriate BTW).

Your help will be appreciated,
--
Jean-Baptiste Quenot

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


jbq at caraldi

Sep 2, 2011, 8:40 AM

Post #3 of 11 (515 views)
Permalink
Re: varnishncsa outage [In reply to]

It turns to be a good old "segmentation fault" crash, caused by the
SLT_ReqEnd tag processing "time to first byte".

Here is the gdb backtrace:

https://gist.github.com/1188943
--
Jean-Baptiste Quenot

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


tfheen at varnish-software

Sep 4, 2011, 11:58 PM

Post #4 of 11 (526 views)
Permalink
Re: varnishncsa outage [In reply to]

]] Jean-Baptiste Quenot

| It turns to be a good old "segmentation fault" crash, caused by the
| SLT_ReqEnd tag processing "time to first byte".

No, it's not, it's from:

case 'm':
VSB_cat(os, lp->df_m);
break;

now why lp->df_m, which is the request method is null, I don't know.

--
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


jbq at caraldi

Sep 6, 2011, 1:18 AM

Post #5 of 11 (511 views)
Permalink
Re: varnishncsa outage [In reply to]

Here is more information about this crash:

(gdb) up
(gdb) print *lp
$1 = {df_H = 0x0, df_U = 0x0, df_q = 0x0, df_b = 0x0, df_h = 0x11e56a0
"84.103.222.181", df_m = 0x0, df_s = 0x0, df_t = {tm_sec = 37, tm_min
= 30, tm_hour = 17, tm_mday = 2, tm_mon = 8, tm_year = 111, tm_wday =
5, tm_yday = 244, tm_isdst = 1, tm_gmtoff = 7200,
tm_zone = 0x11bca30 "CEST"}, df_u = 0x0, df_ttfb = 0x11dbc00
"nan", df_hitmiss = 0x0, df_handling = 0x0, active = 1, complete = 1,
bitmap = 0, req_headers = {vtqh_first = 0x0, vtqh_last = 0x0},
resp_headers = {vtqh_first = 0x0, vtqh_last = 0x0}}

Please find attached a patch that checks for mandatory fields.
Curiously the code was there but disabled.

2011/9/5 Tollef Fog Heen <tfheen [at] varnish-software>:
> ]] Jean-Baptiste Quenot
>
> | It turns to be a good old "segmentation fault" crash, caused by the
> | SLT_ReqEnd tag processing "time to first byte".
>
> No, it's not, it's from:
>
>                case 'm':
>                        VSB_cat(os, lp->df_m);
>                        break;
>
> now why lp->df_m, which is the request method is null, I don't know.
>
> --
> Tollef Fog Heen
> Varnish Software
> t: +47 21 98 92 64
>



--
Jean-Baptiste Quenot
Attachments: Varnish_3.0.1-Skip-log-line-when-a-mandatory-field-is-missing.patch (0.84 KB)


jbq at caraldi

Sep 6, 2011, 1:21 AM

Post #6 of 11 (504 views)
Permalink
Re: varnishncsa outage [In reply to]

I notice another crash with varnishncsa, related to header processing:

https://gist.github.com/1196892

AFAICT this is fixed in last commits "Ignore invalid HTTP headers"
from Andreas Plesner Jacobsen in Varnish master branch. Can someone
confirm?
--
Jean-Baptiste Quenot

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


apj at mutt

Sep 6, 2011, 1:26 AM

Post #7 of 11 (499 views)
Permalink
Re: varnishncsa outage [In reply to]

On Tue, Sep 06, 2011 at 10:21:22AM +0200, Jean-Baptiste Quenot wrote:

> I notice another crash with varnishncsa, related to header processing:
>
> https://gist.github.com/1196892
>
> AFAICT this is fixed in last commits "Ignore invalid HTTP headers"
> from Andreas Plesner Jacobsen in Varnish master branch. Can someone
> confirm?

Looks like the one I fixed, yes.

--
Andreas

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


tfheen at varnish-software

Sep 6, 2011, 2:41 AM

Post #8 of 11 (501 views)
Permalink
Re: varnishncsa outage [In reply to]

]] Jean-Baptiste Quenot

| Here is more information about this crash:
|
| (gdb) up
| (gdb) print *lp
| $1 = {df_H = 0x0, df_U = 0x0, df_q = 0x0, df_b = 0x0, df_h = 0x11e56a0
| "84.103.222.181", df_m = 0x0, df_s = 0x0, df_t = {tm_sec = 37, tm_min
| = 30, tm_hour = 17, tm_mday = 2, tm_mon = 8, tm_year = 111, tm_wday =
| 5, tm_yday = 244, tm_isdst = 1, tm_gmtoff = 7200,
| tm_zone = 0x11bca30 "CEST"}, df_u = 0x0, df_ttfb = 0x11dbc00
| "nan", df_hitmiss = 0x0, df_handling = 0x0, active = 1, complete = 1,
| bitmap = 0, req_headers = {vtqh_first = 0x0, vtqh_last = 0x0},
| resp_headers = {vtqh_first = 0x0, vtqh_last = 0x0}}

Can you please capture varnishlog from a request which causes the crash?
The backtrace above does unfortunately not help me.

| Please find attached a patch that checks for mandatory fields.
| Curiously the code was there but disabled.

That code isn't correct with user-specifiable fields, so it should
probably just be removed.

Regards,
--
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


jbq at caraldi

Sep 7, 2011, 12:33 PM

Post #9 of 11 (501 views)
Permalink
Re: varnishncsa outage [In reply to]

2011/9/6 Tollef Fog Heen <tfheen [at] varnish-software>:
>
> Can you please capture varnishlog from a request which causes the crash?
> The backtrace above does unfortunately not help me.

I'm sorry I can't afford to crash varnishncsa anymore. My boss is
getting angry with holes in the monitoring graphs.

> | Please find attached a patch that checks for mandatory fields.
> | Curiously the code was there but disabled.
>
> That code isn't correct with user-specifiable fields, so it should
> probably just be removed.

What do you mean with user-specifiable fields? Since I did that
change (and the broken header patch) varnishncsa did not crash at all.

All the best,
--
Jean-Baptiste Quenot

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


tfheen at varnish-software

Sep 8, 2011, 12:11 AM

Post #10 of 11 (492 views)
Permalink
Re: varnishncsa outage [In reply to]

]] Jean-Baptiste Quenot

| 2011/9/6 Tollef Fog Heen <tfheen [at] varnish-software>:
| >
| > Can you please capture varnishlog from a request which causes the crash?
| > The backtrace above does unfortunately not help me.
|
| I'm sorry I can't afford to crash varnishncsa anymore. My boss is
| getting angry with holes in the monitoring graphs.

You can run two varnishncsas side by side, one patched and one
unpatched.

| > | Please find attached a patch that checks for mandatory fields.
| > | Curiously the code was there but disabled.
| >
| > That code isn't correct with user-specifiable fields, so it should
| > probably just be removed.
|
| What do you mean with user-specifiable fields? Since I did that
| change (and the broken header patch) varnishncsa did not crash at all.

Using the -F format.

--
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


jbq at caraldi

Sep 8, 2011, 12:58 AM

Post #11 of 11 (490 views)
Permalink
Re: varnishncsa outage [In reply to]

2011/9/8 Tollef Fog Heen <tfheen [at] varnish-software>:
> | > That code isn't correct with user-specifiable fields, so it should
> | > probably just be removed.
> |
> | What do you mean with user-specifiable fields?  Since I did that
> | change (and the broken header patch) varnishncsa did not crash at all.
>
> Using the -F format.

I do use the -F command-line option. Usually when user-specified
fields are missing, it is replaced with a hyphen: "-". This is fine,
except for very important fields like request method, protocol
version, path and status, hence the verification.

So I gave a try, ran telnet on varnish http port and typed a slash
followed by two newlines. Here is the result:

12 SessionOpen c 127.0.0.1 57867 :80
12 Debug c herding
12 SessionClose c junk
12 ReqStart c 127.0.0.1 57867 1750343256
12 HttpGarbage c /


12 ReqEnd c 1750343256 1315468371.652468920
1315468371.652515173 0.640084982 nan nan
12 StatSess c 127.0.0.1 57867 1 1 1 0 0 0 0 0

As you can see there is no RxRequest tag so df_m is not set.
Varnishncsa crashed, as expected.
--
Jean-Baptiste Quenot

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Varnish dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.