Mailing List Archive: Varnish: Bugs

#1051: child process died

Index | Next | Previous | View Threaded

varnish-bugs at varnish-cache

Nov 5, 2011, 6:28 AM

Post #1 of 8 (149 views)
Permalink

#1051: child process died

#1051: child process died
---------------------+------------------------------------------------------
Reporter: sreniaw | Type: defect
Status: new | Priority: normal
Milestone: | Component: varnishd
Version: 3.0.2 | Severity: normal
Keywords: |
---------------------+------------------------------------------------------
I am having problem with varnish 64bit on centos 5.5.
Child process died. New process is not respawning.

Can you help ?

{{{
Nov 5 13:37:41 blade414 Condition((const void*)(o) >= (const
void*)((sg->sc)->base) && (const void*)(o) < (const void *)((sg->sc)->base
+ (sg->sc)->mediasize)) not true.
Nov 5 13:37:41 blade414 thread = (cache-worker)
Nov 5 13:37:41 blade414 ident =
Linux,2.6.18-238.el5,x86_64,-smalloc,-spersistent,-smalloc,-hclassic,epoll
Nov 5 13:37:41 blade414 Backtrace:
Nov 5 13:37:41 blade414 0x42c7a6: /usr/sbin/varnishd [0x42c7a6]
Nov 5 13:37:41 blade414 0x448edc: /usr/sbin/varnishd [0x448edc]
Nov 5 13:37:41 blade414 0x425873: /usr/sbin/varnishd(HSH_Lookup+0x3a3)
[0x425873]
Nov 5 13:37:41 blade414 0x41423e: /usr/sbin/varnishd [0x41423e]
Nov 5 13:37:41 blade414 0x417a92: /usr/sbin/varnishd(CNT_Session+0x9d2)
[0x417a92]
Nov 5 13:37:41 blade414 0x42efb8: /usr/sbin/varnishd [0x42efb8]
Nov 5 13:37:41 blade414 0x42e19b: /usr/sbin/varnishd [0x42e19b]
Nov 5 13:37:41 blade414 0x38be20673d: /lib64/libpthread.so.0
[0x38be20673d]
Nov 5 13:37:41 blade414 0x38bdad3f6d: /lib64/libc.so.6(clone+0x6d)
[0x38bdad3f6d]
Nov 5 13:37:41 blade414 sp = 0x2aaf85cbd008 {
Nov 5 13:37:41 blade414 fd = 1692, id = 1692, xid = 518917304,
Nov 5 13:37:41 blade414 client = 1.1.1.1 32755,
Nov 5 13:37:41 blade414 step = STP_LOOKUP,
Nov 5 13:37:41 blade414 handling = hash,
Nov 5 13:37:41 blade414 restarts = 0, esi_level = 0
Nov 5 13:37:41 blade414 flags =
Nov 5 13:37:41 blade414 bodystatus = 4
Nov 5 13:37:41 blade414 ws = 0x2aaf85cbd080 {
Nov 5 13:37:41 blade414 id = "sess",
Nov 5 13:37:41 blade414 {s,f,r,e} = {0x2aaf85cbdc90,+504,+65536,+65536},
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 http[req] = {
Nov 5 13:37:41 blade414 ws = 0x2aaf85cbd080[sess]
Nov 5 13:37:41 blade414 "GET",
Nov 5 13:37:41 blade414 "/ph/az/22/22/az222289.jpg",
Nov 5 13:37:41 blade414 "HTTP/1.1",
Nov 5 13:37:41 blade414 "User-Agent: Mozilla/5.0 (Windows NT 5.1;
rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
Nov 5 13:37:41 blade414 "Accept: image/png,image/*;q=0.8,*/*;q=0.5",
Nov 5 13:37:41 blade414 "Accept-Language: pl,en-us;q=0.7,en;q=0.3",
Nov 5 13:37:41 blade414 "Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7",
Nov 5 13:37:41 blade414 "Connection: keep-alive",
Nov 5 13:37:41 blade414 "host: static",
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 worker = 0x2aaf4ce0ccf0 {
Nov 5 13:37:41 blade414 ws = 0x2aaf4ce0cf30 {
Nov 5 13:37:41 blade414 id = "wrk",
Nov 5 13:37:41 blade414 {s,f,r,e} =
{0x2aaf4cdfaca0,0x2aaf4cdfaca0,(nil),+65536},
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 vcl = {
Nov 5 13:37:41 blade414 srcname = {
Nov 5 13:37:41 blade414 "input",
Nov 5 13:37:41 blade414 "Default",
Nov 5 13:37:41 blade414 "/etc/varnish/url.vcl",
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 },
Nov 5 13:37:41 blade414 CLI communication error (hdr)
Nov 5 13:37:41 blade414 Condition(sg1->p.offset != sg->p.offset) not
true.
Nov 5 13:37:41 blade414 thread = (cache-main)
Nov 5 13:37:41 blade414 ident =
Linux,2.6.18-238.el5,x86_64,-smalloc,-spersistent,-smalloc,-hclassic,no_waiter
Nov 5 13:37:41 blade414 Backtrace:
Nov 5 13:37:41 blade414 0x42c7a6: /usr/sbin/varnishd [0x42c7a6]
Nov 5 13:37:41 blade414 0x4472f5: /usr/sbin/varnishd [0x4472f5]
Nov 5 13:37:41 blade414 0x4474bb: /usr/sbin/varnishd [0x4474bb]
Nov 5 13:37:41 blade414 0x444d57: /usr/sbin/varnishd(STV_open+0x27)
[0x444d57]
Nov 5 13:37:41 blade414 0x42b525: /usr/sbin/varnishd(child_main+0xc5)
[0x42b525]
Nov 5 13:37:41 blade414 0x43d5ec: /usr/sbin/varnishd [0x43d5ec]
Nov 5 13:37:41 blade414 0x43de7c: /usr/sbin/varnishd [0x43de7c]
Nov 5 13:37:41 blade414 0x3f88c094c7: /usr/lib64/varnish/libvarnish.so
[0x3f88c094c7]
Nov 5 13:37:41 blade414 0x3f88c09b58:
/usr/lib64/varnish/libvarnish.so(vev_schedule+0x88) [0x3f88c09b58]
Nov 5 13:37:41 blade414 0x43d7c2: /usr/sbin/varnishd(MGT_Run+0x132)
[0x43d7c2]

}}}

{{{
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 28818 0.0 0.0 31570672 1296 ? Ss Oct31 0:00
/usr/sbin/varnishd
}}}

{{{
-P /var/run/varnish.pid
-f /etc/varnish/default.vcl
-p lru_interval 20
-p cli_timeout 60
-h classic,500009
-s malloc,30G
-s persistent,/var/lib/varnish/store/silo,30G
-p thread_pool_min 500
-p thread_pool_max 4000
-p thread_pools 4
-p thread_pool_add_delay 2
-p session_linger 100
-p send_timeout 120
-p sess_timeout 10
-p listen_depth 4096
-p nuke_limit 50
}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1051>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs

varnish-bugs at varnish-cache

Nov 5, 2011, 6:58 AM

Post #2 of 8 (148 views)
Permalink

Re: #1051: child process died [In reply to]

varnish-bugs at varnish-cache

Nov 7, 2011, 3:13 AM

Post #3 of 8 (150 views)
Permalink

Re: #1051: child process died [In reply to]

varnish-bugs at varnish-cache

Jan 17, 2012, 7:16 AM

Post #4 of 8 (108 views)
Permalink

Re: #1051: child process died [In reply to]

#1051: child process died
-----------------------+----------------------------------------------------
Reporter: sreniaw | Type: defect
Status: reopened | Priority: normal
Milestone: | Component: varnishd
Version: 3.0.2 | Severity: normal
Resolution: | Keywords:
-----------------------+----------------------------------------------------
Changes (by sreniaw2):

* status: closed => reopened
* resolution: fixed =>

Comment:

I am having problem with persistent cache.
Silo file is created using dd command with initial size of 35GBs.
After two weeks varnish is starting crashing.

I am using varnish-3.0.2-1.el5 on Centos Linux 5 x86_64 with 32 GB of
physical memory and 33 GB of swap.

dm-7 is only for silo file
{{{
# iostat -x 1
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
dm-7 0.00 0.00 1.29 163.92 111.77 1311.38 8.61
0.60 3.64 1.29 21.30
dm-7 0.00 0.00 0.00 662.00 0.00 5296.00 8.00
113.08 171.04 1.51 100.10
dm-7 0.00 0.00 2.00 646.00 256.00 5168.00 8.37
116.20 177.84 1.54 100.10
dm-7 0.00 0.00 0.00 647.52 0.00 5180.20 8.00
113.79 182.72 1.53 99.11
dm-7 0.00 0.00 4.00 640.00 312.00 5120.00 8.43
94.62 131.55 1.55 100.10
dm-7 0.00 0.00 2.00 620.00 216.00 4960.00 8.32
118.29 202.90 1.61 100.20
dm-7 0.00 0.00 4.00 645.00 416.00 5160.00 8.59
114.23 179.15 1.54 100.10
dm-7 0.00 0.00 0.00 611.00 0.00 4888.00 8.00
112.30 164.37 1.64 100.10
dm-7 0.00 0.00 3.00 581.00 424.00 4648.00 8.68
113.39 209.71 1.71 100.00
dm-7 0.00 0.00 4.00 594.00 512.00 4752.00 8.80
91.58 168.89 1.67 100.10
dm-7 0.00 0.00 2.00 708.00 112.00 5664.00 8.14
115.57 158.32 1.41 100.10
dm-7 0.00 0.00 4.00 658.00 376.00 5264.00 8.52
115.12 166.39 1.51 100.10
dm-7 0.00 0.00 2.00 527.00 256.00 4216.00 8.45
114.82 213.90 1.89 100.10
dm-7 0.00 0.00 0.00 680.00 0.00 5440.00 8.00
115.64 171.43 1.47 100.20
dm-7 0.00 0.00 19.80 599.01 633.66 4792.08 8.77
101.46 172.11 1.60 99.21
dm-7 0.00 0.00 1.00 664.00 128.00 5312.00 8.18
115.89 167.74 1.51 100.20
dm-7 0.00 0.00 5.00 622.00 360.00 4976.00 8.51
114.20 172.12 1.60 100.10
dm-7 0.00 0.00 3.00 582.00 224.00 4656.00 8.34
112.75 205.01 1.71 100.10
dm-7 0.00 0.00 0.00 667.00 0.00 5336.00 8.00
110.22 164.68 1.50 100.20
dm-7 0.00 0.00 1.00 669.00 128.00 5352.00 8.18
103.50 155.61 1.50 100.20
dm-7 0.00 0.00 3.00 596.00 240.00 4768.00 8.36
113.84 184.50 1.67 100.10
dm-7 0.00 0.00 10.89 564.36 807.92 4514.85 9.25
114.11 203.60 1.72 99.21
dm-7 0.00 0.00 4.00 617.00 264.00 4936.00 8.37
116.19 184.58 1.61 100.10
dm-7 0.00 0.00 3.00 607.00 224.00 4856.00 8.33
112.08 186.60 1.64 100.00
dm-7 0.00 0.00 0.00 698.00 0.00 5584.00 8.00
113.24 145.28 1.44 100.20
dm-7 0.00 0.00 0.00 6062.00 0.00 48496.00 8.00
1007.00 106.46 0.17 100.10
dm-7 0.00 0.00 0.00 750.00 0.00 6000.00 8.00
268.74 834.84 1.34 100.20
dm-7 0.00 0.00 0.00 656.00 0.00 5248.00 8.00
115.95 209.27 1.52 100.00
dm-7 0.00 0.00 0.00 730.00 0.00 5840.00 8.00
152.54 187.88 1.37 100.20
dm-7 0.00 0.00 15.84 55.45 2146.53 443.56 36.33
21.85 652.62 6.14 43.76
dm-7 0.00 0.00 4.00 0.00 512.00 0.00 128.00
0.03 6.75 6.75 2.70
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-7 0.00 0.00 3.00 0.00 288.00 0.00 96.00
0.00 1.33 1.33 0.40
dm-7 0.00 0.00 3.00 456.00 384.00 3648.00 8.78
38.76 64.82 0.66 30.40
dm-7 0.00 0.00 0.00 991.00 0.00 7928.00 8.00
119.20 120.89 1.01 100.20
dm-7 0.00 0.00 2.00 945.00 112.00 7560.00 8.10
114.86 114.68 1.06 100.20
dm-7 0.00 0.00 4.00 834.00 216.00 6672.00 8.22
118.92 138.72 1.19 100.10
dm-7 0.00 0.00 2.97 743.56 205.94 5948.51 8.24
117.64 156.17 1.33 99.21
dm-7 0.00 0.00 11.00 681.00 592.00 5448.00 8.73
117.81 176.87 1.45 100.00
}}}

{{{
# ps axu | grep varnishd
root 6462 0.0 0.0 34716380 1392 ? Ss Jan02 0:00
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -T localhost:6082 -f
/etc/varnish/default.vcl -u varnish
+-g varnish -S /etc/varnish/secret -p lru_interval 20 -p cli_timeout 60 -h
classic,500009 -s malloc,30G -s
persistent,/var/lib/varnish/store/silo,35433480192 -p
+thread_pool_min 500 -p thread_pool_max 4000 -p thread_pools 4 -p
thread_pool_add_delay 2 -p session_linger 100 -p send_timeout 120 -p
sess_timeout 10 -p listen_depth 4096
+-p nuke_limit 50
varnish 29817 25.9 0.0 0 0 ? Zl 22:29 4:48
[varnishd] <defunct>
}}}

{{{
# ps axu | grep varnishd
varnish 1870 25.4 2.4 55950024 808356 ? Sl 22:48 0:09
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -T localhost:6082 -f
/etc/varnish/default.vcl -u varnish
+-g varnish -S /etc/varnish/secret -p lru_interval 20 -p cli_timeout 60 -h
classic,500009 -s malloc,30G -s
persistent,/var/lib/varnish/store/silo,35433480192 -p
+thread_pool_min 500 -p thread_pool_max 4000 -p thread_pools 4 -p
thread_pool_add_delay 2 -p session_linger 100 -p send_timeout 120 -p
sess_timeout 10 -p listen_depth 4096
+-p nuke_limit 50
root 3969 0.0 0.0 61172 728 pts/3 R+ 22:48 0:00 grep
varnishd
root 6462 0.0 0.0 34716380 1400 ? Ss Jan02 0:00
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -T localhost:6082 -f
/etc/varnish/default.vcl -u varnish
+-g varnish -S /etc/varnish/secret -p lru_interval 20 -p cli_timeout 60 -h
classic,500009 -s malloc,30G -s
persistent,/var/lib/varnish/store/silo,35433480192 -p
+thread_pool_min 500 -p thread_pool_max 4000 -p thread_pools 4 -p
thread_pool_add_delay 2 -p session_linger 100 -p send_timeout 120 -p
sess_timeout 10 -p listen_depth 4096
+-p nuke_limit 50
}}}

Next day:
{{{
# ps axu | grep varnishd
root 6462 0.0 0.0 34716380 1524 ? Ss Jan02 0:00
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -T localhost:6082 -f
/etc/varnish/default.vcl -u varnish -g varnish -S /etc/varnish/secret -p
lru_interval 20 -p cli_timeout 60 -h classic,500009 -s malloc,30G -s
persistent,/var/lib/varnish/store/silo,35433480192 -p thread_pool_min 500
-p thread_pool_max 4000 -p thread_pools 4 -p thread_pool_add_delay 2 -p
session_linger 100 -p send_timeout 120 -p sess_timeout 10 -p listen_depth
4096 -p nuke_limit 50
varnish 14624 11.4 17.9 56139464 5921276 ? Sl 15:14 4:29
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -T localhost:6082 -f
/etc/varnish/default.vcl -u varnish -g varnish -S /etc/varnish/secret -p
lru_interval 20 -p cli_timeout 60 -h classic,500009 -s malloc,30G -s
persistent,/var/lib/varnish/store/silo,35433480192 -p thread_pool_min 500
-p thread_pool_max 4000 -p thread_pools 4 -p thread_pool_add_delay 2 -p
session_linger 100 -p send_timeout 120 -p sess_timeout 10 -p listen_depth
4096 -p nuke_limit 50
}}}

{{{
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-u varnish -g varnish \
-S /etc/varnish/secret \
-p lru_interval=20 \
-p cli_timeout=60 \
-h classic,500009 \
-s malloc,30G -s
persistent,/var/lib/varnish/store/silo,35433480192 \
-p thread_pool_min=500 \
-p thread_pool_max=4000 \
-p thread_pools=4 \
-p thread_pool_add_delay=2 \
-p session_linger=100 \
-p send_timeout=120 \
-p sess_timeout=10 \
-p listen_depth=4096 \
-p nuke_limit=50"
}}}

Any clues ?

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1051#comment:3>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs

varnish-bugs at varnish-cache

Jan 17, 2012, 7:36 AM

Post #5 of 8 (108 views)
Permalink

Re: #1051: child process died [In reply to]

varnish-bugs at varnish-cache

Jan 17, 2012, 7:37 AM

Post #6 of 8 (112 views)
Permalink

Re: #1051: child process died [In reply to]

varnish-bugs at varnish-cache

Jan 23, 2012, 5:28 AM

Post #7 of 8 (109 views)
Permalink

Re: #1051: child process died [In reply to]

#1051: child process died
----------------------+-----------------------------------------------------
Reporter: sreniaw | Type: defect
Status: closed | Priority: normal
Milestone: | Component: varnishd
Version: 3.0.2 | Severity: normal
Resolution: fixed | Keywords:
----------------------+-----------------------------------------------------
Changes (by kristian):

* status: reopened => closed
* resolution: => fixed

Comment:

@sreniaw2:

What you are experiencing is running out of space, and persistent storage
currently has no code to handle it, therefor it restarts.

The fix PHK committed fixed corruption, which made restarting problematic,
but does not solve the lack of LRU, which is distinct and a feature
request.

The only remedy for this is to make sure you do not run out of space.
Either by caching content for a shorter duration or by increasing the
available space.

As this is a known limitation of persistent storage, and the correct code
to fix it is a feature request, I'm closing this. This does not indicate
we're not planning to fix it, just that we do not track feature requests
in trac.

I'm closing this as "fixed" because the corruption and origional assert
should have been fixed. The LRU-thing is not.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1051#comment:6>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs

varnish-bugs at varnish-cache

Mar 15, 2012, 7:46 AM

Post #8 of 8 (61 views)
Permalink

Re: #1051: child process died [In reply to]

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads