Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

rsyslog 6.3.10 segfault on RHEL 6.2

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


suffestuffie at gmail

Jun 11, 2012, 5:35 AM

Post #1 of 8 (367 views)
Permalink
rsyslog 6.3.10 segfault on RHEL 6.2

rsyslog crashes after a few seconds on redhat enterprise linux 6.2.

gdb info:

[root [at] spi tmp]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
Q\:Reg-11-0-0-11100-1339415918
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/rsyslogd...done.
[New Thread 11105]
[New Thread 11106]
[New Thread 11107]
[New Thread 11108]
[New Thread 11109]
[New Thread 11100]
[New Thread 11101]
[New Thread 11102]
[New Thread 11103]
[New Thread 11104]
Missing separate debuginfo for /usr/lib64/libee.so.0
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
Missing separate debuginfo for /usr/lib64/libestr.so.0
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
Missing separate debuginfo for /usr/lib64/rsyslog/ommongodb.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/3c/e13e40b6a06c38813268d1c0ded376c5a0bbe2
Missing separate debuginfo for /usr/lib64/libmongo-client.so.0
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/15/78059d9c73e4553a2d6be2a39fa64c406443e5
Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
/usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
done.
Loaded symbols for /lib64/libz.so.1.2.3
Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
[Thread debugging using libthread_db enabled]
done.
Loaded symbols for /lib64/libpthread-2.12.so
Reading symbols from /lib64/libdl-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libdl-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libdl-2.12.so
Reading symbols from /lib64/librt-2.12.so...Reading symbols from
/usr/lib/debug/lib64/librt-2.12.so.debug...done.
done.
Loaded symbols for /lib64/librt-2.12.so
Reading symbols from /usr/lib64/libee.so.0...done.
Loaded symbols for /usr/lib64/libee.so.0
Reading symbols from /usr/lib64/libestr.so.0...done.
Loaded symbols for /usr/lib64/libestr.so.0
Reading symbols from /lib64/libm-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libm-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libm-2.12.so
Reading symbols from /lib64/libc-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libc-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libc-2.12.so
Reading symbols from /lib64/ld-2.12.so...Reading symbols from
/usr/lib/debug/lib64/ld-2.12.so.debug...done.
done.
Loaded symbols for /lib64/ld-2.12.so
Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmnet.so
Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done.
Loaded symbols for /usr/lib64/rsyslog/imuxsock.so
Reading symbols from /usr/lib64/rsyslog/imklog.so...done.
Loaded symbols for /usr/lib64/rsyslog/imklog.so
Reading symbols from /usr/lib64/rsyslog/imudp.so...done.
Loaded symbols for /usr/lib64/rsyslog/imudp.so
Reading symbols from /usr/lib64/rsyslog/imtcp.so...done.
Loaded symbols for /usr/lib64/rsyslog/imtcp.so
Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so
Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so
Reading symbols from /usr/lib64/rsyslog/ommongodb.so...done.
Loaded symbols for /usr/lib64/rsyslog/ommongodb.so
Reading symbols from /usr/lib64/libmongo-client.so.0...done.
Loaded symbols for /usr/lib64/libmongo-client.so.0
Reading symbols from /lib64/libglib-2.0.so.0.2200.5...Reading symbols from
/usr/lib/debug/lib64/libglib-2.0.so.0.2200.5.debug...done.
done.
Loaded symbols for /lib64/libglib-2.0.so.0.2200.5
Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so
Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files-2.12.so
Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_dns-2.12.so
Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libresolv-2.12.so
Core was generated by `/usr/sbin/rsyslogd -f /etc/rsyslog.conf.v6.mongodb'.
Program terminated with signal 11, Segmentation fault.
#0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at
malloc.c:4879
4879 old_idx = fastbin_index(chunksize(old));
(gdb) info thread
10 Thread 0x7f42b1f36700 (LWP 11104) 0x00007f42b55a12c3 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:82
9 Thread 0x7f42b2937700 (LWP 11103) 0x00007f42b55a12c3 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:82
8 Thread 0x7f42b3338700 (LWP 11102) 0x00007f42b62f64ed in read () at
../sysdeps/unix/syscall-template.S:82
7 Thread 0x7f42b3d39700 (LWP 11101) 0x00007f42b55998b3 in select () at
../sysdeps/unix/syscall-template.S:82
6 Thread 0x7f42b692c700 (LWP 11100) 0x00007f42b55998b3 in select () at
../sysdeps/unix/syscall-template.S:82
5 Thread 0x7f42b6894700 (LWP 11109) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
4 Thread 0x7f42b68c6700 (LWP 11108) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
3 Thread 0x7f42b68f8700 (LWP 11107) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
2 Thread 0x7f42b692a700 (LWP 11106) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
* 1 Thread 0x7f42b1535700 (LWP 11105) _int_free (av=0x7f42a4000020,
p=0x7f42a4007d50, have_lock=1) at malloc.c:4879
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f42b1535700 (LWP 11105))]#0 _int_free
(av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at malloc.c:4879
4879 old_idx = fastbin_index(chunksize(old));
(gdb) bt
#0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at
malloc.c:4879
#1 0x00007f42b5535900 in _int_realloc (av=0x7f42a4000020,
oldp=0x7f42a4007d50, oldsize=<value optimized out>, nb=<value optimized
out>) at malloc.c:5345
#2 0x00007f42b5535af5 in __libc_realloc (oldmem=0x7f42a4007d60, bytes=32)
at malloc.c:3822
#3 0x00007f42b3f81621 in IA__g_realloc (mem=<value optimized out>,
n_bytes=32) at gmem.c:171
#4 0x00007f42b3f542db in g_array_maybe_expand (array=0x7f42a4002e40,
len=<value optimized out>) at garray.c:416
#5 0x00007f42b3f54712 in IA__g_array_append_vals (farray=<value optimized
out>, data=0x7f42a4004550, len=8) at garray.c:207
#6 0x00007f42b3f54769 in IA__g_byte_array_append (array=0x7f42a4002e40,
data=<value optimized out>, len=<value optimized out>) at garray.c:880
#7 0x00007f42b422bc19 in _bson_append_string_element (b=0x7f42a40078b0,
type=<value optimized out>, name=0x7f42b443a1c4 "sys", val=<value optimized
out>, length=<value optimized out>) at bson.c:138
#8 0x00007f42b422d24c in bson_build (type=BSON_TYPE_STRING,
name=0x7f42b443a1c4 "sys") at bson.c:458
#9 0x00007f42b4439cf0 in writeMongoDB_msg (pMsg=<value optimized out>,
pData=0x1dc1550) at ommongodb.c:261
#10 0x0000000000446dd6 in actionCallDoAction (pThis=0x1dc1910,
pMsg=0x7f42a8000df0, actParams=0x1ddca50) at ../action.c:915
#11 0x000000000044737f in actionProcessMessage (pAction=0x1dc1910,
pBatch=0x1dafb80, nElem=2) at ../action.c:962
#12 tryDoAction (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at
../action.c:1057
#13 submitBatch (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at
../action.c:1133
#14 0x0000000000447790 in processAction (pAction=0x1dc1910,
pBatch=0x1dafb80, pbShutdownImmediate=<value optimized out>) at
../action.c:1224
#15 processBatchMain (pAction=0x1dc1910, pBatch=0x1dafb80,
pbShutdownImmediate=<value optimized out>) at ../action.c:1258
#16 0x0000000000444fdf in doQueueEnqObjDirectBatch (pAction=0x1dc1910,
pBatch=0x1dafb80) at ../action.c:1727
#17 0x0000000000445c2b in doSubmitToActionQNotAllMarkBatch
(pAction=0x1dc1910, pBatch=0x1dafb80) at ../action.c:1636
#18 0x0000000000440bf3 in processBatchDoActions (pData=0x1dc1910,
pParam=<value optimized out>) at rule.c:103
#19 0x000000000042aec7 in llExecFunc (pThis=0x1dc12a0, pFunc=0x440bc0
<processBatchDoActions>, pParam=0x1dafb80) at linkedlist.c:389
#20 0x00000000004411f3 in processBatch (pThis=0x1dc1240, pBatch=0x1dafb80)
at rule.c:293
#21 0x000000000043ff22 in processBatchDoRules (pData=0x1dc1240,
pParam=0x1dafb80) at ruleset.c:147
#22 0x000000000042aec7 in llExecFunc (pThis=0x1da3a68, pFunc=0x43fee0
<processBatchDoRules>, pParam=0x1dafb80) at linkedlist.c:389
#23 0x00000000004401fa in processBatch (pBatch=0x1dafb80) at ruleset.c:229
#24 0x000000000040d056 in msgConsumer (notNeeded=<value optimized out>,
pBatch=0x1dafb80, pbShutdownImmediate=0x1dc8eb0) at syslogd.c:608
#25 0x000000000043f42b in ConsumerReg (pThis=0x1dc8e90, pWti=0x1dafb50) at
queue.c:1791
#26 0x000000000043a415 in wtiWorker (pThis=0x1dafb50) at wti.c:313
#27 0x0000000000438efa in wtpWorker (arg=0x1dafb50) at wtp.c:387
#28 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b1535700) at
pthread_create.c:301
#29 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f42b692a700 (LWP 11106))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842da0) at tcpsrv.c:629
#2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b692a700) at
pthread_create.c:301
#3 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f42b68f8700 (LWP 11107))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842e08) at tcpsrv.c:629
#2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b68f8700) at
pthread_create.c:301
#3 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f42b68c6700 (LWP 11108))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842e70) at tcpsrv.c:629
#2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b68c6700) at
pthread_create.c:301
#3 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 5
[Switching to thread 5 (Thread 0x7f42b6894700 (LWP 11109))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842ed8) at tcpsrv.c:629
#2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b6894700) at
pthread_create.c:301
#3 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 6
[Switching to thread 6 (Thread 0x7f42b692c700 (LWP 11100))]#0
0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x000000000040e36d in mainloop (argc=<value optimized out>, argv=<value
optimized out>) at syslogd.c:1314
#2 realMain (argc=<value optimized out>, argv=<value optimized out>) at
syslogd.c:2061
#3 0x00007f42b54d9cdd in __libc_start_main (main=0x40e750 <main>, argc=3,
ubp_av=0x7fff59743d08, init=<value optimized out>, fini=<value optimized
out>, rtld_fini=<value optimized out>, stack_end=0x7fff59743cf8) at
libc-start.c:226
#4 0x000000000040bf89 in _start ()
(gdb) thread 7
[Switching to thread 7 (Thread 0x7f42b3d39700 (LWP 11101))]#0
0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f42b50648bb in runInput (pThrd=<value optimized out>) at
imuxsock.c:1140
#2 0x000000000044857a in thrdStarter (arg=0x1ddd700) at ../threads.c:210
#3 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b3d39700) at
pthread_create.c:301
#4 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 8
[Switching to thread 8 (Thread 0x7f42b3338700 (LWP 11102))]#0
0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f42b4e513d8 in readklog () at bsd.c:224
#2 0x00007f42b4e51509 in klogLogKMsg (pModConf=<value optimized out>) at
bsd.c:282
#3 0x00007f42b4e50b73 in runInput (pThrd=0x1ddd8f0) at imklog.c:271
#4 0x000000000044857a in thrdStarter (arg=0x1ddd8f0) at ../threads.c:210
#5 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b3338700) at
pthread_create.c:301
#6 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 9
[Switching to thread 9 (Thread 0x7f42b2937700 (LWP 11103))]#0
0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f42b55a12c3 in epoll_wait () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f42b4c4d0a8 in rcvMainLoop (pThrd=0x1dddae0) at imudp.c:593
#2 0x000000000044857a in thrdStarter (arg=0x1dddae0) at ../threads.c:210
#3 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b2937700) at
pthread_create.c:301
#4 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 10
[Switching to thread 10 (Thread 0x7f42b1f36700 (LWP 11104))]#0
0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f42b55a12c3 in epoll_wait () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f42b3d3da4f in Wait (pNsdpoll=0x7f42ac000e10, timeout=-1,
numEntries=0x7f42b1f35dbc, workset=0x7f42b1f35170) at nsdpoll_ptcp.c:242
#2 0x00007f42b464038c in Run (pThis=0x1dae7d0) at tcpsrv.c:884
#3 0x000000000044857a in thrdStarter (arg=0x1dddcd0) at ../threads.c:210
#4 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b1f36700) at
pthread_create.c:301
#5 0x00007f42b55a0ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)


If you need more, please let me know, but remember: i am not a C programmer,

Stefan Arts
Holland.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


Thomas.Lau at principleone

Jun 11, 2012, 5:40 AM

Post #2 of 8 (324 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 11, 2012, 5:40 AM

Post #3 of 8 (332 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

Does it work without ommongodb? If so, have you made sure you use the newest version of libmongo-client? I notice the crash points into its code (that doesn't mean libmonogo-client has a problem - but I know that there were a couple of fixes, so better check...).

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Stefan Arts
> Sent: Monday, June 11, 2012 2:36 PM
> To: rsyslog [at] lists
> Subject: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
>
> rsyslog crashes after a few seconds on redhat enterprise linux 6.2.
>
> gdb info:
>
> [root [at] spi tmp]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
> Q\:Reg-11-0-0-11100-1339415918
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) Copyright (C) 2010 Free
> Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html
> >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/rsyslogd...done.
> [New Thread 11105]
> [New Thread 11106]
> [New Thread 11107]
> [New Thread 11108]
> [New Thread 11109]
> [New Thread 11100]
> [New Thread 11101]
> [New Thread 11102]
> [New Thread 11103]
> [New Thread 11104]
> Missing separate debuginfo for /usr/lib64/libee.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
> Missing separate debuginfo for /usr/lib64/libestr.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
> Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
> Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
> Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
> Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
> Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
> Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
> Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
> Missing separate debuginfo for /usr/lib64/rsyslog/ommongodb.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/3c/e13e40b6a06c38813268d1c0ded376c5a0bbe2
> Missing separate debuginfo for /usr/lib64/libmongo-client.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/15/78059d9c73e4553a2d6be2a39fa64c406443e5
> Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
> Missing separate debuginfo for
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
> Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
> /usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
> done.
> Loaded symbols for /lib64/libz.so.1.2.3
> Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> [Thread debugging using libthread_db enabled] done.
> Loaded symbols for /lib64/libpthread-2.12.so Reading symbols from
> /lib64/libdl-2.12.so...Reading symbols from /usr/lib/debug/lib64/libdl-
> 2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libdl-2.12.so
> Reading symbols from /lib64/librt-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/librt-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/librt-2.12.so
> Reading symbols from /usr/lib64/libee.so.0...done.
> Loaded symbols for /usr/lib64/libee.so.0 Reading symbols from
> /usr/lib64/libestr.so.0...done.
> Loaded symbols for /usr/lib64/libestr.so.0 Reading symbols from /lib64/libm-
> 2.12.so...Reading symbols from /usr/lib/debug/lib64/libm-
> 2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libm-2.12.so
> Reading symbols from /lib64/libc-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libc-2.12.so
> Reading symbols from /lib64/ld-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-2.12.so
> Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmnet.so Reading symbols from
> /usr/lib64/rsyslog/imuxsock.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imuxsock.so Reading symbols from
> /usr/lib64/rsyslog/imklog.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imklog.so Reading symbols from
> /usr/lib64/rsyslog/imudp.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imudp.so Reading symbols from
> /usr/lib64/rsyslog/imtcp.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imtcp.so Reading symbols from
> /usr/lib64/rsyslog/lmnetstrms.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so Reading symbols from
> /usr/lib64/rsyslog/lmtcpsrv.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so Reading symbols from
> /usr/lib64/rsyslog/ommongodb.so...done.
> Loaded symbols for /usr/lib64/rsyslog/ommongodb.so Reading symbols from
> /usr/lib64/libmongo-client.so.0...done.
> Loaded symbols for /usr/lib64/libmongo-client.so.0 Reading symbols from
> /lib64/libglib-2.0.so.0.2200.5...Reading symbols from
> /usr/lib/debug/lib64/libglib-2.0.so.0.2200.5.debug...done.
> done.
> Loaded symbols for /lib64/libglib-2.0.so.0.2200.5 Reading symbols from
> /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so Reading symbols from
> /lib64/libnss_files-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from
> /lib64/libnss_dns-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_dns-2.12.so Reading symbols from
> /lib64/libresolv-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libresolv-2.12.so Core was generated by
> `/usr/sbin/rsyslogd -f /etc/rsyslog.conf.v6.mongodb'.
> Program terminated with signal 11, Segmentation fault.
> #0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at
> malloc.c:4879
> 4879 old_idx = fastbin_index(chunksize(old));
> (gdb) info thread
> 10 Thread 0x7f42b1f36700 (LWP 11104) 0x00007f42b55a12c3 in epoll_wait ()
> at ../sysdeps/unix/syscall-template.S:82
> 9 Thread 0x7f42b2937700 (LWP 11103) 0x00007f42b55a12c3 in epoll_wait ()
> at ../sysdeps/unix/syscall-template.S:82
> 8 Thread 0x7f42b3338700 (LWP 11102) 0x00007f42b62f64ed in read () at
> ../sysdeps/unix/syscall-template.S:82
> 7 Thread 0x7f42b3d39700 (LWP 11101) 0x00007f42b55998b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> 6 Thread 0x7f42b692c700 (LWP 11100) 0x00007f42b55998b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> 5 Thread 0x7f42b6894700 (LWP 11109) pthread_cond_wait@@GLIBC_2.3.2
> () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 4 Thread 0x7f42b68c6700 (LWP 11108) pthread_cond_wait@@GLIBC_2.3.2
> () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 3 Thread 0x7f42b68f8700 (LWP 11107) pthread_cond_wait@@GLIBC_2.3.2
> () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 2 Thread 0x7f42b692a700 (LWP 11106) pthread_cond_wait@@GLIBC_2.3.2
> () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> * 1 Thread 0x7f42b1535700 (LWP 11105) _int_free (av=0x7f42a4000020,
> p=0x7f42a4007d50, have_lock=1) at malloc.c:4879
> (gdb) thread 1
> [Switching to thread 1 (Thread 0x7f42b1535700 (LWP 11105))]#0 _int_free
> (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at malloc.c:4879
> 4879 old_idx = fastbin_index(chunksize(old));
> (gdb) bt
> #0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at
> malloc.c:4879
> #1 0x00007f42b5535900 in _int_realloc (av=0x7f42a4000020,
> oldp=0x7f42a4007d50, oldsize=<value optimized out>, nb=<value optimized
> out>) at malloc.c:5345
> #2 0x00007f42b5535af5 in __libc_realloc (oldmem=0x7f42a4007d60,
> bytes=32) at malloc.c:3822
> #3 0x00007f42b3f81621 in IA__g_realloc (mem=<value optimized out>,
> n_bytes=32) at gmem.c:171
> #4 0x00007f42b3f542db in g_array_maybe_expand (array=0x7f42a4002e40,
> len=<value optimized out>) at garray.c:416
> #5 0x00007f42b3f54712 in IA__g_array_append_vals (farray=<value
> optimized
> out>, data=0x7f42a4004550, len=8) at garray.c:207
> #6 0x00007f42b3f54769 in IA__g_byte_array_append
> (array=0x7f42a4002e40, data=<value optimized out>, len=<value optimized
> out>) at garray.c:880
> #7 0x00007f42b422bc19 in _bson_append_string_element
> (b=0x7f42a40078b0, type=<value optimized out>, name=0x7f42b443a1c4
> "sys", val=<value optimized
> out>, length=<value optimized out>) at bson.c:138
> #8 0x00007f42b422d24c in bson_build (type=BSON_TYPE_STRING,
> name=0x7f42b443a1c4 "sys") at bson.c:458
> #9 0x00007f42b4439cf0 in writeMongoDB_msg (pMsg=<value optimized
> out>,
> pData=0x1dc1550) at ommongodb.c:261
> #10 0x0000000000446dd6 in actionCallDoAction (pThis=0x1dc1910,
> pMsg=0x7f42a8000df0, actParams=0x1ddca50) at ../action.c:915
> #11 0x000000000044737f in actionProcessMessage (pAction=0x1dc1910,
> pBatch=0x1dafb80, nElem=2) at ../action.c:962
> #12 tryDoAction (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at
> ../action.c:1057
> #13 submitBatch (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at
> ../action.c:1133
> #14 0x0000000000447790 in processAction (pAction=0x1dc1910,
> pBatch=0x1dafb80, pbShutdownImmediate=<value optimized out>) at
> ../action.c:1224
> #15 processBatchMain (pAction=0x1dc1910, pBatch=0x1dafb80,
> pbShutdownImmediate=<value optimized out>) at ../action.c:1258
> #16 0x0000000000444fdf in doQueueEnqObjDirectBatch (pAction=0x1dc1910,
> pBatch=0x1dafb80) at ../action.c:1727
> #17 0x0000000000445c2b in doSubmitToActionQNotAllMarkBatch
> (pAction=0x1dc1910, pBatch=0x1dafb80) at ../action.c:1636
> #18 0x0000000000440bf3 in processBatchDoActions (pData=0x1dc1910,
> pParam=<value optimized out>) at rule.c:103
> #19 0x000000000042aec7 in llExecFunc (pThis=0x1dc12a0, pFunc=0x440bc0
> <processBatchDoActions>, pParam=0x1dafb80) at linkedlist.c:389
> #20 0x00000000004411f3 in processBatch (pThis=0x1dc1240,
> pBatch=0x1dafb80) at rule.c:293
> #21 0x000000000043ff22 in processBatchDoRules (pData=0x1dc1240,
> pParam=0x1dafb80) at ruleset.c:147
> #22 0x000000000042aec7 in llExecFunc (pThis=0x1da3a68, pFunc=0x43fee0
> <processBatchDoRules>, pParam=0x1dafb80) at linkedlist.c:389
> #23 0x00000000004401fa in processBatch (pBatch=0x1dafb80) at ruleset.c:229
> #24 0x000000000040d056 in msgConsumer (notNeeded=<value optimized
> out>, pBatch=0x1dafb80, pbShutdownImmediate=0x1dc8eb0) at
> syslogd.c:608
> #25 0x000000000043f42b in ConsumerReg (pThis=0x1dc8e90,
> pWti=0x1dafb50) at
> queue.c:1791
> #26 0x000000000043a415 in wtiWorker (pThis=0x1dafb50) at wti.c:313
> #27 0x0000000000438efa in wtpWorker (arg=0x1dafb50) at wtp.c:387
> #28 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b1535700) at
> pthread_create.c:301
> #29 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f42b692a700 (LWP 11106))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842da0) at tcpsrv.c:629
> #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b692a700) at
> pthread_create.c:301
> #3 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 3
> [Switching to thread 3 (Thread 0x7f42b68f8700 (LWP 11107))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842e08) at tcpsrv.c:629
> #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b68f8700) at
> pthread_create.c:301
> #3 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 4
> [Switching to thread 4 (Thread 0x7f42b68c6700 (LWP 11108))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842e70) at tcpsrv.c:629
> #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b68c6700) at
> pthread_create.c:301
> #3 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 5
> [Switching to thread 5 (Thread 0x7f42b6894700 (LWP 11109))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842ed8) at tcpsrv.c:629
> #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b6894700) at
> pthread_create.c:301
> #3 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 6
> [Switching to thread 6 (Thread 0x7f42b692c700 (LWP 11100))]#0
> 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
> #1 0x000000000040e36d in mainloop (argc=<value optimized out>,
> argv=<value optimized out>) at syslogd.c:1314
> #2 realMain (argc=<value optimized out>, argv=<value optimized out>) at
> syslogd.c:2061
> #3 0x00007f42b54d9cdd in __libc_start_main (main=0x40e750 <main>,
> argc=3, ubp_av=0x7fff59743d08, init=<value optimized out>, fini=<value
> optimized
> out>, rtld_fini=<value optimized out>, stack_end=0x7fff59743cf8) at
> libc-start.c:226
> #4 0x000000000040bf89 in _start ()
> (gdb) thread 7
> [Switching to thread 7 (Thread 0x7f42b3d39700 (LWP 11101))]#0
> 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f42b50648bb in runInput (pThrd=<value optimized out>) at
> imuxsock.c:1140
> #2 0x000000000044857a in thrdStarter (arg=0x1ddd700) at ../threads.c:210
> #3 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b3d39700) at
> pthread_create.c:301
> #4 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 8
> [Switching to thread 8 (Thread 0x7f42b3338700 (LWP 11102))]#0
> 0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f42b4e513d8 in readklog () at bsd.c:224
> #2 0x00007f42b4e51509 in klogLogKMsg (pModConf=<value optimized out>)
> at
> bsd.c:282
> #3 0x00007f42b4e50b73 in runInput (pThrd=0x1ddd8f0) at imklog.c:271
> #4 0x000000000044857a in thrdStarter (arg=0x1ddd8f0) at ../threads.c:210
> #5 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b3338700) at
> pthread_create.c:301
> #6 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 9
> [Switching to thread 9 (Thread 0x7f42b2937700 (LWP 11103))]#0
> 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f42b55a12c3 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f42b4c4d0a8 in rcvMainLoop (pThrd=0x1dddae0) at imudp.c:593
> #2 0x000000000044857a in thrdStarter (arg=0x1dddae0) at ../threads.c:210
> #3 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b2937700) at
> pthread_create.c:301
> #4 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 10
> [Switching to thread 10 (Thread 0x7f42b1f36700 (LWP 11104))]#0
> 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f42b55a12c3 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f42b3d3da4f in Wait (pNsdpoll=0x7f42ac000e10, timeout=-1,
> numEntries=0x7f42b1f35dbc, workset=0x7f42b1f35170) at
> nsdpoll_ptcp.c:242
> #2 0x00007f42b464038c in Run (pThis=0x1dae7d0) at tcpsrv.c:884
> #3 0x000000000044857a in thrdStarter (arg=0x1dddcd0) at ../threads.c:210
> #4 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b1f36700) at
> pthread_create.c:301
> #5 0x00007f42b55a0ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb)
>
>
> If you need more, please let me know, but remember: i am not a C
> programmer,
>
> Stefan Arts
> Holland.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


suffestuffie at gmail

Jun 11, 2012, 6:03 AM

Post #4 of 8 (331 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

I have checked, it crashes without mongodb enabled

This one is compiled with ./configure --libdir=/usr/lib64/
--includedir=/usr/include --enable-debug --prefix=/usr:

[root [at] spi etc]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
Q\:Reg-11-0-0-32492-1339419164
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/rsyslogd...done.
[New Thread 32497]
[New Thread 32498]
[New Thread 32499]
[New Thread 32500]
[New Thread 32501]
[New Thread 32492]
[New Thread 32493]
[New Thread 32494]
[New Thread 32496]
[New Thread 32495]
Missing separate debuginfo for /usr/lib64/libee.so.0
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
Missing separate debuginfo for /usr/lib64/libestr.so.0
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
/usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
done.
Loaded symbols for /lib64/libz.so.1.2.3
Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
[Thread debugging using libthread_db enabled]
done.
Loaded symbols for /lib64/libpthread-2.12.so
Reading symbols from /lib64/libdl-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libdl-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libdl-2.12.so
Reading symbols from /lib64/librt-2.12.so...Reading symbols from
/usr/lib/debug/lib64/librt-2.12.so.debug...done.
done.
Loaded symbols for /lib64/librt-2.12.so
Reading symbols from /usr/lib64/libee.so.0...done.
Loaded symbols for /usr/lib64/libee.so.0
Reading symbols from /usr/lib64/libestr.so.0...done.
Loaded symbols for /usr/lib64/libestr.so.0
Reading symbols from /lib64/libm-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libm-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libm-2.12.so
Reading symbols from /lib64/libc-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libc-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libc-2.12.so
Reading symbols from /lib64/ld-2.12.so...Reading symbols from
/usr/lib/debug/lib64/ld-2.12.so.debug...done.
done.
Loaded symbols for /lib64/ld-2.12.so
Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmnet.so
Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done.
Loaded symbols for /usr/lib64/rsyslog/imuxsock.so
Reading symbols from /usr/lib64/rsyslog/imklog.so...done.
Loaded symbols for /usr/lib64/rsyslog/imklog.so
Reading symbols from /usr/lib64/rsyslog/imudp.so...done.
Loaded symbols for /usr/lib64/rsyslog/imudp.so
Reading symbols from /usr/lib64/rsyslog/imtcp.so...done.
Loaded symbols for /usr/lib64/rsyslog/imtcp.so
Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so
Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so
Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so
Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files-2.12.so
Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_dns-2.12.so
Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from
/usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libresolv-2.12.so
Core was generated by `/usr/sbin/rsyslogd -f /etc/rsyslog.conf.v6.orig'.
Program terminated with signal 11, Segmentation fault.
#0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
5161 nextsize = chunksize(nextchunk);
(gdb) info thread
10 Thread 0x7f8dda783700 (LWP 32495) 0x00007f8ddccf22c3 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:82
9 Thread 0x7f8dd9d82700 (LWP 32496) 0x00007f8ddccf22c3 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:82
8 Thread 0x7f8ddb184700 (LWP 32494) 0x00007f8ddda474ed in read () at
../sysdeps/unix/syscall-template.S:82
7 Thread 0x7f8ddbb85700 (LWP 32493) 0x00007f8ddccea8b3 in select () at
../sysdeps/unix/syscall-template.S:82
6 Thread 0x7f8dde07d700 (LWP 32492) 0x00007f8ddccea8b3 in select () at
../sysdeps/unix/syscall-template.S:82
5 Thread 0x7f8dddfe5700 (LWP 32501) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
4 Thread 0x7f8dde017700 (LWP 32500) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
3 Thread 0x7f8dde049700 (LWP 32499) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
2 Thread 0x7f8dde07b700 (LWP 32498) pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
* 1 Thread 0x7f8dd9381700 (LWP 32497) malloc_consolidate
(av=0x7f8dcc000020) at malloc.c:5161
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f8dd9381700 (LWP 32497))]#0
malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
5161 nextsize = chunksize(nextchunk);
(gdb) bt
#0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
#1 0x00007f8ddcc83cd8 in _int_free (av=0x7f8dcc000020, p=0x7f8dcc0070c0,
have_lock=0) at malloc.c:5040
#2 0x00007f8ddcc71a1d in _IO_new_fclose (fp=0x7f8dcc0070d0) at
iofclose.c:88
#3 0x00007f8dd877750e in internal_endent (addr=<value optimized out>,
len=<value optimized out>, af=2, result=0x7f8dd937f040,
buffer=0x7f8dd937ea10 "\177", buflen=1024, errnop=0x7f8dd93816a8,
herrnop=0x7f8dd937f07c)
at nss_files/files-XXX.c:162
#4 _nss_files_gethostbyaddr_r (addr=<value optimized out>, len=<value
optimized out>, af=2, result=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
buflen=1024, errnop=0x7f8dd93816a8, herrnop=0x7f8dd937f07c) at
nss_files/files-hosts.c:361
#5 0x00007f8ddcd0c228 in __gethostbyaddr_r (addr=<value optimized out>,
len=4, type=2, resbuf=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
buflen=1024, result=0x7f8dd937f070, h_errnop=0x7f8dd937f07c) at
../nss/getXXbyYY_r.c:253
#6 0x00007f8ddcd15893 in getnameinfo (sa=0x7f8dd0000b20, addrlen=<value
optimized out>, host=0x7f8dd937f0e0 "core-ca.han.nl", hostlen=1025,
serv=0x0, servlen=0, flags=8) at getnameinfo.c:223
#7 0x000000000041dcac in mygetnameinfo (addr=0x7f8dd0000b20,
pEtry=0x7f8dd937fac8) at dnscache.c:141
#8 resolveAddr (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
dnscache.c:179
#9 addEntry (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
dnscache.c:286 \177", ip=0x7f8dd9\177",
pszIP=<value optimized out>) at net.c:1134
#12 0x000000000042569c in resolveDNS (pMsg=0x7f8dd00008e0) at
msg.c:4327f8dd9380350 "", pszHostFQDN=0x7f8dd937fb30 "\340\t8Ù
#13 0x0000000000425b9d in getHOSTNAME (pM=0x7f8dd00008e0) at msg.c:1828
#14 0x0000000000429036 in MsgGetProp (pMsg=0x7f8dd00008e0, pTpe=0x0,
propid=<value optimized out>, propName=<value optimized out>,
pPropLen=0x7f8dd9380920, pbMustBeFreed=0x7f8dd938092c) at msg.c:2608
#15 0x000000000042a7dc in msgGetMsgVarNew (pThis=0x7f8dd00008e0,
name=<value optimized out>) at msg.c:3429
#16 0x0000000000421775 in cnfGetVar (name=0x1ae7800 "$hostname",
usrptr=<value optimized out>) at rsconf.c:459
#17 0x0000000000419608 in cnfexprEval (expr=0x1ae7240, ret=0x7f8dd93809d0,
usrptr=0x7f8dd00008e0) at rainerscript.c:1166
#18 0x0000000000419287 in cnfexprEval (expr=0x1ae7350, ret=0x7f8dd9380a30,
usrptr=0x7f8dd00008e0) at rainerscript.c:901
#19 0x0000000000418f96 in cnfexprEval (expr=0x1ae9a00, ret=0x7f8dd9380a90,
usrptr=0x7f8dd00008e0) at rainerscript.c:1123
#20 0x0000000000419d8f in cnfexprEvalBool (expr=<value optimized out>,
usrptr=<value optimized out>) at rainerscript.c:1210
#21 0x00000000004413ec in shouldProcessThisMessage (pThis=0x1aea160,
pBatch=0x1b07f00) at rule.c:181
#22 processBatch (pThis=0x1aea160, pBatch=0x1b07f00) at rule.c:281
#23 0x000000000043ff22 in processBatchDoRules (pData=0x1aea160,
pParam=0x1b07f00) at ruleset.c:147
#24 0x000000000042aec7 in llExecFunc (pThis=0x1ad3a68, pFunc=0x43fee0
<processBatchDoRules>, pParam=0x1b07f00) at linkedlist.c:389
#25 0x00000000004401fa in processBatch (pBatch=0x1b07f00) at ruleset.c:229
#26 0x000000000040d056 in msgConsumer (notNeeded=<value optimized out>,
pBatch=0x1b07f00, pbShutdownImmediate=0x1af4200) at syslogd.c:608
#27 0x000000000043f42b in ConsumerReg (pThis=0x1af41e0, pWti=0x1b07ed0) at
queue.c:1791
#28 0x000000000043a415 in wtiWorker (pThis=0x1b07ed0) at wti.c:313
#29 0x0000000000438efa in wtpWorker (arg=0x1b07ed0) at wtp.c:387
#30 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9381700) at
pthread_create.c:301
#31 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f8dde07b700 (LWP 32498))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93da0) at tcpsrv.c:629
#2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde07b700) at
pthread_create.c:301
#3 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f8dde049700 (LWP 32499))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e08) at tcpsrv.c:629
#2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde049700) at
pthread_create.c:301
#3 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f8dde017700 (LWP 32500))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e70) at tcpsrv.c:629
#2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde017700) at
pthread_create.c:301
#3 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 5
[Switching to thread 5 (Thread 0x7f8dddfe5700 (LWP 32501))]#0
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
162 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93ed8) at tcpsrv.c:629
#2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dddfe5700) at
pthread_create.c:301
#3 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 6
[Switching to thread 6 (Thread 0x7f8dde07d700 (LWP 32492))]#0
0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x000000000040e36d in mainloop (argc=<value optimized out>, argv=<value
optimized out>) at syslogd.c:1314
#2 realMain (argc=<value optimized out>, argv=<value optimized out>) at
syslogd.c:2061
#3 0x00007f8ddcc2acdd in __libc_start_main (main=0x40e750 <main>, argc=3,
ubp_av=0x7fff18777008, init=<value optimized out>, fini=<value optimized
libc-start.c:226
#4 0x000000000040bf89 in _start ()
(gdb) thread 7
[Switching to thread 7 (Thread 0x7f8ddbb85700 (LWP 32493))]#0
0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f8ddc7b58bb in runInput (pThrd=<value optimized out>) at
imuxsock.c:1140
#2 0x000000000044857a in thrdStarter (arg=0x1b08ce0) at ../threads.c:210
#3 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddbb85700) at
pthread_create.c:301
#4 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 8
[Switching to thread 8 (Thread 0x7f8ddb184700 (LWP 32494))]#0
0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f8ddc5a23d8 in readklog () at bsd.c:224
#2 0x00007f8ddc5a2509 in klogLogKMsg (pModConf=<value optimized out>) at
bsd.c:282
#3 0x00007f8ddc5a1b73 in runInput (pThrd=0x1b08ed0) at imklog.c:271
#4 0x000000000044857a in thrdStarter (arg=0x1b08ed0) at ../threads.c:210
#5 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddb184700) at
pthread_create.c:301
#6 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 9
[Switching to thread 9 (Thread 0x7f8dd9d82700 (LWP 32496))]#0
0x00007f8ddccf22c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f8ddccf22c3 in epoll_wait () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f8ddbb89a4f in Wait (pNsdpoll=0x7f8dd4000e10, timeout=-1,
numEntries=0x7f8dd9d81dbc, workset=0x7f8dd9d81170) at nsdpoll_ptcp.c:242
#2 0x00007f8ddbd9138c in Run (pThis=0x1adebf0) at tcpsrv.c:884
#3 0x000000000044857a in thrdStarter (arg=0x1b092b0) at ../threads.c:210
#4 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9d82700) at
pthread_create.c:301
#5 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) thread 10
[Switching to thread 10 (Thread 0x7f8dda783700 (LWP 32495))]#0
0x00007f8ddccf22c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f8ddccf22c3 in epoll_wait () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f8ddc39e0a8 in rcvMainLoop (pThrd=0x1b090c0) at imudp.c:593
#2 0x000000000044857a in thrdStarter (arg=0x1b090c0) at ../threads.c:210
#3 0x00007f8ddda407f1 in start_thread (arg=0x7f8dda783700) at
pthread_create.c:301
#4 0x00007f8ddccf1ccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)


[root [at] spi etc]# /usr/sbin/rsyslogd -v2
rsyslogd 6.3.10, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): Yes
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No




2012/6/11 Rainer Gerhards <rgerhards [at] hq>

> Does it work without ommongodb? If so, have you made sure you use the
> newest version of libmongo-client? I notice the crash points into its code
> (that doesn't mean libmonogo-client has a problem - but I know that there
> were a couple of fixes, so better check...).
>
> Rainer
>

.
.
.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 11, 2012, 7:57 AM

Post #5 of 8 (320 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

Thanks, this points into a fresh area of code. I have a recent bug report about expression based filters that seems to be similar. I am about to investigate that other one. If you like, you can simply wait for that outcome. Otherwise, it would be great if you could send me a rsyslog debug log - that will provide more insight on when excactly it crashes.

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Stefan Arts
> Sent: Monday, June 11, 2012 3:03 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
>
> I have checked, it crashes without mongodb enabled
>
> This one is compiled with ./configure --libdir=/usr/lib64/
> --includedir=/usr/include --enable-debug --prefix=/usr:
>
> [root [at] spi etc]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
> Q\:Reg-11-0-0-32492-1339419164
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
> >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/rsyslogd...done.
> [New Thread 32497]
> [New Thread 32498]
> [New Thread 32499]
> [New Thread 32500]
> [New Thread 32501]
> [New Thread 32492]
> [New Thread 32493]
> [New Thread 32494]
> [New Thread 32496]
> [New Thread 32495]
> Missing separate debuginfo for /usr/lib64/libee.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
> Missing separate debuginfo for /usr/lib64/libestr.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
> Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
> Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
> Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
> Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
> Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
> Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
> Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
> Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
> Missing separate debuginfo for
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> /usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
> Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
> /usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
> done.
> Loaded symbols for /lib64/libz.so.1.2.3
> Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> [Thread debugging using libthread_db enabled]
> done.
> Loaded symbols for /lib64/libpthread-2.12.so
> Reading symbols from /lib64/libdl-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libdl-2.12.so
> Reading symbols from /lib64/librt-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/librt-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/librt-2.12.so
> Reading symbols from /usr/lib64/libee.so.0...done.
> Loaded symbols for /usr/lib64/libee.so.0
> Reading symbols from /usr/lib64/libestr.so.0...done.
> Loaded symbols for /usr/lib64/libestr.so.0
> Reading symbols from /lib64/libm-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libm-2.12.so
> Reading symbols from /lib64/libc-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libc-2.12.so
> Reading symbols from /lib64/ld-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-2.12.so
> Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmnet.so
> Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imuxsock.so
> Reading symbols from /usr/lib64/rsyslog/imklog.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imklog.so
> Reading symbols from /usr/lib64/rsyslog/imudp.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imudp.so
> Reading symbols from /usr/lib64/rsyslog/imtcp.so...done.
> Loaded symbols for /usr/lib64/rsyslog/imtcp.so
> Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so
> Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so
> Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
> Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so
> Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_files-2.12.so
> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_dns-2.12.so
> Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libresolv-2.12.so
> Core was generated by `/usr/sbin/rsyslogd -f /etc/rsyslog.conf.v6.orig'.
> Program terminated with signal 11, Segmentation fault.
> #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> 5161 nextsize = chunksize(nextchunk);
> (gdb) info thread
> 10 Thread 0x7f8dda783700 (LWP 32495) 0x00007f8ddccf22c3 in epoll_wait ()
> at ../sysdeps/unix/syscall-template.S:82
> 9 Thread 0x7f8dd9d82700 (LWP 32496) 0x00007f8ddccf22c3 in epoll_wait ()
> at ../sysdeps/unix/syscall-template.S:82
> 8 Thread 0x7f8ddb184700 (LWP 32494) 0x00007f8ddda474ed in read () at
> ../sysdeps/unix/syscall-template.S:82
> 7 Thread 0x7f8ddbb85700 (LWP 32493) 0x00007f8ddccea8b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> 6 Thread 0x7f8dde07d700 (LWP 32492) 0x00007f8ddccea8b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> 5 Thread 0x7f8dddfe5700 (LWP 32501) pthread_cond_wait@@GLIBC_2.3.2 ()
> at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 4 Thread 0x7f8dde017700 (LWP 32500) pthread_cond_wait@@GLIBC_2.3.2 ()
> at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 3 Thread 0x7f8dde049700 (LWP 32499) pthread_cond_wait@@GLIBC_2.3.2 ()
> at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 2 Thread 0x7f8dde07b700 (LWP 32498) pthread_cond_wait@@GLIBC_2.3.2 ()
> at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> * 1 Thread 0x7f8dd9381700 (LWP 32497) malloc_consolidate
> (av=0x7f8dcc000020) at malloc.c:5161
> (gdb) thread 1
> [Switching to thread 1 (Thread 0x7f8dd9381700 (LWP 32497))]#0
> malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> 5161 nextsize = chunksize(nextchunk);
> (gdb) bt
> #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> #1 0x00007f8ddcc83cd8 in _int_free (av=0x7f8dcc000020, p=0x7f8dcc0070c0,
> have_lock=0) at malloc.c:5040
> #2 0x00007f8ddcc71a1d in _IO_new_fclose (fp=0x7f8dcc0070d0) at
> iofclose.c:88
> #3 0x00007f8dd877750e in internal_endent (addr=<value optimized out>,
> len=<value optimized out>, af=2, result=0x7f8dd937f040,
> buffer=0x7f8dd937ea10 "\177", buflen=1024, errnop=0x7f8dd93816a8,
> herrnop=0x7f8dd937f07c)
> at nss_files/files-XXX.c:162
> #4 _nss_files_gethostbyaddr_r (addr=<value optimized out>, len=<value
> optimized out>, af=2, result=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
> buflen=1024, errnop=0x7f8dd93816a8, herrnop=0x7f8dd937f07c) at
> nss_files/files-hosts.c:361
> #5 0x00007f8ddcd0c228 in __gethostbyaddr_r (addr=<value optimized out>,
> len=4, type=2, resbuf=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
> buflen=1024, result=0x7f8dd937f070, h_errnop=0x7f8dd937f07c) at
> ../nss/getXXbyYY_r.c:253
> #6 0x00007f8ddcd15893 in getnameinfo (sa=0x7f8dd0000b20, addrlen=<value
> optimized out>, host=0x7f8dd937f0e0 "core-ca.han.nl", hostlen=1025,
> serv=0x0, servlen=0, flags=8) at getnameinfo.c:223
> #7 0x000000000041dcac in mygetnameinfo (addr=0x7f8dd0000b20,
> pEtry=0x7f8dd937fac8) at dnscache.c:141
> #8 resolveAddr (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> dnscache.c:179
> #9 addEntry (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> dnscache.c:286 \177", ip=0x7f8dd9\177",
> pszIP=<value optimized out>) at net.c:1134
> #12 0x000000000042569c in resolveDNS (pMsg=0x7f8dd00008e0) at
> msg.c:4327f8dd9380350 "", pszHostFQDN=0x7f8dd937fb30 "\340\t8Ù
> #13 0x0000000000425b9d in getHOSTNAME (pM=0x7f8dd00008e0) at
> msg.c:1828
> #14 0x0000000000429036 in MsgGetProp (pMsg=0x7f8dd00008e0, pTpe=0x0,
> propid=<value optimized out>, propName=<value optimized out>,
> pPropLen=0x7f8dd9380920, pbMustBeFreed=0x7f8dd938092c) at msg.c:2608
> #15 0x000000000042a7dc in msgGetMsgVarNew (pThis=0x7f8dd00008e0,
> name=<value optimized out>) at msg.c:3429
> #16 0x0000000000421775 in cnfGetVar (name=0x1ae7800 "$hostname",
> usrptr=<value optimized out>) at rsconf.c:459
> #17 0x0000000000419608 in cnfexprEval (expr=0x1ae7240,
> ret=0x7f8dd93809d0,
> usrptr=0x7f8dd00008e0) at rainerscript.c:1166
> #18 0x0000000000419287 in cnfexprEval (expr=0x1ae7350,
> ret=0x7f8dd9380a30,
> usrptr=0x7f8dd00008e0) at rainerscript.c:901
> #19 0x0000000000418f96 in cnfexprEval (expr=0x1ae9a00,
> ret=0x7f8dd9380a90,
> usrptr=0x7f8dd00008e0) at rainerscript.c:1123
> #20 0x0000000000419d8f in cnfexprEvalBool (expr=<value optimized out>,
> usrptr=<value optimized out>) at rainerscript.c:1210
> #21 0x00000000004413ec in shouldProcessThisMessage (pThis=0x1aea160,
> pBatch=0x1b07f00) at rule.c:181
> #22 processBatch (pThis=0x1aea160, pBatch=0x1b07f00) at rule.c:281
> #23 0x000000000043ff22 in processBatchDoRules (pData=0x1aea160,
> pParam=0x1b07f00) at ruleset.c:147
> #24 0x000000000042aec7 in llExecFunc (pThis=0x1ad3a68, pFunc=0x43fee0
> <processBatchDoRules>, pParam=0x1b07f00) at linkedlist.c:389
> #25 0x00000000004401fa in processBatch (pBatch=0x1b07f00) at ruleset.c:229
> #26 0x000000000040d056 in msgConsumer (notNeeded=<value optimized out>,
> pBatch=0x1b07f00, pbShutdownImmediate=0x1af4200) at syslogd.c:608
> #27 0x000000000043f42b in ConsumerReg (pThis=0x1af41e0, pWti=0x1b07ed0)
> at
> queue.c:1791
> #28 0x000000000043a415 in wtiWorker (pThis=0x1b07ed0) at wti.c:313
> #29 0x0000000000438efa in wtpWorker (arg=0x1b07ed0) at wtp.c:387
> #30 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9381700) at
> pthread_create.c:301
> #31 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f8dde07b700 (LWP 32498))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93da0) at tcpsrv.c:629
> #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde07b700) at
> pthread_create.c:301
> #3 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 3
> [Switching to thread 3 (Thread 0x7f8dde049700 (LWP 32499))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e08) at tcpsrv.c:629
> #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde049700) at
> pthread_create.c:301
> #3 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 4
> [Switching to thread 4 (Thread 0x7f8dde017700 (LWP 32500))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e70) at tcpsrv.c:629
> #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde017700) at
> pthread_create.c:301
> #3 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 5
> [Switching to thread 5 (Thread 0x7f8dddfe5700 (LWP 32501))]#0
> pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> 162 62: movl (%rsp), %edi
> (gdb) bt
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93ed8) at tcpsrv.c:629
> #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dddfe5700) at
> pthread_create.c:301
> #3 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 6
> [Switching to thread 6 (Thread 0x7f8dde07d700 (LWP 32492))]#0
> 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> #1 0x000000000040e36d in mainloop (argc=<value optimized out>, argv=<value
> optimized out>) at syslogd.c:1314
> #2 realMain (argc=<value optimized out>, argv=<value optimized out>) at
> syslogd.c:2061
> #3 0x00007f8ddcc2acdd in __libc_start_main (main=0x40e750 <main>, argc=3,
> ubp_av=0x7fff18777008, init=<value optimized out>, fini=<value optimized
> out>, rtld_fini=<value optimized out>, stack_end=0x7fff18776ff8) at
> libc-start.c:226
> #4 0x000000000040bf89 in _start ()
> (gdb) thread 7
> [Switching to thread 7 (Thread 0x7f8ddbb85700 (LWP 32493))]#0
> 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f8ddc7b58bb in runInput (pThrd=<value optimized out>) at
> imuxsock.c:1140
> #2 0x000000000044857a in thrdStarter (arg=0x1b08ce0) at ../threads.c:210
> #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddbb85700) at
> pthread_create.c:301
> #4 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 8
> [Switching to thread 8 (Thread 0x7f8ddb184700 (LWP 32494))]#0
> 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f8ddc5a23d8 in readklog () at bsd.c:224
> #2 0x00007f8ddc5a2509 in klogLogKMsg (pModConf=<value optimized out>) at
> bsd.c:282
> #3 0x00007f8ddc5a1b73 in runInput (pThrd=0x1b08ed0) at imklog.c:271
> #4 0x000000000044857a in thrdStarter (arg=0x1b08ed0) at ../threads.c:210
> #5 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddb184700) at
> pthread_create.c:301
> #6 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 9
> [Switching to thread 9 (Thread 0x7f8dd9d82700 (LWP 32496))]#0
> 0x00007f8ddccf22c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f8ddccf22c3 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f8ddbb89a4f in Wait (pNsdpoll=0x7f8dd4000e10, timeout=-1,
> numEntries=0x7f8dd9d81dbc, workset=0x7f8dd9d81170) at nsdpoll_ptcp.c:242
> #2 0x00007f8ddbd9138c in Run (pThis=0x1adebf0) at tcpsrv.c:884
> #3 0x000000000044857a in thrdStarter (arg=0x1b092b0) at ../threads.c:210
> #4 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9d82700) at
> pthread_create.c:301
> #5 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb) thread 10
> [Switching to thread 10 (Thread 0x7f8dda783700 (LWP 32495))]#0
> 0x00007f8ddccf22c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
> 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> (gdb) bt
> #0 0x00007f8ddccf22c3 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:82
> #1 0x00007f8ddc39e0a8 in rcvMainLoop (pThrd=0x1b090c0) at imudp.c:593
> #2 0x000000000044857a in thrdStarter (arg=0x1b090c0) at ../threads.c:210
> #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8dda783700) at
> pthread_create.c:301
> #4 0x00007f8ddccf1ccd in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> (gdb)
>
>
> [root [at] spi etc]# /usr/sbin/rsyslogd -v2
> rsyslogd 6.3.10, compiled with:
> FEATURE_REGEXP: Yes
> FEATURE_LARGEFILE: No
> GSSAPI Kerberos 5 support: No
> FEATURE_DEBUG (debug build, slow code): Yes
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> Runtime Instrumentation (slow code): No
>
>
>
>
> 2012/6/11 Rainer Gerhards <rgerhards [at] hq>
>
> > Does it work without ommongodb? If so, have you made sure you use the
> > newest version of libmongo-client? I notice the crash points into its code
> > (that doesn't mean libmonogo-client has a problem - but I know that there
> > were a couple of fixes, so better check...).
> >
> > Rainer
> >
>
> .
> .
> .
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 11, 2012, 8:41 AM

Post #6 of 8 (324 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

I guess this also fixes your problem:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85dea8048aed1b2d74eec57b30155898892daa37

please give it a try and report back.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Monday, June 11, 2012 4:57 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
>
> Thanks, this points into a fresh area of code. I have a recent bug report about
> expression based filters that seems to be similar. I am about to investigate that
> other one. If you like, you can simply wait for that outcome. Otherwise, it would
> be great if you could send me a rsyslog debug log - that will provide more insight
> on when excactly it crashes.
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Stefan Arts
> > Sent: Monday, June 11, 2012 3:03 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
> >
> > I have checked, it crashes without mongodb enabled
> >
> > This one is compiled with ./configure --libdir=/usr/lib64/
> > --includedir=/usr/include --enable-debug --prefix=/usr:
> >
> > [root [at] spi etc]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
> > Q\:Reg-11-0-0-32492-1339419164
> > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
> > Copyright (C) 2010 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
> > >
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>...
> > Reading symbols from /usr/sbin/rsyslogd...done.
> > [New Thread 32497]
> > [New Thread 32498]
> > [New Thread 32499]
> > [New Thread 32500]
> > [New Thread 32501]
> > [New Thread 32492]
> > [New Thread 32493]
> > [New Thread 32494]
> > [New Thread 32496]
> > [New Thread 32495]
> > Missing separate debuginfo for /usr/lib64/libee.so.0
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
> > Missing separate debuginfo for /usr/lib64/libestr.so.0
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
> > Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
> > Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
> > Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
> > Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
> > Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
> > Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
> > Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
> > Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
> > Missing separate debuginfo for
> > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > /usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
> > Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
> > /usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
> > done.
> > Loaded symbols for /lib64/libz.so.1.2.3
> > Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> > [Thread debugging using libthread_db enabled]
> > done.
> > Loaded symbols for /lib64/libpthread-2.12.so
> > Reading symbols from /lib64/libdl-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/libdl-2.12.so
> > Reading symbols from /lib64/librt-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/librt-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/librt-2.12.so
> > Reading symbols from /usr/lib64/libee.so.0...done.
> > Loaded symbols for /usr/lib64/libee.so.0
> > Reading symbols from /usr/lib64/libestr.so.0...done.
> > Loaded symbols for /usr/lib64/libestr.so.0
> > Reading symbols from /lib64/libm-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/libm-2.12.so
> > Reading symbols from /lib64/libc-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/libc-2.12.so
> > Reading symbols from /lib64/ld-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/ld-2.12.so
> > Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/lmnet.so
> > Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/imuxsock.so
> > Reading symbols from /usr/lib64/rsyslog/imklog.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/imklog.so
> > Reading symbols from /usr/lib64/rsyslog/imudp.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/imudp.so
> > Reading symbols from /usr/lib64/rsyslog/imtcp.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/imtcp.so
> > Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so
> > Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so
> > Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
> > Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so
> > Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/libnss_files-2.12.so
> > Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/libnss_dns-2.12.so
> > Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from
> > /usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
> > done.
> > Loaded symbols for /lib64/libresolv-2.12.so
> > Core was generated by `/usr/sbin/rsyslogd -f /etc/rsyslog.conf.v6.orig'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > 5161 nextsize = chunksize(nextchunk);
> > (gdb) info thread
> > 10 Thread 0x7f8dda783700 (LWP 32495) 0x00007f8ddccf22c3 in epoll_wait ()
> > at ../sysdeps/unix/syscall-template.S:82
> > 9 Thread 0x7f8dd9d82700 (LWP 32496) 0x00007f8ddccf22c3 in epoll_wait ()
> > at ../sysdeps/unix/syscall-template.S:82
> > 8 Thread 0x7f8ddb184700 (LWP 32494) 0x00007f8ddda474ed in read () at
> > ../sysdeps/unix/syscall-template.S:82
> > 7 Thread 0x7f8ddbb85700 (LWP 32493) 0x00007f8ddccea8b3 in select () at
> > ../sysdeps/unix/syscall-template.S:82
> > 6 Thread 0x7f8dde07d700 (LWP 32492) 0x00007f8ddccea8b3 in select () at
> > ../sysdeps/unix/syscall-template.S:82
> > 5 Thread 0x7f8dddfe5700 (LWP 32501) pthread_cond_wait@@GLIBC_2.3.2
> ()
> > at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 4 Thread 0x7f8dde017700 (LWP 32500) pthread_cond_wait@@GLIBC_2.3.2
> ()
> > at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 3 Thread 0x7f8dde049700 (LWP 32499) pthread_cond_wait@@GLIBC_2.3.2
> ()
> > at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 2 Thread 0x7f8dde07b700 (LWP 32498) pthread_cond_wait@@GLIBC_2.3.2
> ()
> > at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > * 1 Thread 0x7f8dd9381700 (LWP 32497) malloc_consolidate
> > (av=0x7f8dcc000020) at malloc.c:5161
> > (gdb) thread 1
> > [Switching to thread 1 (Thread 0x7f8dd9381700 (LWP 32497))]#0
> > malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > 5161 nextsize = chunksize(nextchunk);
> > (gdb) bt
> > #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > #1 0x00007f8ddcc83cd8 in _int_free (av=0x7f8dcc000020, p=0x7f8dcc0070c0,
> > have_lock=0) at malloc.c:5040
> > #2 0x00007f8ddcc71a1d in _IO_new_fclose (fp=0x7f8dcc0070d0) at
> > iofclose.c:88
> > #3 0x00007f8dd877750e in internal_endent (addr=<value optimized out>,
> > len=<value optimized out>, af=2, result=0x7f8dd937f040,
> > buffer=0x7f8dd937ea10 "\177", buflen=1024, errnop=0x7f8dd93816a8,
> > herrnop=0x7f8dd937f07c)
> > at nss_files/files-XXX.c:162
> > #4 _nss_files_gethostbyaddr_r (addr=<value optimized out>, len=<value
> > optimized out>, af=2, result=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
> > buflen=1024, errnop=0x7f8dd93816a8, herrnop=0x7f8dd937f07c) at
> > nss_files/files-hosts.c:361
> > #5 0x00007f8ddcd0c228 in __gethostbyaddr_r (addr=<value optimized out>,
> > len=4, type=2, resbuf=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
> > buflen=1024, result=0x7f8dd937f070, h_errnop=0x7f8dd937f07c) at
> > ../nss/getXXbyYY_r.c:253
> > #6 0x00007f8ddcd15893 in getnameinfo (sa=0x7f8dd0000b20, addrlen=<value
> > optimized out>, host=0x7f8dd937f0e0 "core-ca.han.nl", hostlen=1025,
> > serv=0x0, servlen=0, flags=8) at getnameinfo.c:223
> > #7 0x000000000041dcac in mygetnameinfo (addr=0x7f8dd0000b20,
> > pEtry=0x7f8dd937fac8) at dnscache.c:141
> > #8 resolveAddr (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> > dnscache.c:179
> > #9 addEntry (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> > dnscache.c:286 \177", ip=0x7f8dd9\177",
> > pszIP=<value optimized out>) at net.c:1134
> > #12 0x000000000042569c in resolveDNS (pMsg=0x7f8dd00008e0) at
> > msg.c:4327f8dd9380350 "", pszHostFQDN=0x7f8dd937fb30 "\340\t8Ù
> > #13 0x0000000000425b9d in getHOSTNAME (pM=0x7f8dd00008e0) at
> > msg.c:1828
> > #14 0x0000000000429036 in MsgGetProp (pMsg=0x7f8dd00008e0, pTpe=0x0,
> > propid=<value optimized out>, propName=<value optimized out>,
> > pPropLen=0x7f8dd9380920, pbMustBeFreed=0x7f8dd938092c) at msg.c:2608
> > #15 0x000000000042a7dc in msgGetMsgVarNew (pThis=0x7f8dd00008e0,
> > name=<value optimized out>) at msg.c:3429
> > #16 0x0000000000421775 in cnfGetVar (name=0x1ae7800 "$hostname",
> > usrptr=<value optimized out>) at rsconf.c:459
> > #17 0x0000000000419608 in cnfexprEval (expr=0x1ae7240,
> > ret=0x7f8dd93809d0,
> > usrptr=0x7f8dd00008e0) at rainerscript.c:1166
> > #18 0x0000000000419287 in cnfexprEval (expr=0x1ae7350,
> > ret=0x7f8dd9380a30,
> > usrptr=0x7f8dd00008e0) at rainerscript.c:901
> > #19 0x0000000000418f96 in cnfexprEval (expr=0x1ae9a00,
> > ret=0x7f8dd9380a90,
> > usrptr=0x7f8dd00008e0) at rainerscript.c:1123
> > #20 0x0000000000419d8f in cnfexprEvalBool (expr=<value optimized out>,
> > usrptr=<value optimized out>) at rainerscript.c:1210
> > #21 0x00000000004413ec in shouldProcessThisMessage (pThis=0x1aea160,
> > pBatch=0x1b07f00) at rule.c:181
> > #22 processBatch (pThis=0x1aea160, pBatch=0x1b07f00) at rule.c:281
> > #23 0x000000000043ff22 in processBatchDoRules (pData=0x1aea160,
> > pParam=0x1b07f00) at ruleset.c:147
> > #24 0x000000000042aec7 in llExecFunc (pThis=0x1ad3a68, pFunc=0x43fee0
> > <processBatchDoRules>, pParam=0x1b07f00) at linkedlist.c:389
> > #25 0x00000000004401fa in processBatch (pBatch=0x1b07f00) at
> ruleset.c:229
> > #26 0x000000000040d056 in msgConsumer (notNeeded=<value optimized
> out>,
> > pBatch=0x1b07f00, pbShutdownImmediate=0x1af4200) at syslogd.c:608
> > #27 0x000000000043f42b in ConsumerReg (pThis=0x1af41e0,
> pWti=0x1b07ed0)
> > at
> > queue.c:1791
> > #28 0x000000000043a415 in wtiWorker (pThis=0x1b07ed0) at wti.c:313
> > #29 0x0000000000438efa in wtpWorker (arg=0x1b07ed0) at wtp.c:387
> > #30 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9381700) at
> > pthread_create.c:301
> > #31 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 2
> > [Switching to thread 2 (Thread 0x7f8dde07b700 (LWP 32498))]#0
> > pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 162 62: movl (%rsp), %edi
> > (gdb) bt
> > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93da0) at tcpsrv.c:629
> > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde07b700) at
> > pthread_create.c:301
> > #3 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 3
> > [Switching to thread 3 (Thread 0x7f8dde049700 (LWP 32499))]#0
> > pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 162 62: movl (%rsp), %edi
> > (gdb) bt
> > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e08) at tcpsrv.c:629
> > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde049700) at
> > pthread_create.c:301
> > #3 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 4
> > [Switching to thread 4 (Thread 0x7f8dde017700 (LWP 32500))]#0
> > pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 162 62: movl (%rsp), %edi
> > (gdb) bt
> > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e70) at tcpsrv.c:629
> > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde017700) at
> > pthread_create.c:301
> > #3 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 5
> > [Switching to thread 5 (Thread 0x7f8dddfe5700 (LWP 32501))]#0
> > pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > 162 62: movl (%rsp), %edi
> > (gdb) bt
> > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93ed8) at tcpsrv.c:629
> > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dddfe5700) at
> > pthread_create.c:301
> > #3 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 6
> > [Switching to thread 6 (Thread 0x7f8dde07d700 (LWP 32492))]#0
> > 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > (gdb) bt
> > #0 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> > #1 0x000000000040e36d in mainloop (argc=<value optimized out>,
> argv=<value
> > optimized out>) at syslogd.c:1314
> > #2 realMain (argc=<value optimized out>, argv=<value optimized out>) at
> > syslogd.c:2061
> > #3 0x00007f8ddcc2acdd in __libc_start_main (main=0x40e750 <main>,
> argc=3,
> > ubp_av=0x7fff18777008, init=<value optimized out>, fini=<value optimized
> > out>, rtld_fini=<value optimized out>, stack_end=0x7fff18776ff8) at
> > libc-start.c:226
> > #4 0x000000000040bf89 in _start ()
> > (gdb) thread 7
> > [Switching to thread 7 (Thread 0x7f8ddbb85700 (LWP 32493))]#0
> > 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > (gdb) bt
> > #0 0x00007f8ddccea8b3 in select () at ../sysdeps/unix/syscall-template.S:82
> > #1 0x00007f8ddc7b58bb in runInput (pThrd=<value optimized out>) at
> > imuxsock.c:1140
> > #2 0x000000000044857a in thrdStarter (arg=0x1b08ce0) at ../threads.c:210
> > #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddbb85700) at
> > pthread_create.c:301
> > #4 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 8
> > [Switching to thread 8 (Thread 0x7f8ddb184700 (LWP 32494))]#0
> > 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
> > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > (gdb) bt
> > #0 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
> > #1 0x00007f8ddc5a23d8 in readklog () at bsd.c:224
> > #2 0x00007f8ddc5a2509 in klogLogKMsg (pModConf=<value optimized out>)
> at
> > bsd.c:282
> > #3 0x00007f8ddc5a1b73 in runInput (pThrd=0x1b08ed0) at imklog.c:271
> > #4 0x000000000044857a in thrdStarter (arg=0x1b08ed0) at ../threads.c:210
> > #5 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddb184700) at
> > pthread_create.c:301
> > #6 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 9
> > [Switching to thread 9 (Thread 0x7f8dd9d82700 (LWP 32496))]#0
> > 0x00007f8ddccf22c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
> > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > (gdb) bt
> > #0 0x00007f8ddccf22c3 in epoll_wait () at
> > ../sysdeps/unix/syscall-template.S:82
> > #1 0x00007f8ddbb89a4f in Wait (pNsdpoll=0x7f8dd4000e10, timeout=-1,
> > numEntries=0x7f8dd9d81dbc, workset=0x7f8dd9d81170) at
> nsdpoll_ptcp.c:242
> > #2 0x00007f8ddbd9138c in Run (pThis=0x1adebf0) at tcpsrv.c:884
> > #3 0x000000000044857a in thrdStarter (arg=0x1b092b0) at ../threads.c:210
> > #4 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9d82700) at
> > pthread_create.c:301
> > #5 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb) thread 10
> > [Switching to thread 10 (Thread 0x7f8dda783700 (LWP 32495))]#0
> > 0x00007f8ddccf22c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
> > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > (gdb) bt
> > #0 0x00007f8ddccf22c3 in epoll_wait () at
> > ../sysdeps/unix/syscall-template.S:82
> > #1 0x00007f8ddc39e0a8 in rcvMainLoop (pThrd=0x1b090c0) at imudp.c:593
> > #2 0x000000000044857a in thrdStarter (arg=0x1b090c0) at ../threads.c:210
> > #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8dda783700) at
> > pthread_create.c:301
> > #4 0x00007f8ddccf1ccd in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > (gdb)
> >
> >
> > [root [at] spi etc]# /usr/sbin/rsyslogd -v2
> > rsyslogd 6.3.10, compiled with:
> > FEATURE_REGEXP: Yes
> > FEATURE_LARGEFILE: No
> > GSSAPI Kerberos 5 support: No
> > FEATURE_DEBUG (debug build, slow code): Yes
> > 32bit Atomic operations supported: Yes
> > 64bit Atomic operations supported: Yes
> > Runtime Instrumentation (slow code): No
> >
> >
> >
> >
> > 2012/6/11 Rainer Gerhards <rgerhards [at] hq>
> >
> > > Does it work without ommongodb? If so, have you made sure you use the
> > > newest version of libmongo-client? I notice the crash points into its code
> > > (that doesn't mean libmonogo-client has a problem - but I know that there
> > > were a couple of fixes, so better check...).
> > >
> > > Rainer
> > >
> >
> > .
> > .
> > .
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


suffestuffie at gmail

Jun 12, 2012, 12:02 AM

Post #7 of 8 (323 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

problem is gone indeed. rsyslog is running for 15 hours now (and writing
log into mongodb).

thanks.

2012/6/11 Rainer Gerhards <rgerhards [at] hq>

> I guess this also fixes your problem:
>
>
> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85dea8048aed1b2d74eec57b30155898892daa37
>
> please give it a try and report back.
>
> Thanks,
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Rainer Gerhards
> > Sent: Monday, June 11, 2012 4:57 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
> >
> > Thanks, this points into a fresh area of code. I have a recent bug
> report about
> > expression based filters that seems to be similar. I am about to
> investigate that
> > other one. If you like, you can simply wait for that outcome. Otherwise,
> it would
> > be great if you could send me a rsyslog debug log - that will provide
> more insight
> > on when excactly it crashes.
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Stefan Arts
> > > Sent: Monday, June 11, 2012 3:03 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
> > >
> > > I have checked, it crashes without mongodb enabled
> > >
> > > This one is compiled with ./configure --libdir=/usr/lib64/
> > > --includedir=/usr/include --enable-debug --prefix=/usr:
> > >
> > > [root [at] spi etc]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
> > > Q\:Reg-11-0-0-32492-1339419164
> > > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
> > > Copyright (C) 2010 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html
> > > >
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> > > and "show warranty" for details.
> > > This GDB was configured as "x86_64-redhat-linux-gnu".
> > > For bug reporting instructions, please see:
> > > <http://www.gnu.org/software/gdb/bugs/>...
> > > Reading symbols from /usr/sbin/rsyslogd...done.
> > > [New Thread 32497]
> > > [New Thread 32498]
> > > [New Thread 32499]
> > > [New Thread 32500]
> > > [New Thread 32501]
> > > [New Thread 32492]
> > > [New Thread 32493]
> > > [New Thread 32494]
> > > [New Thread 32496]
> > > [New Thread 32495]
> > > Missing separate debuginfo for /usr/lib64/libee.so.0
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
> > > Missing separate debuginfo for /usr/lib64/libestr.so.0
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
> > > Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
> > > Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
> > > Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
> > > Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
> > > Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
> > > Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
> > > Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
> > > Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
> > > Missing separate debuginfo for
> > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > /usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
> > > Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
> > > /usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libz.so.1.2.3
> > > Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> > > [Thread debugging using libthread_db enabled]
> > > done.
> > > Loaded symbols for /lib64/libpthread-2.12.so
> > > Reading symbols from /lib64/libdl-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libdl-2.12.so
> > > Reading symbols from /lib64/librt-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/librt-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/librt-2.12.so
> > > Reading symbols from /usr/lib64/libee.so.0...done.
> > > Loaded symbols for /usr/lib64/libee.so.0
> > > Reading symbols from /usr/lib64/libestr.so.0...done.
> > > Loaded symbols for /usr/lib64/libestr.so.0
> > > Reading symbols from /lib64/libm-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libm-2.12.so
> > > Reading symbols from /lib64/libc-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libc-2.12.so
> > > Reading symbols from /lib64/ld-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/ld-2.12.so
> > > Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/lmnet.so
> > > Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/imuxsock.so
> > > Reading symbols from /usr/lib64/rsyslog/imklog.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/imklog.so
> > > Reading symbols from /usr/lib64/rsyslog/imudp.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/imudp.so
> > > Reading symbols from /usr/lib64/rsyslog/imtcp.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/imtcp.so
> > > Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so
> > > Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so
> > > Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
> > > Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so
> > > Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libnss_files-2.12.so
> > > Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libnss_dns-2.12.so
> > > Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from
> > > /usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
> > > done.
> > > Loaded symbols for /lib64/libresolv-2.12.so
> > > Core was generated by `/usr/sbin/rsyslogd -f
> /etc/rsyslog.conf.v6.orig'.
> > > Program terminated with signal 11, Segmentation fault.
> > > #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > > 5161 nextsize = chunksize(nextchunk);
> > > (gdb) info thread
> > > 10 Thread 0x7f8dda783700 (LWP 32495) 0x00007f8ddccf22c3 in
> epoll_wait ()
> > > at ../sysdeps/unix/syscall-template.S:82
> > > 9 Thread 0x7f8dd9d82700 (LWP 32496) 0x00007f8ddccf22c3 in
> epoll_wait ()
> > > at ../sysdeps/unix/syscall-template.S:82
> > > 8 Thread 0x7f8ddb184700 (LWP 32494) 0x00007f8ddda474ed in read () at
> > > ../sysdeps/unix/syscall-template.S:82
> > > 7 Thread 0x7f8ddbb85700 (LWP 32493) 0x00007f8ddccea8b3 in select ()
> at
> > > ../sysdeps/unix/syscall-template.S:82
> > > 6 Thread 0x7f8dde07d700 (LWP 32492) 0x00007f8ddccea8b3 in select ()
> at
> > > ../sysdeps/unix/syscall-template.S:82
> > > 5 Thread 0x7f8dddfe5700 (LWP 32501) pthread_cond_wait@@GLIBC_2.3.2
> > ()
> > > at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 4 Thread 0x7f8dde017700 (LWP 32500) pthread_cond_wait@@GLIBC_2.3.2
> > ()
> > > at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 3 Thread 0x7f8dde049700 (LWP 32499) pthread_cond_wait@@GLIBC_2.3.2
> > ()
> > > at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 2 Thread 0x7f8dde07b700 (LWP 32498) pthread_cond_wait@@GLIBC_2.3.2
> > ()
> > > at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > * 1 Thread 0x7f8dd9381700 (LWP 32497) malloc_consolidate
> > > (av=0x7f8dcc000020) at malloc.c:5161
> > > (gdb) thread 1
> > > [Switching to thread 1 (Thread 0x7f8dd9381700 (LWP 32497))]#0
> > > malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > > 5161 nextsize = chunksize(nextchunk);
> > > (gdb) bt
> > > #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > > #1 0x00007f8ddcc83cd8 in _int_free (av=0x7f8dcc000020,
> p=0x7f8dcc0070c0,
> > > have_lock=0) at malloc.c:5040
> > > #2 0x00007f8ddcc71a1d in _IO_new_fclose (fp=0x7f8dcc0070d0) at
> > > iofclose.c:88
> > > #3 0x00007f8dd877750e in internal_endent (addr=<value optimized out>,
> > > len=<value optimized out>, af=2, result=0x7f8dd937f040,
> > > buffer=0x7f8dd937ea10 "\177", buflen=1024, errnop=0x7f8dd93816a8,
> > > herrnop=0x7f8dd937f07c)
> > > at nss_files/files-XXX.c:162
> > > #4 _nss_files_gethostbyaddr_r (addr=<value optimized out>, len=<value
> > > optimized out>, af=2, result=0x7f8dd937f040, buffer=0x7f8dd937ea10
> "\177",
> > > buflen=1024, errnop=0x7f8dd93816a8, herrnop=0x7f8dd937f07c) at
> > > nss_files/files-hosts.c:361
> > > #5 0x00007f8ddcd0c228 in __gethostbyaddr_r (addr=<value optimized
> out>,
> > > len=4, type=2, resbuf=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
> > > buflen=1024, result=0x7f8dd937f070, h_errnop=0x7f8dd937f07c) at
> > > ../nss/getXXbyYY_r.c:253
> > > #6 0x00007f8ddcd15893 in getnameinfo (sa=0x7f8dd0000b20,
> addrlen=<value
> > > optimized out>, host=0x7f8dd937f0e0 "core-ca.han.nl", hostlen=1025,
> > > serv=0x0, servlen=0, flags=8) at getnameinfo.c:223
> > > #7 0x000000000041dcac in mygetnameinfo (addr=0x7f8dd0000b20,
> > > pEtry=0x7f8dd937fac8) at dnscache.c:141
> > > #8 resolveAddr (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> > > dnscache.c:179
> > > #9 addEntry (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> > > dnscache.c:286 \177", ip=0x7f8dd9\177",
> > > pszIP=<value optimized out>) at net.c:1134
> > > #12 0x000000000042569c in resolveDNS (pMsg=0x7f8dd00008e0) at
> > > msg.c:4327f8dd9380350 "", pszHostFQDN=0x7f8dd937fb30 "\340\t8Ù
> > > #13 0x0000000000425b9d in getHOSTNAME (pM=0x7f8dd00008e0) at
> > > msg.c:1828
> > > #14 0x0000000000429036 in MsgGetProp (pMsg=0x7f8dd00008e0, pTpe=0x0,
> > > propid=<value optimized out>, propName=<value optimized out>,
> > > pPropLen=0x7f8dd9380920, pbMustBeFreed=0x7f8dd938092c) at msg.c:2608
> > > #15 0x000000000042a7dc in msgGetMsgVarNew (pThis=0x7f8dd00008e0,
> > > name=<value optimized out>) at msg.c:3429
> > > #16 0x0000000000421775 in cnfGetVar (name=0x1ae7800 "$hostname",
> > > usrptr=<value optimized out>) at rsconf.c:459
> > > #17 0x0000000000419608 in cnfexprEval (expr=0x1ae7240,
> > > ret=0x7f8dd93809d0,
> > > usrptr=0x7f8dd00008e0) at rainerscript.c:1166
> > > #18 0x0000000000419287 in cnfexprEval (expr=0x1ae7350,
> > > ret=0x7f8dd9380a30,
> > > usrptr=0x7f8dd00008e0) at rainerscript.c:901
> > > #19 0x0000000000418f96 in cnfexprEval (expr=0x1ae9a00,
> > > ret=0x7f8dd9380a90,
> > > usrptr=0x7f8dd00008e0) at rainerscript.c:1123
> > > #20 0x0000000000419d8f in cnfexprEvalBool (expr=<value optimized out>,
> > > usrptr=<value optimized out>) at rainerscript.c:1210
> > > #21 0x00000000004413ec in shouldProcessThisMessage (pThis=0x1aea160,
> > > pBatch=0x1b07f00) at rule.c:181
> > > #22 processBatch (pThis=0x1aea160, pBatch=0x1b07f00) at rule.c:281
> > > #23 0x000000000043ff22 in processBatchDoRules (pData=0x1aea160,
> > > pParam=0x1b07f00) at ruleset.c:147
> > > #24 0x000000000042aec7 in llExecFunc (pThis=0x1ad3a68, pFunc=0x43fee0
> > > <processBatchDoRules>, pParam=0x1b07f00) at linkedlist.c:389
> > > #25 0x00000000004401fa in processBatch (pBatch=0x1b07f00) at
> > ruleset.c:229
> > > #26 0x000000000040d056 in msgConsumer (notNeeded=<value optimized
> > out>,
> > > pBatch=0x1b07f00, pbShutdownImmediate=0x1af4200) at syslogd.c:608
> > > #27 0x000000000043f42b in ConsumerReg (pThis=0x1af41e0,
> > pWti=0x1b07ed0)
> > > at
> > > queue.c:1791
> > > #28 0x000000000043a415 in wtiWorker (pThis=0x1b07ed0) at wti.c:313
> > > #29 0x0000000000438efa in wtpWorker (arg=0x1b07ed0) at wtp.c:387
> > > #30 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9381700) at
> > > pthread_create.c:301
> > > #31 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 2
> > > [Switching to thread 2 (Thread 0x7f8dde07b700 (LWP 32498))]#0
> > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 162 62: movl (%rsp), %edi
> > > (gdb) bt
> > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93da0) at tcpsrv.c:629
> > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde07b700) at
> > > pthread_create.c:301
> > > #3 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 3
> > > [Switching to thread 3 (Thread 0x7f8dde049700 (LWP 32499))]#0
> > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 162 62: movl (%rsp), %edi
> > > (gdb) bt
> > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e08) at tcpsrv.c:629
> > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde049700) at
> > > pthread_create.c:301
> > > #3 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 4
> > > [Switching to thread 4 (Thread 0x7f8dde017700 (LWP 32500))]#0
> > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 162 62: movl (%rsp), %edi
> > > (gdb) bt
> > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e70) at tcpsrv.c:629
> > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde017700) at
> > > pthread_create.c:301
> > > #3 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 5
> > > [Switching to thread 5 (Thread 0x7f8dddfe5700 (LWP 32501))]#0
> > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > 162 62: movl (%rsp), %edi
> > > (gdb) bt
> > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93ed8) at tcpsrv.c:629
> > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dddfe5700) at
> > > pthread_create.c:301
> > > #3 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 6
> > > [Switching to thread 6 (Thread 0x7f8dde07d700 (LWP 32492))]#0
> > > 0x00007f8ddccea8b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > (gdb) bt
> > > #0 0x00007f8ddccea8b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> > > #1 0x000000000040e36d in mainloop (argc=<value optimized out>,
> > argv=<value
> > > optimized out>) at syslogd.c:1314
> > > #2 realMain (argc=<value optimized out>, argv=<value optimized out>)
> at
> > > syslogd.c:2061
> > > #3 0x00007f8ddcc2acdd in __libc_start_main (main=0x40e750 <main>,
> > argc=3,
> > > ubp_av=0x7fff18777008, init=<value optimized out>, fini=<value
> optimized
> > > out>, rtld_fini=<value optimized out>, stack_end=0x7fff18776ff8) at
> > > libc-start.c:226
> > > #4 0x000000000040bf89 in _start ()
> > > (gdb) thread 7
> > > [Switching to thread 7 (Thread 0x7f8ddbb85700 (LWP 32493))]#0
> > > 0x00007f8ddccea8b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > (gdb) bt
> > > #0 0x00007f8ddccea8b3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> > > #1 0x00007f8ddc7b58bb in runInput (pThrd=<value optimized out>) at
> > > imuxsock.c:1140
> > > #2 0x000000000044857a in thrdStarter (arg=0x1b08ce0) at
> ../threads.c:210
> > > #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddbb85700) at
> > > pthread_create.c:301
> > > #4 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 8
> > > [Switching to thread 8 (Thread 0x7f8ddb184700 (LWP 32494))]#0
> > > 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
> > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > (gdb) bt
> > > #0 0x00007f8ddda474ed in read () at
> ../sysdeps/unix/syscall-template.S:82
> > > #1 0x00007f8ddc5a23d8 in readklog () at bsd.c:224
> > > #2 0x00007f8ddc5a2509 in klogLogKMsg (pModConf=<value optimized out>)
> > at
> > > bsd.c:282
> > > #3 0x00007f8ddc5a1b73 in runInput (pThrd=0x1b08ed0) at imklog.c:271
> > > #4 0x000000000044857a in thrdStarter (arg=0x1b08ed0) at
> ../threads.c:210
> > > #5 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddb184700) at
> > > pthread_create.c:301
> > > #6 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 9
> > > [Switching to thread 9 (Thread 0x7f8dd9d82700 (LWP 32496))]#0
> > > 0x00007f8ddccf22c3 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:82
> > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > (gdb) bt
> > > #0 0x00007f8ddccf22c3 in epoll_wait () at
> > > ../sysdeps/unix/syscall-template.S:82
> > > #1 0x00007f8ddbb89a4f in Wait (pNsdpoll=0x7f8dd4000e10, timeout=-1,
> > > numEntries=0x7f8dd9d81dbc, workset=0x7f8dd9d81170) at
> > nsdpoll_ptcp.c:242
> > > #2 0x00007f8ddbd9138c in Run (pThis=0x1adebf0) at tcpsrv.c:884
> > > #3 0x000000000044857a in thrdStarter (arg=0x1b092b0) at
> ../threads.c:210
> > > #4 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9d82700) at
> > > pthread_create.c:301
> > > #5 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb) thread 10
> > > [Switching to thread 10 (Thread 0x7f8dda783700 (LWP 32495))]#0
> > > 0x00007f8ddccf22c3 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:82
> > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > (gdb) bt
> > > #0 0x00007f8ddccf22c3 in epoll_wait () at
> > > ../sysdeps/unix/syscall-template.S:82
> > > #1 0x00007f8ddc39e0a8 in rcvMainLoop (pThrd=0x1b090c0) at imudp.c:593
> > > #2 0x000000000044857a in thrdStarter (arg=0x1b090c0) at
> ../threads.c:210
> > > #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8dda783700) at
> > > pthread_create.c:301
> > > #4 0x00007f8ddccf1ccd in clone () at
> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > (gdb)
> > >
> > >
> > > [root [at] spi etc]# /usr/sbin/rsyslogd -v2
> > > rsyslogd 6.3.10, compiled with:
> > > FEATURE_REGEXP: Yes
> > > FEATURE_LARGEFILE: No
> > > GSSAPI Kerberos 5 support: No
> > > FEATURE_DEBUG (debug build, slow code): Yes
> > > 32bit Atomic operations supported: Yes
> > > 64bit Atomic operations supported: Yes
> > > Runtime Instrumentation (slow code): No
> > >
> > >
> > >
> > >
> > > 2012/6/11 Rainer Gerhards <rgerhards [at] hq>
> > >
> > > > Does it work without ommongodb? If so, have you made sure you use the
> > > > newest version of libmongo-client? I notice the crash points into
> its code
> > > > (that doesn't mean libmonogo-client has a problem - but I know that
> there
> > > > were a couple of fixes, so better check...).
> > > >
> > > > Rainer
> > > >
> > >
> > > .
> > > .
> > > .
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 12, 2012, 12:03 AM

Post #8 of 8 (319 views)
Permalink
Re: rsyslog 6.3.10 segfault on RHEL 6.2 [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Stefan Arts
> Sent: Tuesday, June 12, 2012 9:03 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
>
> problem is gone indeed. rsyslog is running for 15 hours now (and writing
> log into mongodb).

Thanks for the feedback. I'll see that I get this version released pretty soon, as this is really a nasty bug ;)

Rainer
>
> thanks.
>
> 2012/6/11 Rainer Gerhards <rgerhards [at] hq>
>
> > I guess this also fixes your problem:
> >
> >
> >
> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85dea8048aed1b2d74ee
> c57b30155898892daa37
> >
> > please give it a try and report back.
> >
> > Thanks,
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Rainer Gerhards
> > > Sent: Monday, June 11, 2012 4:57 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
> > >
> > > Thanks, this points into a fresh area of code. I have a recent bug
> > report about
> > > expression based filters that seems to be similar. I am about to
> > investigate that
> > > other one. If you like, you can simply wait for that outcome. Otherwise,
> > it would
> > > be great if you could send me a rsyslog debug log - that will provide
> > more insight
> > > on when excactly it crashes.
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Stefan Arts
> > > > Sent: Monday, June 11, 2012 3:03 PM
> > > > To: rsyslog-users
> > > > Subject: Re: [rsyslog] rsyslog 6.3.10 segfault on RHEL 6.2
> > > >
> > > > I have checked, it crashes without mongodb enabled
> > > >
> > > > This one is compiled with ./configure --libdir=/usr/lib64/
> > > > --includedir=/usr/include --enable-debug --prefix=/usr:
> > > >
> > > > [root [at] spi etc]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\
> > > > Q\:Reg-11-0-0-32492-1339419164
> > > > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
> > > > Copyright (C) 2010 Free Software Foundation, Inc.
> > > > License GPLv3+: GNU GPL version 3 or later <
> > http://gnu.org/licenses/gpl.html
> > > > >
> > > > This is free software: you are free to change and redistribute it.
> > > > There is NO WARRANTY, to the extent permitted by law. Type "show
> > copying"
> > > > and "show warranty" for details.
> > > > This GDB was configured as "x86_64-redhat-linux-gnu".
> > > > For bug reporting instructions, please see:
> > > > <http://www.gnu.org/software/gdb/bugs/>...
> > > > Reading symbols from /usr/sbin/rsyslogd...done.
> > > > [New Thread 32497]
> > > > [New Thread 32498]
> > > > [New Thread 32499]
> > > > [New Thread 32500]
> > > > [New Thread 32501]
> > > > [New Thread 32492]
> > > > [New Thread 32493]
> > > > [New Thread 32494]
> > > > [New Thread 32496]
> > > > [New Thread 32495]
> > > > Missing separate debuginfo for /usr/lib64/libee.so.0
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3
> > > > Missing separate debuginfo for /usr/lib64/libestr.so.0
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79
> > > > Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b
> > > > Missing separate debuginfo for
> > > > Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
> > > > /usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602
> > > > Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from
> > > > /usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libz.so.1.2.3
> > > > Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libpthread-2.12.so.debug...done.
> > > > [Thread debugging using libthread_db enabled]
> > > > done.
> > > > Loaded symbols for /lib64/libpthread-2.12.so
> > > > Reading symbols from /lib64/libdl-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libdl-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libdl-2.12.so
> > > > Reading symbols from /lib64/librt-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/librt-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/librt-2.12.so
> > > > Reading symbols from /usr/lib64/libee.so.0...done.
> > > > Loaded symbols for /usr/lib64/libee.so.0
> > > > Reading symbols from /usr/lib64/libestr.so.0...done.
> > > > Loaded symbols for /usr/lib64/libestr.so.0
> > > > Reading symbols from /lib64/libm-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libm-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libm-2.12.so
> > > > Reading symbols from /lib64/libc-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libc-2.12.so
> > > > Reading symbols from /lib64/ld-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/ld-2.12.so
> > > > Reading symbols from /usr/lib64/rsyslog/lmnet.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/lmnet.so
> > > > Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/imuxsock.so
> > > > Reading symbols from /usr/lib64/rsyslog/imklog.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/imklog.so
> > > > Reading symbols from /usr/lib64/rsyslog/imudp.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/imudp.so
> > > > Reading symbols from /usr/lib64/rsyslog/imtcp.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/imtcp.so
> > > > Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so
> > > > Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so
> > > > Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done.
> > > > Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so
> > > > Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libnss_files-2.12.so
> > > > Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libnss_dns-2.12.so
> > > > Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from
> > > > /usr/lib/debug/lib64/libresolv-2.12.so.debug...done.
> > > > done.
> > > > Loaded symbols for /lib64/libresolv-2.12.so
> > > > Core was generated by `/usr/sbin/rsyslogd -f
> > /etc/rsyslog.conf.v6.orig'.
> > > > Program terminated with signal 11, Segmentation fault.
> > > > #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > > > 5161 nextsize = chunksize(nextchunk);
> > > > (gdb) info thread
> > > > 10 Thread 0x7f8dda783700 (LWP 32495) 0x00007f8ddccf22c3 in
> > epoll_wait ()
> > > > at ../sysdeps/unix/syscall-template.S:82
> > > > 9 Thread 0x7f8dd9d82700 (LWP 32496) 0x00007f8ddccf22c3 in
> > epoll_wait ()
> > > > at ../sysdeps/unix/syscall-template.S:82
> > > > 8 Thread 0x7f8ddb184700 (LWP 32494) 0x00007f8ddda474ed in read () at
> > > > ../sysdeps/unix/syscall-template.S:82
> > > > 7 Thread 0x7f8ddbb85700 (LWP 32493) 0x00007f8ddccea8b3 in select ()
> > at
> > > > ../sysdeps/unix/syscall-template.S:82
> > > > 6 Thread 0x7f8dde07d700 (LWP 32492) 0x00007f8ddccea8b3 in select ()
> > at
> > > > ../sysdeps/unix/syscall-template.S:82
> > > > 5 Thread 0x7f8dddfe5700 (LWP 32501)
> pthread_cond_wait@@GLIBC_2.3.2
> > > ()
> > > > at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 4 Thread 0x7f8dde017700 (LWP 32500)
> pthread_cond_wait@@GLIBC_2.3.2
> > > ()
> > > > at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 3 Thread 0x7f8dde049700 (LWP 32499)
> pthread_cond_wait@@GLIBC_2.3.2
> > > ()
> > > > at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 2 Thread 0x7f8dde07b700 (LWP 32498)
> pthread_cond_wait@@GLIBC_2.3.2
> > > ()
> > > > at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > * 1 Thread 0x7f8dd9381700 (LWP 32497) malloc_consolidate
> > > > (av=0x7f8dcc000020) at malloc.c:5161
> > > > (gdb) thread 1
> > > > [Switching to thread 1 (Thread 0x7f8dd9381700 (LWP 32497))]#0
> > > > malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > > > 5161 nextsize = chunksize(nextchunk);
> > > > (gdb) bt
> > > > #0 malloc_consolidate (av=0x7f8dcc000020) at malloc.c:5161
> > > > #1 0x00007f8ddcc83cd8 in _int_free (av=0x7f8dcc000020,
> > p=0x7f8dcc0070c0,
> > > > have_lock=0) at malloc.c:5040
> > > > #2 0x00007f8ddcc71a1d in _IO_new_fclose (fp=0x7f8dcc0070d0) at
> > > > iofclose.c:88
> > > > #3 0x00007f8dd877750e in internal_endent (addr=<value optimized out>,
> > > > len=<value optimized out>, af=2, result=0x7f8dd937f040,
> > > > buffer=0x7f8dd937ea10 "\177", buflen=1024, errnop=0x7f8dd93816a8,
> > > > herrnop=0x7f8dd937f07c)
> > > > at nss_files/files-XXX.c:162
> > > > #4 _nss_files_gethostbyaddr_r (addr=<value optimized out>, len=<value
> > > > optimized out>, af=2, result=0x7f8dd937f040, buffer=0x7f8dd937ea10
> > "\177",
> > > > buflen=1024, errnop=0x7f8dd93816a8, herrnop=0x7f8dd937f07c) at
> > > > nss_files/files-hosts.c:361
> > > > #5 0x00007f8ddcd0c228 in __gethostbyaddr_r (addr=<value optimized
> > out>,
> > > > len=4, type=2, resbuf=0x7f8dd937f040, buffer=0x7f8dd937ea10 "\177",
> > > > buflen=1024, result=0x7f8dd937f070, h_errnop=0x7f8dd937f07c) at
> > > > ../nss/getXXbyYY_r.c:253
> > > > #6 0x00007f8ddcd15893 in getnameinfo (sa=0x7f8dd0000b20,
> > addrlen=<value
> > > > optimized out>, host=0x7f8dd937f0e0 "core-ca.han.nl", hostlen=1025,
> > > > serv=0x0, servlen=0, flags=8) at getnameinfo.c:223
> > > > #7 0x000000000041dcac in mygetnameinfo (addr=0x7f8dd0000b20,
> > > > pEtry=0x7f8dd937fac8) at dnscache.c:141
> > > > #8 resolveAddr (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> > > > dnscache.c:179
> > > > #9 addEntry (addr=0x7f8dd0000b20, pEtry=0x7f8dd937fac8) at
> > > > dnscache.c:286 \177", ip=0x7f8dd9\177",
> > > > pszIP=<value optimized out>) at net.c:1134
> > > > #12 0x000000000042569c in resolveDNS (pMsg=0x7f8dd00008e0) at
> > > > msg.c:4327f8dd9380350 "", pszHostFQDN=0x7f8dd937fb30 "\340\t8Ù
> > > > #13 0x0000000000425b9d in getHOSTNAME (pM=0x7f8dd00008e0) at
> > > > msg.c:1828
> > > > #14 0x0000000000429036 in MsgGetProp (pMsg=0x7f8dd00008e0,
> pTpe=0x0,
> > > > propid=<value optimized out>, propName=<value optimized out>,
> > > > pPropLen=0x7f8dd9380920, pbMustBeFreed=0x7f8dd938092c) at
> msg.c:2608
> > > > #15 0x000000000042a7dc in msgGetMsgVarNew (pThis=0x7f8dd00008e0,
> > > > name=<value optimized out>) at msg.c:3429
> > > > #16 0x0000000000421775 in cnfGetVar (name=0x1ae7800 "$hostname",
> > > > usrptr=<value optimized out>) at rsconf.c:459
> > > > #17 0x0000000000419608 in cnfexprEval (expr=0x1ae7240,
> > > > ret=0x7f8dd93809d0,
> > > > usrptr=0x7f8dd00008e0) at rainerscript.c:1166
> > > > #18 0x0000000000419287 in cnfexprEval (expr=0x1ae7350,
> > > > ret=0x7f8dd9380a30,
> > > > usrptr=0x7f8dd00008e0) at rainerscript.c:901
> > > > #19 0x0000000000418f96 in cnfexprEval (expr=0x1ae9a00,
> > > > ret=0x7f8dd9380a90,
> > > > usrptr=0x7f8dd00008e0) at rainerscript.c:1123
> > > > #20 0x0000000000419d8f in cnfexprEvalBool (expr=<value optimized out>,
> > > > usrptr=<value optimized out>) at rainerscript.c:1210
> > > > #21 0x00000000004413ec in shouldProcessThisMessage
> (pThis=0x1aea160,
> > > > pBatch=0x1b07f00) at rule.c:181
> > > > #22 processBatch (pThis=0x1aea160, pBatch=0x1b07f00) at rule.c:281
> > > > #23 0x000000000043ff22 in processBatchDoRules (pData=0x1aea160,
> > > > pParam=0x1b07f00) at ruleset.c:147
> > > > #24 0x000000000042aec7 in llExecFunc (pThis=0x1ad3a68,
> pFunc=0x43fee0
> > > > <processBatchDoRules>, pParam=0x1b07f00) at linkedlist.c:389
> > > > #25 0x00000000004401fa in processBatch (pBatch=0x1b07f00) at
> > > ruleset.c:229
> > > > #26 0x000000000040d056 in msgConsumer (notNeeded=<value optimized
> > > out>,
> > > > pBatch=0x1b07f00, pbShutdownImmediate=0x1af4200) at syslogd.c:608
> > > > #27 0x000000000043f42b in ConsumerReg (pThis=0x1af41e0,
> > > pWti=0x1b07ed0)
> > > > at
> > > > queue.c:1791
> > > > #28 0x000000000043a415 in wtiWorker (pThis=0x1b07ed0) at wti.c:313
> > > > #29 0x0000000000438efa in wtpWorker (arg=0x1b07ed0) at wtp.c:387
> > > > #30 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9381700) at
> > > > pthread_create.c:301
> > > > #31 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 2
> > > > [Switching to thread 2 (Thread 0x7f8dde07b700 (LWP 32498))]#0
> > > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 162 62: movl (%rsp), %edi
> > > > (gdb) bt
> > > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93da0) at tcpsrv.c:629
> > > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde07b700) at
> > > > pthread_create.c:301
> > > > #3 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 3
> > > > [Switching to thread 3 (Thread 0x7f8dde049700 (LWP 32499))]#0
> > > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 162 62: movl (%rsp), %edi
> > > > (gdb) bt
> > > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e08) at tcpsrv.c:629
> > > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde049700) at
> > > > pthread_create.c:301
> > > > #3 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 4
> > > > [Switching to thread 4 (Thread 0x7f8dde017700 (LWP 32500))]#0
> > > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 162 62: movl (%rsp), %edi
> > > > (gdb) bt
> > > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93e70) at tcpsrv.c:629
> > > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dde017700) at
> > > > pthread_create.c:301
> > > > #3 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 5
> > > > [Switching to thread 5 (Thread 0x7f8dddfe5700 (LWP 32501))]#0
> > > > pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > 162 62: movl (%rsp), %edi
> > > > (gdb) bt
> > > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> > > > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
> > > > #1 0x00007f8ddbd916bf in wrkr (myself=0x7f8ddbf93ed8) at tcpsrv.c:629
> > > > #2 0x00007f8ddda407f1 in start_thread (arg=0x7f8dddfe5700) at
> > > > pthread_create.c:301
> > > > #3 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 6
> > > > [Switching to thread 6 (Thread 0x7f8dde07d700 (LWP 32492))]#0
> > > > 0x00007f8ddccea8b3 in select () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > > (gdb) bt
> > > > #0 0x00007f8ddccea8b3 in select () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > #1 0x000000000040e36d in mainloop (argc=<value optimized out>,
> > > argv=<value
> > > > optimized out>) at syslogd.c:1314
> > > > #2 realMain (argc=<value optimized out>, argv=<value optimized out>)
> > at
> > > > syslogd.c:2061
> > > > #3 0x00007f8ddcc2acdd in __libc_start_main (main=0x40e750 <main>,
> > > argc=3,
> > > > ubp_av=0x7fff18777008, init=<value optimized out>, fini=<value
> > optimized
> > > > out>, rtld_fini=<value optimized out>, stack_end=0x7fff18776ff8) at
> > > > libc-start.c:226
> > > > #4 0x000000000040bf89 in _start ()
> > > > (gdb) thread 7
> > > > [Switching to thread 7 (Thread 0x7f8ddbb85700 (LWP 32493))]#0
> > > > 0x00007f8ddccea8b3 in select () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > > (gdb) bt
> > > > #0 0x00007f8ddccea8b3 in select () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > #1 0x00007f8ddc7b58bb in runInput (pThrd=<value optimized out>) at
> > > > imuxsock.c:1140
> > > > #2 0x000000000044857a in thrdStarter (arg=0x1b08ce0) at
> > ../threads.c:210
> > > > #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddbb85700) at
> > > > pthread_create.c:301
> > > > #4 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 8
> > > > [Switching to thread 8 (Thread 0x7f8ddb184700 (LWP 32494))]#0
> > > > 0x00007f8ddda474ed in read () at ../sysdeps/unix/syscall-template.S:82
> > > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > > (gdb) bt
> > > > #0 0x00007f8ddda474ed in read () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > #1 0x00007f8ddc5a23d8 in readklog () at bsd.c:224
> > > > #2 0x00007f8ddc5a2509 in klogLogKMsg (pModConf=<value optimized
> out>)
> > > at
> > > > bsd.c:282
> > > > #3 0x00007f8ddc5a1b73 in runInput (pThrd=0x1b08ed0) at imklog.c:271
> > > > #4 0x000000000044857a in thrdStarter (arg=0x1b08ed0) at
> > ../threads.c:210
> > > > #5 0x00007f8ddda407f1 in start_thread (arg=0x7f8ddb184700) at
> > > > pthread_create.c:301
> > > > #6 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 9
> > > > [Switching to thread 9 (Thread 0x7f8dd9d82700 (LWP 32496))]#0
> > > > 0x00007f8ddccf22c3 in epoll_wait () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > > (gdb) bt
> > > > #0 0x00007f8ddccf22c3 in epoll_wait () at
> > > > ../sysdeps/unix/syscall-template.S:82
> > > > #1 0x00007f8ddbb89a4f in Wait (pNsdpoll=0x7f8dd4000e10, timeout=-1,
> > > > numEntries=0x7f8dd9d81dbc, workset=0x7f8dd9d81170) at
> > > nsdpoll_ptcp.c:242
> > > > #2 0x00007f8ddbd9138c in Run (pThis=0x1adebf0) at tcpsrv.c:884
> > > > #3 0x000000000044857a in thrdStarter (arg=0x1b092b0) at
> > ../threads.c:210
> > > > #4 0x00007f8ddda407f1 in start_thread (arg=0x7f8dd9d82700) at
> > > > pthread_create.c:301
> > > > #5 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb) thread 10
> > > > [Switching to thread 10 (Thread 0x7f8dda783700 (LWP 32495))]#0
> > > > 0x00007f8ddccf22c3 in epoll_wait () at
> > ../sysdeps/unix/syscall-template.S:82
> > > > 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
> > > > (gdb) bt
> > > > #0 0x00007f8ddccf22c3 in epoll_wait () at
> > > > ../sysdeps/unix/syscall-template.S:82
> > > > #1 0x00007f8ddc39e0a8 in rcvMainLoop (pThrd=0x1b090c0) at
> imudp.c:593
> > > > #2 0x000000000044857a in thrdStarter (arg=0x1b090c0) at
> > ../threads.c:210
> > > > #3 0x00007f8ddda407f1 in start_thread (arg=0x7f8dda783700) at
> > > > pthread_create.c:301
> > > > #4 0x00007f8ddccf1ccd in clone () at
> > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> > > > (gdb)
> > > >
> > > >
> > > > [root [at] spi etc]# /usr/sbin/rsyslogd -v2
> > > > rsyslogd 6.3.10, compiled with:
> > > > FEATURE_REGEXP: Yes
> > > > FEATURE_LARGEFILE: No
> > > > GSSAPI Kerberos 5 support: No
> > > > FEATURE_DEBUG (debug build, slow code): Yes
> > > > 32bit Atomic operations supported: Yes
> > > > 64bit Atomic operations supported: Yes
> > > > Runtime Instrumentation (slow code): No
> > > >
> > > >
> > > >
> > > >
> > > > 2012/6/11 Rainer Gerhards <rgerhards [at] hq>
> > > >
> > > > > Does it work without ommongodb? If so, have you made sure you use
> the
> > > > > newest version of libmongo-client? I notice the crash points into
> > its code
> > > > > (that doesn't mean libmonogo-client has a problem - but I know that
> > there
> > > > > were a couple of fixes, so better check...).
> > > > >
> > > > > Rainer
> > > > >
> > > >
> > > > .
> > > > .
> > > > .
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.