Mailing List Archive: OpenSSH: Dev

Static build segfaults on x86_64

Index | Next | Previous | View Threaded

bostjan at a2o

Dec 28, 2009, 5:51 PM

Post #1 of 4 (1700 views)
Permalink

Static build segfaults on x86_64

Hello everyone,

I would like to ask you for advice on how to approach (or solve) this
particular problem.

I use Slackware Linux and compile Openssh from source. I prefer to
compile it statically so it doesn't get messed up if I update openssl
libraries. Up until now this approach was working OK for me.

Lately I have been challenged with Slackware64 installations and I
have come across a problem with Openssh version (5.3p1, but result is
the same with 5.2p1). What happens is that sshd daemon keeps accepting
connections as long as no one disconnects. On the first DISconnection
the server daemon dies with segfault message, of which a strace output
I have included below. This only happens on 64-bit platform, on 32-bit
it is still working hanky-dory. Non-static build works ok on both
platforms.

Sum:
32-bit shared: OK
32-bit static: OK
64-bit shared: OK
64-bit static: SEGFAULT on first client disconnect

How should I start solving this problem? I am proficient in PHP and
other Untyped languages but I am only moderately familiar with C
programming.

Thank you for your responses,
Bostjan Skufca

--------------[ Segfault output from strace ]-------------------------------
# strace -p 21976
Process 21976 attached - interrupt to quit
select(6, [3], NULL, NULL, NULL) = ? ERESTARTNOHAND (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], WNOHANG, NULL) = 21984
wait4(-1, 0x7fffc6243994, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigaction(SIGCHLD, NULL, {0x400be0, [], SA_RESTORER,
0xf0000000fc0c748}, 8) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 21976 detached
-----------------------------------------------------------------------------------

--------------[ Environment description ]-------------------------------
- kernel: 2.6.32.2 (64 bit)
- OS: Slackware64-13
- gcc: from distro (whole toolchain actually)
- libs: glibc is from distro
- libs: openssl is customly compiled 0.9.8l (./config
--prefix=$PDESTDIR_OPENSSL shared)
- openssh: 5.3p1 configured like this:

./configure --prefix=$PDESTDIR \
--sysconfdir=/etc/ssh \
--with-ssl-dir=$PDESTDIR_OPENSSL \
--with-ldflags=-static \
--with-privsep-path=/var/empty \
--with-default-path=...

Using "LDFLAGS=-static" produces the same results.
------------------------------------------------------------------------------------
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dtucker at zip

Dec 29, 2009, 1:14 PM

Post #2 of 4 (1628 views)
Permalink

Re: Static build segfaults on x86_64 [In reply to]

Bostjan Skufca wrote:
> Hello everyone,
>
> I would like to ask you for advice on how to approach (or solve) this
> particular problem.
>
> I use Slackware Linux and compile Openssh from source. I prefer to
> compile it statically so it doesn't get messed up if I update openssl
> libraries. Up until now this approach was working OK for me.

As an aside: you don't need to statically link the whole binary just to
get the crypto libs. If you build OpenSSL with just the static library
(libcrypto.a) and configure OpenSSH --with-ssl-dir=thatdir then the
linker will pick up the crypto functions from the static library and the
remainder from the system dynamic libraries.

> Lately I have been challenged with Slackware64 installations and I
> have come across a problem with Openssh version (5.3p1, but result is
> the same with 5.2p1). What happens is that sshd daemon keeps accepting
> connections as long as no one disconnects. On the first DISconnection
> the server daemon dies with segfault message, of which a strace output
> I have included below.

Unfortunately the strace does not show anything useful, but from your
description it sounds like it's crashing in the SIGCHLD handler,
although I don't know why it would show up only with static linking.

You might get something more useful from enabling sshd's debugging, eg

# /path/to/sshd -o LogLevel=debug3 -D

[...]
> How should I start solving this problem? I am proficient in PHP and
> other Untyped languages but I am only moderately familiar with C
> programming.

Did it generate a core dump? if so, your first step is to feed it into
gdb and generate a backtrace, which will tell you where it crashed. You do:

$ gdb /path/to/sshd core
(gdb) bt

If it doesn't generate a core dump (eg because the ulimit prevents it)
then you can attach gdb to the process you expect to crash, wait for it
to crash then generate the backtrace

$ gdb /path/to/sshd [pid of sshd]
[wait for crash]
(gdb) bt

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

bostjan at a2o

Dec 29, 2009, 7:07 PM

Post #3 of 4 (1626 views)
Permalink

Re: Static build segfaults on x86_64 [In reply to]

Thanks for the tip on building with only static instance of OpenSSL.

No core dumps were produced, and so far I was unable to get the daemon
working with gdb attached to it.
I compiled it statically as before and with --disable-strip. In
standalone (no gdb) it produces the same results.
Then the commands I execute are these:

(start the daemon)
# /usr/local/ssh/sbin/sshd

(attach the gdb to process)
# gdb /usr/local/ssh/sbin/sshd PID

Then I try to connect but the client just hangs waiting for a response
from server. If I attach to sshd process with 'strace' it works just
fine. Therefore I am unable to get to the point where I can exit the
first child, wait for the server crash and get a backtrace. Any
additional hints?

b.

2009/12/29 Darren Tucker <dtucker [at] zip>:
> Bostjan Skufca wrote:
>>
>> Hello everyone,
>>
>> I would like to ask you for advice on how to approach (or solve) this
>> particular problem.
>>
>> I use Slackware Linux and compile Openssh from source. I prefer to
>> compile it statically so it doesn't get messed up if I update openssl
>> libraries. Up until now this approach was working OK for me.
>
> As an aside: you don't need to statically link the whole binary just to get
> the crypto libs. �If you build OpenSSL with just the static library
> (libcrypto.a) and configure OpenSSH --with-ssl-dir=thatdir then the linker
> will pick up the crypto functions from the static library and the remainder
> from the system dynamic libraries.
>
>> Lately I have been challenged with Slackware64 installations and I
>> have come across a problem with Openssh version (5.3p1, but result is
>> the same with 5.2p1). What happens is that sshd daemon keeps accepting
>> connections as long as no one disconnects. On the first DISconnection
>> the server daemon dies with segfault message, of which a strace output
>> I have included below.
>
> Unfortunately the strace does not show anything useful, but from your
> description it sounds like it's crashing in the SIGCHLD handler, although I
> don't know why it would show up only with static linking.
>
> You might get something more useful from enabling sshd's debugging, eg
>
> # /path/to/sshd -o LogLevel=debug3 -D
>
> [...]
>>
>> How should I start solving this problem? I am proficient in PHP and
>> other Untyped languages but I am only moderately familiar with C
>> programming.
>
> Did it generate a core dump? �if so, your first step is to feed it into gdb
> and generate a backtrace, which will tell you where it crashed. �You do:
>
> $ gdb /path/to/sshd core
> (gdb) bt
>
> If it doesn't generate a core dump (eg because the ulimit prevents it) then
> you can attach gdb to the process you expect to crash, wait for it to crash
> then generate the backtrace
>
> $ gdb /path/to/sshd [pid of sshd]
> [wait for crash]
> (gdb) bt
>
> --
> Darren Tucker (dtucker at zip.com.au)
> GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 �37C9 C982 80C7 8FF4 FA69
> � �Good judgement comes with experience. Unfortunately, the experience
> usually comes from bad judgement.
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dtucker at zip

Dec 29, 2009, 7:33 PM

Post #4 of 4 (1616 views)
Permalink

Re: Static build segfaults on x86_64 [In reply to]

Bostjan Skufca wrote:
> Thanks for the tip on building with only static instance of OpenSSL.
>
> No core dumps were produced, and so far I was unable to get the daemon
> working with gdb attached to it.
> I compiled it statically as before and with --disable-strip. In
> standalone (no gdb) it produces the same results.
> Then the commands I execute are these:
>
> (start the daemon)
> # /usr/local/ssh/sbin/sshd
>
> (attach the gdb to process)
> # gdb /usr/local/ssh/sbin/sshd PID
>
> Then I try to connect but the client just hangs waiting for a response
> from server. If I attach to sshd process with 'strace' it works just
> fine. Therefore I am unable to get to the point where I can exit the
> first child, wait for the server crash and get a backtrace. Any
> additional hints?

gdb probably got confused when the server forked. Assuming my
understanding of your description is correct, I suggest trying in the
following order:

start sshd. Note pid.
connect with client
attach gdb to pid previously noted.
disconnect client.

alternatively you could try what you did before with "set follow-fork
parent" in gdb before connecting with the ssh client.

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads