Mailing List Archive: OpenSSH: Dev

ssh 'connection reset by peer' problem since 5.8p1

1 2

View All

Index | Next | Previous | View Threaded

oren at held

Feb 16, 2011, 2:27 AM

Post #1 of 26 (11573 views)
Permalink

ssh 'connection reset by peer' problem since 5.8p1

Hi,

As I haven't seen any related discussion here (or bug in bugzilla), I thought
it'll be good to do a heads-up for a recently introduced problem in 5.8p1 (or at
least after 5.5.x).

Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
Arch Linux: https://bugs.archlinux.org/task/22897?project=1

Any other information needed? Shall I file a bug in bugzilla?

Best Regards

Oren
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dtucker at zip

Feb 16, 2011, 2:46 AM

Post #2 of 26 (11496 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On 16/02/11 9:27 PM, Oren Held wrote:
> Hi,
>
> As I haven't seen any related discussion here (or bug in bugzilla), I thought
> it'll be good to do a heads-up for a recently introduced problem in 5.8p1 (or at
> least after 5.5.x).
>
> Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
> Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
> Arch Linux: https://bugs.archlinux.org/task/22897?project=1
>
> Any other information needed? Shall I file a bug in bugzilla?

Can you reproduce the problem with an unmodified tarball from openssh.com?

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 16, 2011, 3:03 AM

Post #3 of 26 (11479 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Wed, Feb 16, 2011 at 09:46:07PM +1100, Darren Tucker wrote:
> On 16/02/11 9:27 PM, Oren Held wrote:
> >Hi,
> >
> >As I haven't seen any related discussion here (or bug in bugzilla), I thought
> >it'll be good to do a heads-up for a recently introduced problem in 5.8p1 (or at
> >least after 5.5.x).
> >
> >Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
> >Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
> >Arch Linux: https://bugs.archlinux.org/task/22897?project=1
> >
> >Any other information needed? Shall I file a bug in bugzilla?
>
> Can you reproduce the problem with an unmodified tarball from openssh.com?

Yes. I've just compiled a "vanilla" 5.8p1, and same result:

orenhe [at] orenhe-lapto:/tmp/openssh-5.8p1$ /opt/ssh/bin/ssh root [at] blade-8
Read from socket failed: Connection reset by peer

As for my compile flags, I used simple ./configure --prefix=/opt/ssh:
User binaries: /opt/ssh/bin
System binaries: /opt/ssh/sbin
Configuration files: /opt/ssh/etc
Askpass program: /opt/ssh/libexec/ssh-askpass
Manual pages: /opt/ssh/share/man/manX
PID file: /var/run
Privilege separation chroot path: /var/empty
sshd default user PATH: /usr/bin:/bin:/usr/sbin:/sbin:/opt/ssh/bin
Manpage format: doc
PAM support: no
OSF SIA support: no
KerberosV support: no
SELinux support: no
Smartcard support:
S/KEY support: no
TCP Wrappers support: no
MD5 password support: no
libedit support: no
Solaris process contract support: no
Solaris project support: no
IP address in $DISPLAY hack: no
Translate v4 in v6 hack: yes
BSD Auth support: no
Random number source: OpenSSL internal ONLY
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dtucker at zip

Feb 16, 2011, 5:27 AM

Post #4 of 26 (11436 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On 16/02/11 10:03 PM, Oren Held wrote:
> On Wed, Feb 16, 2011 at 09:46:07PM +1100, Darren Tucker wrote:
>> On 16/02/11 9:27 PM, Oren Held wrote:
>>> Hi,
>>>
>>> As I haven't seen any related discussion here (or bug in bugzilla), I thought
>>> it'll be good to do a heads-up for a recently introduced problem in 5.8p1 (or at
>>> least after 5.5.x).
>>>
>>> Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
>>> Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
>>> Arch Linux: https://bugs.archlinux.org/task/22897?project=1
>>>
>>> Any other information needed? Shall I file a bug in bugzilla?
>>
>> Can you reproduce the problem with an unmodified tarball from openssh.com?
>
> Yes. I've just compiled a "vanilla" 5.8p1, and same result:
>
> orenhe [at] orenhe-lapto:/tmp/openssh-5.8p1$ /opt/ssh/bin/ssh root [at] blade-8
> Read from socket failed: Connection reset by peer

When this happens, does it happen immediately or is there a delay before
you get the "reset by peer" ?

Could you please collect both server and client side debug output?
(note that you'll need to use sshd -ddde to make sure you get all the
server output on stderr)

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 16, 2011, 5:52 AM

Post #5 of 26 (11418 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Thu, Feb 17, 2011 at 12:27:04AM +1100, Darren Tucker wrote:
> On 16/02/11 10:03 PM, Oren Held wrote:
> >On Wed, Feb 16, 2011 at 09:46:07PM +1100, Darren Tucker wrote:
> >>On 16/02/11 9:27 PM, Oren Held wrote:
> >>>Hi,
> >>>
> >>>As I haven't seen any related discussion here (or bug in bugzilla), I thought
> >>>it'll be good to do a heads-up for a recently introduced problem in 5.8p1 (or at
> >>>least after 5.5.x).
> >>>
> >>>Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
> >>>Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
> >>>Arch Linux: https://bugs.archlinux.org/task/22897?project=1
> >>>
> >>>Any other information needed? Shall I file a bug in bugzilla?
> >>
> >>Can you reproduce the problem with an unmodified tarball from openssh.com?
> >
> >Yes. I've just compiled a "vanilla" 5.8p1, and same result:
> >
> >orenhe [at] orenhe-lapto:/tmp/openssh-5.8p1$ /opt/ssh/bin/ssh root [at] blade-8
> >Read from socket failed: Connection reset by peer
>
> When this happens, does it happen immediately or is there a delay
> before you get the "reset by peer" ?
Yes. It's very quick.

> Could you please collect both server and client side debug output?
> (note that you'll need to use sshd -ddde to make sure you get all
> the server output on stderr)

Attaching two files, one for the server's log and one for the client's.

One more note: the problem occurs when SSHing to *older* servers (I tried 4.3p2,
5.1p1). When SSHing to v5.8p1 it works smoothly.

Attachments:

sshd-side.txt (2.33 KB)

ssh-side.txt (4.62 KB)

dtucker at zip

Feb 16, 2011, 6:42 AM

Post #6 of 26 (11398 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On 17/02/11 12:52 AM, Oren Held wrote:
[...]
> One more note: the problem occurs when SSHing to *older* servers (I tried 4.3p2,
> 5.1p1). When SSHing to v5.8p1 it works smoothly.

I have tried to reproduce with fresh-built 4.3p1 and 5.1p1 servers and
failed. Are the servers configured with any non-default options or
source code changes?

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 16, 2011, 1:15 PM

Post #7 of 26 (11398 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Thu, Feb 17, 2011 at 01:42:45AM +1100, Darren Tucker wrote:
> On 17/02/11 12:52 AM, Oren Held wrote:
> [...]
> >One more note: the problem occurs when SSHing to *older* servers (I tried 4.3p2,
> >5.1p1). When SSHing to v5.8p1 it works smoothly.
>
> I have tried to reproduce with fresh-built 4.3p1 and 5.1p1 servers
> and failed. Are the servers configured with any non-default options
> or source code changes?

New findings: it happens only when compiling and running *the server* on relatively old environments:
e.g. RedHat 5.5, and I think that also Debian Lenny (5.0.8).

While the client is running on a new environment (Debian sid, aka unstable)

It even happened when connecting from 5.8p1 on my Debian unstable to 5.8p1 (same
version!) on RHEL5.5.

- Both are x86_64 OSes.
- Both were vanilla openssh that I compiled by myself with no special compile parameters.
- (Server) Happened even when sshd_config was empty but also with the default RHEL one
- (Client) happened with empty ~/.ssh and no /etc/ssh_config

I'm starting to suspect it has to do with some library and not openssh itself.
ldd spotted some notable difference:
Server is linked to libcrypto.so.6 (RHEL5 default)
Client is linkde to libcrypto.so.0.9.8 (Debian sid default)

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

cyrille.lefevre-lists at laposte

Feb 16, 2011, 1:35 PM

Post #8 of 26 (11484 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

Le 16/02/2011 14:52, Oren Held a �crit :
>> Could you please collect both server and client side debug output?
>> (note that you'll need to use sshd -ddde to make sure you get all
>> the server output on stderr)

on the server side, binding to IPv6 seems to be ok.

debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.

but why can't you bind to port 2222 IPv4 ?

debug1: Bind to port 2222 on 0.0.0.0.
Bind to port 2222 on 0.0.0.0 failed: Address already in use.

could you try another port or kill the other process waiting on this
port and try again ?

your client side try to connect to IPv4 which isn't listenning, so the
connexion is dropped.

debug1: Connecting to blade-8.ps [9.151.146.103] port 2222.

try ssh -6 if possible.

Regards,

Cyrille Lefevre
--
mailto:Cyrille.Lefevre-lists [at] laposte

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 16, 2011, 3:30 PM

Post #9 of 26 (11390 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Wed, Feb 16, 2011 at 11:15:35PM +0200, Oren Held wrote:
> On Thu, Feb 17, 2011 at 01:42:45AM +1100, Darren Tucker wrote:
> > On 17/02/11 12:52 AM, Oren Held wrote:
> > [...]
> > >One more note: the problem occurs when SSHing to *older* servers (I tried 4.3p2,
> > >5.1p1). When SSHing to v5.8p1 it works smoothly.
> >
> > I have tried to reproduce with fresh-built 4.3p1 and 5.1p1 servers
> > and failed. Are the servers configured with any non-default options
> > or source code changes?

[...]
>
> I'm starting to suspect it has to do with some library and not openssh itself.
> ldd spotted some notable difference:
> Server is linked to libcrypto.so.6 (RHEL5 default)
> Client is linkde to libcrypto.so.0.9.8 (Debian sid default)

Additionally, I've refined the diff's: 5.6p1 connects perfectly well to all
ssh servers, the problem begins with 5.7p1.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dtucker at zip

Feb 16, 2011, 7:00 PM

Post #10 of 26 (11477 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On 17/02/11 8:35 AM, Cyrille Lefevre wrote:
> debug1: Bind to port 2222 on ::.
> Server listening on :: port 2222.
>
> but why can't you bind to port 2222 IPv4 ?

On some platforms (eg Linux) listening on an IPv6 socket will also get
you IPv4 connections as mapped 4-in-6 addresses. sshd will latter unmap
it (look for ipv64_normalise_mapped in canohost.c).

It's not related to the problem in this thread.

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

vinschen at redhat

Feb 17, 2011, 3:34 AM

Post #11 of 26 (11454 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Feb 17 01:30, Oren Held wrote:
> On Wed, Feb 16, 2011 at 11:15:35PM +0200, Oren Held wrote:
> > On Thu, Feb 17, 2011 at 01:42:45AM +1100, Darren Tucker wrote:
> > > On 17/02/11 12:52 AM, Oren Held wrote:
> > > [...]
> > > >One more note: the problem occurs when SSHing to *older* servers (I tried 4.3p2,
> > > >5.1p1). When SSHing to v5.8p1 it works smoothly.
> > >
> > > I have tried to reproduce with fresh-built 4.3p1 and 5.1p1 servers
> > > and failed. Are the servers configured with any non-default options
> > > or source code changes?
>
> [...]
> >
> > I'm starting to suspect it has to do with some library and not openssh itself.
> > ldd spotted some notable difference:
> > Server is linked to libcrypto.so.6 (RHEL5 default)
> > Client is linkde to libcrypto.so.0.9.8 (Debian sid default)
>
> Additionally, I've refined the diff's: 5.6p1 connects perfectly well to all
> ssh servers, the problem begins with 5.7p1.

As an additional datapoint, we had a couple of similar bug reports after
I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
them even comes with a set of debug output of working (5.6p1) and
non-working (5.8p1) connection attempts:

http://cygwin.com/ml/cygwin/2011-02/msg00317.html

Another one just shows the output of a failed attempt:

http://cygwin.com/ml/cygwin/2011-02/msg00233.html

However, I tried with various older versions of SSH running on Cygwin,
Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
reproduce this problem.

Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dtucker at zip

Feb 17, 2011, 4:27 AM

Post #12 of 26 (11375 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On 17/02/2011 10:34 PM, Corinna Vinschen wrote:
> As an additional datapoint, we had a couple of similar bug reports after
> I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
> them even comes with a set of debug output of working (5.6p1) and
> non-working (5.8p1) connection attempts:
[...]
> However, I tried with various older versions of SSH running on Cygwin,
> Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
> reproduce this problem.

Thanks for the extra info. I haven't been able to reproduce either.
I've tried building 5.5p1 and 4.3p1 against (locally built) OpenSSL
0.9.6b and 0.9.8d. There seems to be some piece of the puzzle missing...

I diffed the working and non working clients, and one difference is:
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

although I'm not sure that's significant since Oren's output had
SSH2_MSG_KEX_DH_GEX_GROUP. You could try forcing it with "ssh -vvv -o
KexAlgorithms=diffie-hellman-group-exchange-sha1 server"

(aside: I now want to add OpenSSL's version output to the server debug
output)

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

vinschen at redhat

Feb 17, 2011, 6:25 AM

Post #13 of 26 (12348 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Feb 17 23:27, Darren Tucker wrote:
> On 17/02/2011 10:34 PM, Corinna Vinschen wrote:
> >As an additional datapoint, we had a couple of similar bug reports after
> >I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
> >them even comes with a set of debug output of working (5.6p1) and
> >non-working (5.8p1) connection attempts:
> [...]
> >However, I tried with various older versions of SSH running on Cygwin,
> >Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
> >reproduce this problem.
>
> Thanks for the extra info. I haven't been able to reproduce either.
> I've tried building 5.5p1 and 4.3p1 against (locally built) OpenSSL
> 0.9.6b and 0.9.8d. There seems to be some piece of the puzzle
> missing...

What I'm missing in the debug output is a clear statement of the
side which closes the connection, *why* the connection has been
closed. In Andrew's debug output The server side just contains:

debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Read from socket failed: Software caused connection abort

and the client side just contains:

debug1: SSH2_MSG_KEXINIT sent
Read from socket failed: Connection reset by peer

In Oren's debug output, server:

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Write failed: Connection reset by peer

client:

debug1: SSH2_MSG_KEXINIT sent
debug2: Network child is on pid 1403
debug3: preauth child monitor started
debug3: mm_request_receive entering
Read from socket failed: Connection reset by peer

What happened here? The socket got closed, but why? In theory at
least one side of the connection should know...

Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

cjc5 at cwru

Feb 17, 2011, 7:17 AM

Post #14 of 26 (11661 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

In message <20110217142532.GF29762 [at] calimero>, Corinna Vinschen writes:

>What I'm missing in the debug output is a clear statement of the
>side which closes the connection, *why* the connection has been
>closed. In Andrew's debug output The server side just contains:

I have seen something similar but attributed it to a local error
(undiscovered source). I have 3 OpenBSD machines and 2 Ubuntu
machines all running 5.8. All can ssh to each other EXCEPT to one of
the ubuntu machines. The two ubuntu machines should be identical
(same versions of the distribution, same configuration files, ...).
My "solution" was to put
HostKeyAlgorithms ssh-rsa-cert-v01 [at] openssh,ssh-dss-cert-v01 [at] openssh,ssh-rsa-cert-v00 [at] openssh,ssh-dss-cert-v00 [at] openssh,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
in my ~/.ssh/config file. In particular I found that removing the keys
ecdsa-sha2-nistp256-cert-v01 [at] openssh,
ecdsa-sha2-nistp384-cert-v01 [at] openssh,
ecdsa-sha2-nistp521-cert-v01 [at] openssh
allows for all machines to interconnect.

I don't know why this is the case.

Craig
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 19, 2011, 8:22 AM

Post #15 of 26 (11553 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Thu, Feb 17, 2011 at 10:17:26AM -0500, Craig J Copi wrote:
> In message <20110217142532.GF29762 [at] calimero>, Corinna Vinschen writes:
>
> >What I'm missing in the debug output is a clear statement of the
> >side which closes the connection, *why* the connection has been
> >closed. In Andrew's debug output The server side just contains:
>
> I have seen something similar but attributed it to a local error
> (undiscovered source). I have 3 OpenBSD machines and 2 Ubuntu
> machines all running 5.8. All can ssh to each other EXCEPT to one of
> the ubuntu machines. The two ubuntu machines should be identical
> (same versions of the distribution, same configuration files, ...).
> My "solution" was to put
> HostKeyAlgorithms ssh-rsa-cert-v01 [at] openssh,ssh-dss-cert-v01 [at] openssh,ssh-rsa-cert-v00 [at] openssh,ssh-dss-cert-v00 [at] openssh,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
> in my ~/.ssh/config file. In particular I found that removing the keys
> ecdsa-sha2-nistp256-cert-v01 [at] openssh,
> ecdsa-sha2-nistp384-cert-v01 [at] openssh,
> ecdsa-sha2-nistp521-cert-v01 [at] openssh
> allows for all machines to interconnect.

1. I confirm that above fix works for me also. Alternatively, as reported in
Debian bug #612607, adding '-c aes128-ctr' to the ssh command line does the
trick as well.

2. I narrowed it a bit down: it only occurs when *running* ssh v5.[7,8]p1 client
on my Debian-sid (unstable) box. It's a run-time, and not a compile-time thing.
Copying binaries to and fro my box to Ubuntu 10.10 box proved that it only
matter where it's being run, not where it's being compiled.

In that case, I guess it's something in my environment and not in openssh - but
possibly not openssl, because I tried 0.9.8o on both Ubuntu and Debian, and it
fails only in the latter. However, some change in the code of 5.7p1 had
triggered this problem.

Oren
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

andrew at nglee

Feb 23, 2011, 3:18 AM

Post #16 of 26 (11444 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

Oren Held <oren <at> held.org.il> writes:

> 1. I confirm that above fix works for me also. Alternatively, as reported in
> Debian bug #612607, adding '-c aes128-ctr' to the ssh command line does the
> trick as well.

The '-c aes128-ctr' workaround also works for Cygwin OpenSSH 5.8p1 connection
issues. I tried using the '-c' option with the default list of ciphers from the
SSH man page and this once again caused the connection issue. I then tried
trimming down the list and this appeared to also fix the connection issue.

So could it be somehow related to this list of ciphers?

Regards,
Andrew

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 23, 2011, 8:38 AM

Post #17 of 26 (11432 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Wed, Feb 23, 2011 at 11:18:31AM +0000, Andrew Ng wrote:
> Oren Held <oren <at> held.org.il> writes:
>
> > 1. I confirm that above fix works for me also. Alternatively, as reported in
> > Debian bug #612607, adding '-c aes128-ctr' to the ssh command line does the
> > trick as well.
>
> The '-c aes128-ctr' workaround also works for Cygwin OpenSSH 5.8p1 connection
> issues. I tried using the '-c' option with the default list of ciphers from the
> SSH man page and this once again caused the connection issue. I then tried
> trimming down the list and this appeared to also fix the connection issue.
>
> So could it be somehow related to this list of ciphers?

I've researched it a bit deeper. Surprisingly it's not a matter of which cipher to
choose, but of *how long the list of ciphers is*. I'll explain:
Doesn't work:
-c 'aes128-ctr' and 94 commas (i.e. -c 'aes128-ctr,,,,,,,,,,,,,,,,,,' etc),
Does work:
-c 'aes128-ctr' and 95 commas

Now the number above varies. On my home computer it was 105 commas vs. 104
commas. So eventually I bet it has to do with SSH packet size. For instance in
my place, according to Wireshark, SSH "Client: Key Exchange Init" packet length
is 1044+10(padding) in the bad case, 1036+4 in the good case.

So to sum it all up:
1. Problem started with 5.7p1
2. Occurs on specific, bleeding edge Linux clients (probably some lib involved)
3. Occurs when connecting to specific old Linux ssh servers.
4. Long cipher list triggers the problem, shortening cipher list works around it.
Length of doom is relative to the client or its network configuration.

#2 + #3 are still not clear to me - what environment is needed for reproduction.

Oren
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Feb 23, 2011, 12:24 PM

Post #18 of 26 (11419 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

> > > 1. I confirm that above fix works for me also. Alternatively, as reported in
> > > Debian bug #612607, adding '-c aes128-ctr' to the ssh command line does the
> > > trick as well.
> >
> > The '-c aes128-ctr' workaround also works for Cygwin OpenSSH 5.8p1 connection
> > issues. I tried using the '-c' option with the default list of ciphers from the
> > SSH man page and this once again caused the connection issue. I then tried
> > trimming down the list and this appeared to also fix the connection issue.
> >
> > So could it be somehow related to this list of ciphers?
>
> I've researched it a bit deeper. Surprisingly it's not a matter of which cipher to
> choose, but of *how long the list of ciphers is*. I'll explain:
> Doesn't work:
> -c 'aes128-ctr' and 94 commas (i.e. -c 'aes128-ctr,,,,,,,,,,,,,,,,,,' etc),
> Does work:
> -c 'aes128-ctr' and 95 commas

Just a correction, I meant the opposite: the shorter string (94 in this case) DOES work,
longer string DOESN'T.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

cjwatson at debian

Mar 3, 2011, 6:31 AM

Post #19 of 26 (11373 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Wed, Feb 23, 2011 at 04:40:00PM +0000, Oren Held wrote:
>I've researched it a bit deeper. Surprisingly it's not a matter of which cipher to
>choose, but of *how long the list of ciphers is*. I'll explain:
>Doesn't work:
>-c 'aes128-ctr' and 94 commas (i.e. -c 'aes128-ctr,,,,,,,,,,,,,,,,,,' etc),
>Does work:
>-c 'aes128-ctr' and 95 commas
>
>Now the number above varies. On my home computer it was 105 commas vs. 104
>commas. So eventually I bet it has to do with SSH packet size. For instance in
>my place, according to Wireshark, SSH "Client: Key Exchange Init" packet length
>is 1044+10(padding) in the bad case, 1036+4 in the good case.

What are the MTU values on the relevant network interfaces on the client
and the server?

--
Colin Watson [cjwatson [at] debian]
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

oren at held

Mar 3, 2011, 8:18 AM

Post #20 of 26 (11293 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

On Thu, Mar 03, 2011 at 02:31:58PM +0000, Colin Watson wrote:
> On Wed, Feb 23, 2011 at 04:40:00PM +0000, Oren Held wrote:
> >I've researched it a bit deeper. Surprisingly it's not a matter of which cipher to
> >choose, but of *how long the list of ciphers is*. I'll explain:
> >Doesn't work:
> >-c 'aes128-ctr' and 94 commas (i.e. -c 'aes128-ctr,,,,,,,,,,,,,,,,,,' etc),
> >Does work:
> >-c 'aes128-ctr' and 95 commas
> >
> >Now the number above varies. On my home computer it was 105 commas vs. 104
> >commas. So eventually I bet it has to do with SSH packet size. For instance in
> >my place, according to Wireshark, SSH "Client: Key Exchange Init" packet length
> >is 1044+10(padding) in the bad case, 1036+4 in the good case.
>
> What are the MTU values on the relevant network interfaces on the client
> and the server?

MTU is 1500 on both client and server, in my case.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

vinschen at redhat

Mar 7, 2011, 11:29 AM

Post #21 of 26 (11283 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

Hi Darren,

On Feb 17 23:27, Darren Tucker wrote:
> On 17/02/2011 10:34 PM, Corinna Vinschen wrote:
> >As an additional datapoint, we had a couple of similar bug reports after
> >I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
> >them even comes with a set of debug output of working (5.6p1) and
> >non-working (5.8p1) connection attempts:
> [...]
> >However, I tried with various older versions of SSH running on Cygwin,
> >Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
> >reproduce this problem.
>
> Thanks for the extra info. I haven't been able to reproduce either.
> I've tried building 5.5p1 and 4.3p1 against (locally built) OpenSSL
> 0.9.6b and 0.9.8d. There seems to be some piece of the puzzle
> missing...
>
> I diffed the working and non working clients, and one difference is:
> debug1: sending SSH2_MSG_KEX_ECDH_INIT
> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
>
> although I'm not sure that's significant since Oren's output had
> SSH2_MSG_KEX_DH_GEX_GROUP. You could try forcing it with "ssh -vvv
> -o KexAlgorithms=diffie-hellman-group-exchange-sha1 server"
>
> (aside: I now want to add OpenSSL's version output to the server
> debug output)

is there any progress in that matter?

Thanks,
Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

dyle at dyle

Mar 8, 2011, 1:21 AM

Post #22 of 26 (11417 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

#!/bin/hi

Sorry to interfere, but finally I find someone talking about my problem.

I encountered the very same problem with a mix of Gentoo/Ubuntu/Debian
machines whereas I could not connect from my Gentoo Box (5.8p1) to any
machine behind the firewall in the wild (Debian; 5.1p1). But connecting to
a Ubuntu box right next to me within the very same subnet and then SSH from
this very machine to a machine outside worked. I also could connect to any
machine inside the subnet from my Gentoo/5.8p1.

Then reducing the number of cypher-options on the client side by stating
in /etc/ssh/ssh_config:

--- 8< snip ---
...
# Ciphers
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
# MACs hmac-md5,hmac-sha1,umac-64 [at] openssh,hmac-ripemd160
MACs hmac-md5,hmac-sha1,hmac-ripemd160
...
--- >8 snap ---

worked like charm. I finally can SSH again to my machines behind the
firewall.

Thanks for this workaround.

KR, Oliver
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

vinschen at redhat

Mar 18, 2011, 12:19 PM

Post #23 of 26 (11156 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

Ping?

On Mar 7 20:29, Corinna Vinschen wrote:
> Hi Darren,
>
> On Feb 17 23:27, Darren Tucker wrote:
> > On 17/02/2011 10:34 PM, Corinna Vinschen wrote:
> > >As an additional datapoint, we had a couple of similar bug reports after
> > >I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
> > >them even comes with a set of debug output of working (5.6p1) and
> > >non-working (5.8p1) connection attempts:
> > [...]
> > >However, I tried with various older versions of SSH running on Cygwin,
> > >Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
> > >reproduce this problem.
> >
> > Thanks for the extra info. I haven't been able to reproduce either.
> > I've tried building 5.5p1 and 4.3p1 against (locally built) OpenSSL
> > 0.9.6b and 0.9.8d. There seems to be some piece of the puzzle
> > missing...
> >
> > I diffed the working and non working clients, and one difference is:
> > debug1: sending SSH2_MSG_KEX_ECDH_INIT
> > debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> >
> > although I'm not sure that's significant since Oren's output had
> > SSH2_MSG_KEX_DH_GEX_GROUP. You could try forcing it with "ssh -vvv
> > -o KexAlgorithms=diffie-hellman-group-exchange-sha1 server"
> >
> > (aside: I now want to add OpenSSL's version output to the server
> > debug output)
>
> is there any progress in that matter?
>
>
> Thanks,
> Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

vinschen at redhat

Mar 25, 2011, 6:20 AM

Post #24 of 26 (11062 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

Ping 2.

If there's a good reason that none of the core developers bothers to
comment further on this serious problem, it would be nice to let us
folks at least know why.

On Mar 18 20:19, Corinna Vinschen wrote:
> Ping?
>
> On Mar 7 20:29, Corinna Vinschen wrote:
> > Hi Darren,
> >
> > On Feb 17 23:27, Darren Tucker wrote:
> > > On 17/02/2011 10:34 PM, Corinna Vinschen wrote:
> > > >As an additional datapoint, we had a couple of similar bug reports after
> > > >I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
> > > >them even comes with a set of debug output of working (5.6p1) and
> > > >non-working (5.8p1) connection attempts:
> > > [...]
> > > >However, I tried with various older versions of SSH running on Cygwin,
> > > >Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
> > > >reproduce this problem.
> > >
> > > Thanks for the extra info. I haven't been able to reproduce either.
> > > I've tried building 5.5p1 and 4.3p1 against (locally built) OpenSSL
> > > 0.9.6b and 0.9.8d. There seems to be some piece of the puzzle
> > > missing...
> > >
> > > I diffed the working and non working clients, and one difference is:
> > > debug1: sending SSH2_MSG_KEX_ECDH_INIT
> > > debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> > >
> > > although I'm not sure that's significant since Oren's output had
> > > SSH2_MSG_KEX_DH_GEX_GROUP. You could try forcing it with "ssh -vvv
> > > -o KexAlgorithms=diffie-hellman-group-exchange-sha1 server"
> > >
> > > (aside: I now want to add OpenSSL's version output to the server
> > > debug output)
> >
> > is there any progress in that matter?

Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

djm at mindrot

Mar 27, 2011, 3:33 AM

Post #25 of 26 (11150 views)
Permalink

Re: ssh 'connection reset by peer' problem since 5.8p1 [In reply to]

I don't use Cygwin myself, so I don't have anything to add. It looks like
something related to ECDH is crashing, but there is insufficient information
for anyone to debug it.

On Fri, 25 Mar 2011, Corinna Vinschen wrote:

> Ping 2.
>
>
> If there's a good reason that none of the core developers bothers to
> comment further on this serious problem, it would be nice to let us
> folks at least know why.
>
>
> On Mar 18 20:19, Corinna Vinschen wrote:
> > Ping?
> >
> > On Mar 7 20:29, Corinna Vinschen wrote:
> > > Hi Darren,
> > >
> > > On Feb 17 23:27, Darren Tucker wrote:
> > > > On 17/02/2011 10:34 PM, Corinna Vinschen wrote:
> > > > >As an additional datapoint, we had a couple of similar bug reports after
> > > > >I upgraded openssh in the Cygwin distro to 5.7p1 and then 5.8p1. One of
> > > > >them even comes with a set of debug output of working (5.6p1) and
> > > > >non-working (5.8p1) connection attempts:
> > > > [...]
> > > > >However, I tried with various older versions of SSH running on Cygwin,
> > > > >Linux and Solaris to connect from 5.8p1 myself, and I'm unable to
> > > > >reproduce this problem.
> > > >
> > > > Thanks for the extra info. I haven't been able to reproduce either.
> > > > I've tried building 5.5p1 and 4.3p1 against (locally built) OpenSSL
> > > > 0.9.6b and 0.9.8d. There seems to be some piece of the puzzle
> > > > missing...
> > > >
> > > > I diffed the working and non working clients, and one difference is:
> > > > debug1: sending SSH2_MSG_KEX_ECDH_INIT
> > > > debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> > > >
> > > > although I'm not sure that's significant since Oren's output had
> > > > SSH2_MSG_KEX_DH_GEX_GROUP. You could try forcing it with "ssh -vvv
> > > > -o KexAlgorithms=diffie-hellman-group-exchange-sha1 server"
> > > >
> > > > (aside: I now want to add OpenSSL's version output to the server
> > > > debug output)
> > >
> > > is there any progress in that matter?
>
>
> Corinna
>
> --
> Corinna Vinschen
> Cygwin Project Co-Leader
> Red Hat
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev [at] mindrot
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads