Mailing List Archive: OpenSSH: Dev

ssh localhost yes | true

Index | Next | Previous | View Threaded

mackyle at gmail

Apr 20, 2009, 7:12 PM

Post #1 of 6 (1180 views)
Permalink

ssh localhost yes \| true

Referring to "CLOSED FIXED" Bug 85:

https://bugzilla.mindrot.org/show_bug.cgi?id=85

Assuming that you have your machine setup so that
the following commands run without prompting:

ssh -2 localhost pwd
ssh -1 localhost pwd

Then this command:

ssh -1 localhost yes | true

always produces this output:

Write failed flushing stdout buffer.
write stdout: Broken pipe

Yet this command has varying behavior:

ssh -2 localhost yes | true

Sometimes it returns right away without producing any
output at all (which begs the question why the output is
so different between -1 and -2, but that's not the primary
question of this email).

The no-output case happens on Ubuntu where the output
of ssh -V is:

OpenSSH_5.1p1 Debian-3ubuntu1, OpenSSL 0.9.8g 19 Oct 2007

The same thing happens when running both server AND client
from a MacPorts installation where the output of ssh -V is:

OpenSSH_5.2p1, OpenSSL 0.9.8k 25 Mar 2009

HOWEVER, the "ssh -2 localhost yes | true" command just
hangs indefinitely on Mac OS X 10.5.6 where the output of
ssh -V is:

OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006

It turns out that it's the version of sshd that seems to matter
as the older ssh OpenSSL 0.9.7l client talking to a newer sshd
daemon doesn't hang (and the logs at the end of this message
seem to bear that out).

Now back to the "CLOSED FIXED" Bug 85. The last comment is:

---- Comment #9 From Damien Miller 2008-07-22 12:06:22 ----
Mass update RESOLVED->CLOSED after release of openssh-5.1

The problem ssh version on Mac OS X is marked "OpenSSH_5.1p1"
and yet it clearly has the bug. The comment on Bug 85
leads me to believe it should have the fix but the date of
the comment suggests that it might not (which turns out to
be the case). And the bug doesn't mention any dependency on
and configure options or on a particular version of OpenSSL
for the fix.

Can anyone shed some light on how to tell based on the versions
reported by ssh -V / sshd -V whether or not the hanging problem
will be present?

And is there anyway to see the OpenSSL version of the remote
sshd via ssh from another host? (The -v option to ssh only
reports "remote software version OpenSSH_5.1" and does
not include the OpenSSL version.)

Thanks,

Kyle

P.S. No, I don't actually need to run "ssh localhost yes | true"
however I often use Subversion with the svn+ssh protocol and the
recent Subversion fix for issue 2580 creates a situation very
much like "ssh localhost yes | true" where the ssh process never
exits.

http://subversion.tigris.org/issues/show_bug.cgi?id=2580

P.P.S. The last part of the debug log from ssh client when it
hangs:

...
debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: send eow
debug2: channel 0: output open -> closed
connection hangs at this point

and sshd log looks like this:

...
debug1: server_input_channel_req: channel 0 request eow [at] openssh
reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
connection hangs at this point

whereas when it doesn't hang, ssh client log looks like this:

...
debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: send eow
debug2: channel 0: output open -> closed
debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0
debug1: client_input_channel_req: channel 0 rtype eow [at] openssh
reply 0
debug2: channel 0: rcvd eow
...

and sshd log looks like this:

...
debug1: server_input_channel_req: channel 0 request eow [at] openssh
reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 24887
debug1: session_exit_message: session 0 channel 0 pid 24887
debug2: channel 0: request exit-signal confirm 0
debug1: session_exit_message: release channel 0
debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: send eow
...

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

djm at mindrot

Apr 20, 2009, 11:10 PM

Post #2 of 6 (1124 views)
Permalink

Re: ssh localhost yes \| true [In reply to]

On Mon, 20 Apr 2009, Kyle McKay wrote:

> Referring to "CLOSED FIXED" Bug 85:
>
> https://bugzilla.mindrot.org/show_bug.cgi?id=85

...

> HOWEVER, the "ssh -2 localhost yes | true" command just
> hangs indefinitely on Mac OS X 10.5.6 where the output of
> ssh -V is:
>
> OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006
>
> It turns out that it's the version of sshd that seems to matter
> as the older ssh OpenSSL 0.9.7l client talking to a newer sshd
> daemon doesn't hang (and the logs at the end of this message
> seem to bear that out).

This doesn't look like bug#85 based on what you sent below. It looks
more like a whatever that you were running failing to notice its stdout
closing.

Either way, the problem is far more likely to be due to modifications
that Apple have made to the ssh/sshd they ship in their base OS rather
than anything to do with OpenSSL.

> Can anyone shed some light on how to tell based on the versions
> reported by ssh -V / sshd -V whether or not the hanging problem
> will be present?

OpenSSH 5.1 has the fix. Since we had to extend the SSH protocol to
fix this bug, both ends (ssh and sshd) need to be this version or greater
for it to work.

> And is there anyway to see the OpenSSL version of the remote
> sshd via ssh from another host? (The -v option to ssh only
> reports "remote software version OpenSSH_5.1" and does
> not include the OpenSSL version.)

No, the OpenSSL version is really not relevant since it handles crypto
and not much else.

> P.S. No, I don't actually need to run "ssh localhost yes | true"
> however I often use Subversion with the svn+ssh protocol and the
> recent Subversion fix for issue 2580 creates a situation very
> much like "ssh localhost yes | true" where the ssh process never
> exits.
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=2580
>
> P.P.S. The last part of the debug log from ssh client when it
> hangs:
>
> ...
> debug2: channel 0: write failed
> debug2: channel 0: close_write
> debug2: channel 0: send eow
> debug2: channel 0: output open -> closed
> connection hangs at this point

The client is doing the right thing.

> and sshd log looks like this:
>
> ...
> debug1: server_input_channel_req: channel 0 request eow [at] openssh
> reply 0
> debug2: channel 0: rcvd eow
> debug2: channel 0: close_read
> debug2: channel 0: input open -> closed
> connection hangs at this point

The server is doing the right thing, but the underlying process is not
terminating when its output it closed.

> whereas when it doesn't hang, ssh client log looks like this:
>
> ...
> debug2: channel 0: write failed
> debug2: channel 0: close_write
> debug2: channel 0: send eow
> debug2: channel 0: output open -> closed
> debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0
> debug1: client_input_channel_req: channel 0 rtype eow [at] openssh
> reply 0
> debug2: channel 0: rcvd eow
> ...

Yes, the client signals that its input fd has closed, the server closed
the output fd of its process and the process exited.

> and sshd log looks like this:
>
> ...
> debug1: server_input_channel_req: channel 0 request eow [at] openssh
> reply 0
> debug2: channel 0: rcvd eow

^^^ this is the client signalling that its input has closed

> debug2: channel 0: close_read
> debug2: channel 0: input open -> closed

^^^ server closes output of child process

> debug1: Received SIGCHLD.

^^^ child process exits

> debug1: session_by_pid: pid 24887
> debug1: session_exit_message: session 0 channel 0 pid 24887
> debug2: channel 0: request exit-signal confirm 0
> debug1: session_exit_message: release channel 0

^^^ server sends notification that it has exited.

-d

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

mackyle at gmail

Apr 20, 2009, 11:39 PM

Post #3 of 6 (1134 views)
Permalink

Re: ssh localhost yes \| true [In reply to]

Thanks for your response.

On Apr 20, 2009, at 23:10, Damien Miller wrote:
> On Mon, 20 Apr 2009, Kyle McKay wrote:
>
>> Referring to "CLOSED FIXED" Bug 85:
>>
>> https://bugzilla.mindrot.org/show_bug.cgi?id=85
>
> ...
>
>> HOWEVER, the "ssh -2 localhost yes | true" command just
>> hangs indefinitely on Mac OS X 10.5.6 where the output of
>> ssh -V is:
>>
>> OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006
>>
>> It turns out that it's the version of sshd that seems to matter
>> as the older ssh OpenSSL 0.9.7l client talking to a newer sshd
>> daemon doesn't hang (and the logs at the end of this message
>> seem to bear that out).
>
> This doesn't look like bug#85 based on what you sent below. It looks
> more like a whatever that you were running failing to notice its
> stdout
> closing.

Actually after looking at this some more, I think Apple's sshd is just
missing the Bug 85 patch and I think the debug messages back this up.

>> ...
>> debug2: channel 0: write failed
>> debug2: channel 0: close_write
>> debug2: channel 0: send eow
>> debug2: channel 0: output open -> closed
>> connection hangs at this point
>
> The client is doing the right thing.
>
>> and sshd log looks like this:
>>
>> ...
>> debug1: server_input_channel_req: channel 0 request eow [at] openssh
>> reply 0
>> debug2: channel 0: rcvd eow
>> debug2: channel 0: close_read
>> debug2: channel 0: input open -> closed
>> connection hangs at this point
>
> The server is doing the right thing, but the underlying process is not
> terminating when its output it closed.

At this point the child should have exited. However, if it's
depending on a SIGPIPE to do so and sshd is using socketpair instead
of pipe to communicate with it, Bug 85 comes into play and the child
doesn't get a SIGPIPE as a result of input transitioning from open ->
closed and so doesn't exit.

>> whereas when it doesn't hang, ssh client log looks like this:
>>
>> ...
>> debug2: channel 0: write failed
>> debug2: channel 0: close_write
>> debug2: channel 0: send eow
>> debug2: channel 0: output open -> closed
>> debug1: client_input_channel_req: channel 0 rtype exit-signal
>> reply 0
>> debug1: client_input_channel_req: channel 0 rtype eow [at] openssh
>> reply 0
>> debug2: channel 0: rcvd eow
>> ...
>
> Yes, the client signals that its input fd has closed, the server
> closed
> the output fd of its process and the process exited.
>
>> and sshd log looks like this:
>>
>> ...
>> debug1: server_input_channel_req: channel 0 request eow [at] openssh
>> reply 0
>> debug2: channel 0: rcvd eow
>
> ^^^ this is the client signalling that its input has closed
>
>> debug2: channel 0: close_read
>> debug2: channel 0: input open -> closed
>
> ^^^ server closes output of child process
>
>> debug1: Received SIGCHLD.
>
> ^^^ child process exits

And in this case sshd included the Bug 85 fix and so was using pipes
to communicate with the child and the child therefore got a SIGPIPE on
the input open -> closed transition and exited.

Kyle
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

djm at mindrot

Apr 20, 2009, 11:53 PM

Post #4 of 6 (1121 views)
Permalink

Re: ssh localhost yes \| true [In reply to]

On Mon, 20 Apr 2009, Kyle McKay wrote:

> Thanks for your response.
>
> On Apr 20, 2009, at 23:10, Damien Miller wrote:
> >
> > This doesn't look like bug#85 based on what you sent below. It looks
> > more like a whatever that you were running failing to notice its stdout
> > closing.
>
> Actually after looking at this some more, I think Apple's sshd is just
> missing the Bug 85 patch and I think the debug messages back this up.

No, Apple's sshd clearly does have the bug #85 patch:

> debug2: channel 0: rcvd eow

Would not be printed otherwise. Perhaps they have forcibly disabled
USE_PIPES in session.c? It is required for the patch to correctly
function.

> And in this case sshd included the Bug 85 fix and so was using pipes to
> communicate with the child and the child therefore got a SIGPIPE on the input
> open -> closed transition and exited.

No, in both cases the signalling between sshd and the child process is
identical from the logs. There may be differences depending on whether
Apple has modified their sshd to avoid using pipes (thwarting the
half-close fix in the process), but you will need to post a full debug
log from the server to tell.

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

mackyle at gmail

Apr 21, 2009, 1:27 AM

Post #5 of 6 (1122 views)
Permalink

Re: ssh localhost yes \| true [In reply to]

Yup, you're right. Apple has disabled the USE_PIPES define in their
source code.

Here's the original file:

http://www.opensource.apple.com/darwinsource/10.5.6/OpenSSH-95.1.5/openssh/session.c.orig

And the file they're using:

http://www.opensource.apple.com/darwinsource/10.5.6/OpenSSH-95.1.5/openssh/session.c

Here's the diff:

--- session.c.orig
+++ session.c
@@ -424,7 +424,9 @@
}
}

+#ifndef __APPLE__
#define USE_PIPES
+#endif
/*
* This is called to fork and execute a command when we have no
tty. This
* will call do_child from the child, and server_loop from the
parent after
@@ -2082,8 +2084,10 @@
n_bytes = packet_remaining();
tty_parse_modes(s->ttyfd, &n_bytes);

+#ifndef __APPLE_PRIVPTY__
if (!use_privsep)
pty_setowner(s->pw, s->tty);
+#endif

/* Set window size from the packet. */
pty_change_window_size(s->ptyfd, s->row, s->col, s->xpixel, s-
>ypixel);
@@ -2321,9 +2325,11 @@
if (s->pid != 0)
record_logout(s->pid, s->tty, s->pw->pw_name);

+#ifndef __APPLE_PRIVPTY__
/* Release the pseudo-tty. */
if (getuid() == 0)
pty_release(s->tty);
+#endif

/*
* Close the server side of the socket pairs. We must do this after

Thanks for your help.

Kyle

On Apr 20, 2009, at 23:53, Damien Miller wrote:
> On Mon, 20 Apr 2009, Kyle McKay wrote:
>
>> Thanks for your response.
>>
>> On Apr 20, 2009, at 23:10, Damien Miller wrote:
>>>
>>> This doesn't look like bug#85 based on what you sent below. It looks
>>> more like a whatever that you were running failing to notice its
>>> stdout
>>> closing.
>>
>> Actually after looking at this some more, I think Apple's sshd is
>> just
>> missing the Bug 85 patch and I think the debug messages back this up.
>
> No, Apple's sshd clearly does have the bug #85 patch:
>
>> debug2: channel 0: rcvd eow
>
> Would not be printed otherwise. Perhaps they have forcibly disabled
> USE_PIPES in session.c? It is required for the patch to correctly
> function.
>
>> And in this case sshd included the Bug 85 fix and so was using
>> pipes to
>> communicate with the child and the child therefore got a SIGPIPE on
>> the input
>> open -> closed transition and exited.
>
> No, in both cases the signalling between sshd and the child process is
> identical from the logs. There may be differences depending on whether
> Apple has modified their sshd to avoid using pipes (thwarting the
> half-close fix in the process), but you will need to post a full debug
> log from the server to tell.
>
> -d

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

mackyle at gmail

Apr 23, 2009, 6:22 AM

Post #6 of 6 (1100 views)
Permalink

Re: ssh localhost yes \| true (follow up) [In reply to]

I filed bug 6810722 with Apple and sent them a patch that makes the
half-close fix work again without causing problems with shells (like
Apple's bash that attempt to detect when they're being run by a remote
shell daemon such as sshd by apparently checking standard input to see
if it's a socket).

On Apr 21, 2009, at 01:27, Kyle McKay wrote:
> On Apr 20, 2009, at 23:53, Damien Miller wrote:
>> There may be differences depending on whether
>> Apple has modified their sshd to avoid using pipes (thwarting the
>> half-close fix in the process), but you will need to post a full
>> debug
>> log from the server to tell.
>>
>> -d
>
> Yup, you're right. Apple has disabled the USE_PIPES define in their
> source code.

FYI, patch is below.

Again, thanks for all your help with this.

Kyle

Patch is against the version of session.c that has all of Apple's
patches already applied and can be found at:

http://www.opensource.apple.com/darwinsource/10.5.6/OpenSSH-95.1.5/openssh/session.c

--- session.c 2008-08-13 17:40:56.000000000 -0700
+++ session.c 2009-04-23 04:39:14.000000000 -0700
@@ -424,9 +424,7 @@
}
}

-#ifndef __APPLE__
#define USE_PIPES
-#endif
/*
* This is called to fork and execute a command when we have no tty.
This
* will call do_child from the child, and server_loop from the parent
after
@@ -440,11 +438,12 @@
#ifdef USE_PIPES
int pin[2], pout[2], perr[2];

- /* Allocate pipes for communicating with the program. */
- if (pipe(pin) < 0) {
- error("%s: pipe in: %.100s", __func__, strerror(errno));
+ /* Allocate socketpair for communicating with the program input. */
+ if (socketpair(AF_UNIX, SOCK_STREAM, 0, pin) < 0) {
+ error("%s: socketpair #1: %.100s", __func__, strerror(errno));
return -1;
}
+ /* Allocate pipes for communicating with the program output. */
if (pipe(pout) < 0) {
error("%s: pipe out: %.100s", __func__, strerror(errno));
close(pin[0]);

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads