Mailing List Archive: Bricolage: devel

net::ssh2 problems

Index | Next | Previous | View Threaded

alex at gossamer-threads

Jan 11, 2009, 12:03 PM

Post #1 of 9 (2151 views)
Permalink

net::ssh2 problems

Hi,

We ran into two problems with the new Net::SSH2 based sftp copy.

The first was due to authentication and the remote server not accepting
auth_passwd, but only auth_keyboard authentication. Can I recommend
applying:

Index: SFTP.pm
===================================================================
--- SFTP.pm (revision 1)
+++ SFTP.pm (working copy)
@@ -142,7 +142,7 @@
my $connect = eval {
$ssh2->connect($hn);
$ssh2->method('CRYPT_CS', SFTP_MOVER_CYPHER ) if SFTP_MOVER_CIPHER;
- $ssh2->auth_password($user,$password);
+ $ssh2->auth( username => $user, password => $password);
};
throw_gen error => "Unable to login to remote server '$hn'.",
payload => $@
@@ -275,7 +275,7 @@
my $connect = eval {
$ssh2->connect($hn);
$ssh2->method('CRYPT_CS', SFTP_MOVER_CYPHER ) if SFTP_MOVER_CIPHER;
- $ssh2->auth_password($user,$password);
+ $ssh2->auth( username => $user, password => $password );
};
throw_gen error => "Unable to login to remote server '$hn'.",
payload => $@

which will first try auth_password, and then follow up with
auth_keyboard should the first fail (and will also try ssh keys if you
happen to have that configured).

The second problem we had was that the rename would not work, and we
would be stuck with .tmp files. This turned out to be a problem with
either libssh2 or the ssh server. It turns out in libssh2, it only
passes the LIBSSH2_SFTP_RENAME_OVERWRITE flag if you are connecting to
an ssh server with version > 5. From sftp.c:

if (sftp->version >= 5) {
libssh2_htonu32(sftp->rename_s, flags);
sftp->rename_s += 4;
}

and since in this case the host was using OpenSSH_4.5p1 FreeBSD, it
never any flags and wouldn't overwrite.

I'm not sure if this is a protocol issue or a library issue. In any case
the fix was simply turning on:

FTP_UNLINK_BEFORE_MOVE = Yes

in bricolage.conf. Perhaps a patch for the doc is warranted? Not sure if
there should be a separate conf option for SFTP?

Cheers,

Alex

--
Alex Krohn <alex [at] gossamer-threads>

david at kineticode

Jan 12, 2009, 9:39 AM

Post #2 of 9 (2052 views)
Permalink

Re: net::ssh2 problems [In reply to]

On Jan 11, 2009, at 12:03 PM, Alex Krohn wrote:

> which will first try auth_password, and then follow up with
> auth_keyboard should the first fail (and will also try ssh keys if you
> happen to have that configured).

Committed in r8400. Thanks.

> The second problem we had was that the rename would not work, and we
> would be stuck with .tmp files. This turned out to be a problem with
> either libssh2 or the ssh server. It turns out in libssh2, it only
> passes the LIBSSH2_SFTP_RENAME_OVERWRITE flag if you are connecting to
> an ssh server with version > 5. From sftp.c:
>
> if (sftp->version >= 5) {
> libssh2_htonu32(sftp->rename_s, flags);
> sftp->rename_s += 4;
> }
>
> and since in this case the host was using OpenSSH_4.5p1 FreeBSD, it
> never any flags and wouldn't overwrite.
>
> I'm not sure if this is a protocol issue or a library issue. In any
> case
> the fix was simply turning on:
>
> FTP_UNLINK_BEFORE_MOVE = Yes
>
> in bricolage.conf. Perhaps a patch for the doc is warranted? Not
> sure if
> there should be a separate conf option for SFTP?

Dunno, what would it look like? Doc patches welcome, of course.

Best,

David

alex at gossamer-threads

Jan 12, 2009, 11:54 AM

Post #3 of 9 (2041 views)
Permalink

Re: net::ssh2 problems [In reply to]

Hi,

> > The second problem we had was that the rename would not work, and we
> > would be stuck with .tmp files. This turned out to be a problem with
> > either libssh2 or the ssh server. It turns out in libssh2, it only
> > passes the LIBSSH2_SFTP_RENAME_OVERWRITE flag if you are connecting to
> > an ssh server with version > 5.
>
> Dunno, what would it look like? Doc patches welcome, of course.

How about:

Index: bricolage.conf
===================================================================
--- bricolage.conf (revision 1)
+++ bricolage.conf (working copy)
@@ -187,12 +187,12 @@
# queue on publish, rather than trying to run them immediately, which is the
# default.
#
-# FTP_UNLINK_BEFORE_MOVE must be set to Yes for some FTP servers (e.g.
-# pure-ftpd has been reported to need this) to correctly handle uploads of
-# already existing files: when publishing via FTP, Bricolage first creates a
-# temporary file, than moves this temp file over the existing, older file,
-# overwriting it. Some servers do not support renaming to existing files, so
-# the original one must be deleted first.
+# FTP_UNLINK_BEFORE_MOVE must be set to Yes for some FTP and SFTP servers (e.g.
+# pure-ftpd has been reported to need this, and sshd less then version 5 requires
+# this) to correctly handle uploads of already existing files: when publishing
+# via FTP, Bricolage first creates a temporary file, than moves this temp
+# file over the existing, older file, overwriting it. Some servers do not
+# support renaming to existing files, so the original one must be deleted first.

ENABLE_DIST = Yes
DIST_ATTEMPTS = 3

Not sure if it should be a separate option. i.e. you may need it on to
connect to a sftp server running openssh 4.5, but not want it on on a
second ftp channel? Is the only downside the fraction of time where you
will get a 404, or the possibility of something going wrong and leaving
a 404?

Cheers,

Alex

--
Alex Krohn <alex [at] gossamer-threads>

david at kineticode

Jan 12, 2009, 12:21 PM

Post #4 of 9 (2037 views)
Permalink

Re: net::ssh2 problems [In reply to]

On Jan 12, 2009, at 11:54 AM, Alex Krohn wrote:

> How about:
>
> Index: bricolage.conf
> ===================================================================
> --- bricolage.conf (revision 1)
> +++ bricolage.conf (working copy)
> @@ -187,12 +187,12 @@
> # queue on publish, rather than trying to run them immediately,
> which is the
> # default.
> #
> -# FTP_UNLINK_BEFORE_MOVE must be set to Yes for some FTP servers
> (e.g.
> -# pure-ftpd has been reported to need this) to correctly handle
> uploads of
> -# already existing files: when publishing via FTP, Bricolage first
> creates a
> -# temporary file, than moves this temp file over the existing,
> older file,
> -# overwriting it. Some servers do not support renaming to existing
> files, so
> -# the original one must be deleted first.
> +# FTP_UNLINK_BEFORE_MOVE must be set to Yes for some FTP and SFTP
> servers (e.g.
> +# pure-ftpd has been reported to need this, and sshd less then
> version 5 requires
> +# this) to correctly handle uploads of already existing files: when
> publishing
> +# via FTP, Bricolage first creates a temporary file, than moves
> this temp
> +# file over the existing, older file, overwriting it. Some servers
> do not
> +# support renaming to existing files, so the original one must be
> deleted first.

Sounds okay. Needs to be updated in Bric::Admin, too.

> Not sure if it should be a separate option. i.e. you may need it on to
> connect to a sftp server running openssh 4.5, but not want it on on a
> second ftp channel? Is the only downside the fraction of time where
> you
> will get a 404, or the possibility of something going wrong and
> leaving
> a 404?

Yeah, I think so.

Best,

David

alex at gossamer-threads

Jan 12, 2009, 12:29 PM

Post #5 of 9 (2048 views)
Permalink

Re: net::ssh2 problems [In reply to]

Hi,

> Sounds okay. Needs to be updated in Bric::Admin, too.

Index: conf/bricolage.conf
===================================================================
--- conf/bricolage.conf (revision 1)
+++ conf/bricolage.conf (working copy)
@@ -187,12 +187,12 @@
# queue on publish, rather than trying to run them immediately, which is the
# default.
#
-# FTP_UNLINK_BEFORE_MOVE must be set to Yes for some FTP servers (e.g.
-# pure-ftpd has been reported to need this) to correctly handle uploads of
-# already existing files: when publishing via FTP, Bricolage first creates a
-# temporary file, than moves this temp file over the existing, older file,
-# overwriting it. Some servers do not support renaming to existing files, so
-# the original one must be deleted first.
+# FTP_UNLINK_BEFORE_MOVE must be set to Yes for some FTP and SFTP servers (e.g.
+# pure-ftpd has been reported to need this and sshd less then version 5 requires
+# this) to correctly handle uploads of already existing files: when publishing
+# via FTP, Bricolage first creates a temporary file, than moves this temp
+# file over the existing, older file, overwriting it. Some servers do not
+# support renaming to existing files, so the original one must be deleted first.

ENABLE_DIST = Yes
DIST_ATTEMPTS = 3
Index: lib/Bric/Admin.pod
===================================================================
--- lib/Bric/Admin.pod (revision 1)
+++ lib/Bric/Admin.pod (working copy)
@@ -2087,11 +2087,12 @@
=item *

B<FTP_UNLINK_BEFORE_MOVE:> This directive must be set to Yes for some FTP
-servers (e.g. pure-ftpd has been reported to need this) to correctly handle
-uploads of already existing files: when publishing via FTP, Bricolage first
-creates a temporary file, than moves this temp file over the existing, older
-file, overwriting it. Some servers do not support renaming to existing files,
-so the original one must be deleted first.
+and SFTP servers (e.g. pure-ftpd has been reported to need this and sshd less
+then version 5 requires this) to correctly handle uploads of already existing
+files: when publishing via FTP, Bricolage first creates a temporary file, than
+moves this temp file over the existing, older file, overwriting it. Some
+servers do not support renaming to existing files, so the original one must be
+deleted first.

=item *

Cheers,

Alex

--
Alex Krohn <alex [at] gossamer-threads>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806

david at kineticode

Jan 12, 2009, 12:37 PM

Post #6 of 9 (2038 views)
Permalink

Re: net::ssh2 problems [In reply to]

On Jan 12, 2009, at 12:29 PM, Alex Krohn wrote:

>> Sounds okay. Needs to be updated in Bric::Admin, too.
>
> Index: conf/bricolage.conf
> ===================================================================
> --- conf/bricolage.conf (revision 1)
> +++ conf/bricolage.conf (working copy)
> @@ -187,12 +187,12 @@

Thanks, committed to rev_1_10 in r8401.

Best,

David

alex at gossamer-threads

Jan 12, 2009, 12:39 PM

Post #7 of 9 (2039 views)
Permalink

Re: net::ssh2 problems [In reply to]

Hi,

On Mon Jan 12 12:37:02, David E. Wheeler wrote:

> On Jan 12, 2009, at 12:29 PM, Alex Krohn wrote:
>
> >> Sounds okay. Needs to be updated in Bric::Admin, too.
> >
> > Index: conf/bricolage.conf
> > ===================================================================
> > --- conf/bricolage.conf (revision 1)
> > +++ conf/bricolage.conf (working copy)
> > @@ -187,12 +187,12 @@
>
> Thanks, committed to rev_1_10 in r8401.

One thought, this only applies to Net::SSH2 (that I know of), so it may
only be useful for trunk? As rev_1_10 still uses Net::SFTP.

Cheers,

Alex

--
Alex Krohn <alex [at] gossamer-threads>

david at kineticode

Jan 12, 2009, 12:44 PM

Post #8 of 9 (2057 views)
Permalink

Re: net::ssh2 problems [In reply to]

On Jan 12, 2009, at 12:39 PM, Alex Krohn wrote:

>> Thanks, committed to rev_1_10 in r8401.
>
> One thought, this only applies to Net::SSH2 (that I know of), so it
> may
> only be useful for trunk? As rev_1_10 still uses Net::SFTP.

Hrm. Good point. Fixed in r8402:8403.

Best,

David

rolfm at denison

Mar 1, 2009, 7:08 AM

Post #9 of 9 (1697 views)
Permalink

Re: net::ssh2 problems [In reply to]

On Jan 11, 2009, at 3:03 PM, Alex Krohn wrote:

> Hi,
>
> We ran into two problems with the new Net::SSH2 based sftp copy.
>
> The second problem we had was that the rename would not work, and we
> would be stuck with .tmp files. This turned out to be a problem with
> either libssh2 or the ssh server. It turns out in libssh2, it only
> passes the LIBSSH2_SFTP_RENAME_OVERWRITE flag if you are connecting to
> an ssh server with version > 5. From sftp.c:

FWIW, I've been having the same problem today with version 4.x of
sshd. Furthermore, setting FTP_UNLINK to Yes does not resolve the
issue, either. Our servers on version 3.9 of openssh don't seem to
have the problem.

Both the servers I'm working on today are NetBSD. I'm going to see if
upgrading openssh to 5.x makes any difference.

-Matt

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads