Mailing List Archive: Bricolage: devel

SFTP Refactor

Index | Next | Previous | View Threaded

david at kineticode

Jul 6, 2010, 1:45 PM

Post #1 of 6 (1012 views)
Permalink

SFTP Refactor

Fellow Bricoleurs,

While working on an SFTP bug for a client, I realized that the implementation of Bric::Util::Trans::SFTP did not do proper error handling. So I refactored it, fixing a few other issues along the way:

http://github.com/bricoleurs/bricolage/commit/256e5509a54adb83253e190be26f948ca741efd1

I would be grateful if a few of you on Bricolage 2.0 and doing distribution via SFTP would grab the refactored copy and restart Bricolage to be sure that it works as expected. I was not able to test it for the client, as they are using 1.10. You can grab the raw file here:

http://github.com/bricoleurs/bricolage/raw/256e5509a54adb83253e190be26f948ca741efd1/lib/Bric/Util/Trans/SFTP.pm

Please let me know if you get any unexpected errors or problems. Or if it just works.

Thanks,

David

bret at pectopah

Jul 9, 2010, 8:08 AM

Post #2 of 6 (950 views)
Permalink

Re: SFTP Refactor [In reply to]

Hi David,

I tried it out, but no luck, I'm afraid. That may be because of
something particular to my system (I haven't managed to get SFTP
distribution working with the default one either), but I'm not sure.

I've put a comment on the commit in github, but I'll repeat it here for
Google's sake.

--------------------

There were a couple of typos that I fixed when trying it out, but I
didn't get it to work, so it feels a little pointless to file a patch.
Here they are:

Line 269: method name should be "_connect", not "_connect_to" (github is
hiding the initial underscores here)
Line 275: "trow_gen" should be "throw_gen".

Now, the errors:

Attempting a preview to an SFTP destination, using key authentication
yields this error:

Error authenticating to 'arse.ca' via SSH2

-37 LIBSSH2_ERROR_EAGAIN Would block requesting userauth list

View Error Details
Error Details (hide)

Fault Class
Bric::Util::Fault::Exception::GEN
Description
General Exception
Timestamp
2010-07-09 07:44:22.000000
Package
Bric::Util::Trans::SFTP
Filename
/usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm
Line
298

Stack:

Trace begun at /usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm line 298
Bric::Util::Trans::SFTP::_connect('Bric::Dist::Server=HASH(0x23c86d38)')
called at /usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm line 132
Bric::Util::Trans::SFTP::put_res('Bric::Util::Trans::SFTP',
'ARRAY(0x23c87248)', 'Bric::Dist::ServerType=HASH(0x23c2f580)') called
at /usr/local/bricolage/lib/Bric/Dist/Action/Mover.pm line 160
Bric::Dist::Action::Mover::do_it('Bric::Dist::Action::Mover=HASH(0x23bfd0d0)', 'ARRAY(0x23c87248)', 'Bric::Dist::ServerType=HASH(0x23c2f580)') called at /usr/local/bricolage/lib/Bric/Util/Job/Dist.pm line 227
Bric::Util::Job::Dist::_do_it('Bric::Util::Job::Dist=HASH(0x23c2cd30)')
called at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1881
eval {...} at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1879
Bric::Util::Job::execute_me('Bric::Util::Job::Dist=HASH(0x23c2cd30)')
called at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1800
Bric::Util::Job::save('Bric::Util::Job::Dist=HASH(0x23c2cd30)') called
at /usr/local/bricolage/lib/Bric/Util/Burner.pm line 1036
eval {...} at /usr/local/bricolage/lib/Bric/Util/Burner.pm line 974
Bric::Util::Burner::preview('Bric::Util::Burner=HASH(0x23bb23b0)',
'Bric::Biz::Asset::Business::Story=HASH(0x23b86008)', 'story', 0, undef)
called at /usr/local/bricolage/lib/Bric/App/Callback/Publish.pm line 117
Bric::App::Callback::Publish::preview('Bric::App::Callback::Publish=HASH(0x23a8ee08)') called at /usr/local/bricolage/comp/workflow/profile/preview/dhandler line 32
HTML::Mason::Commands::__ANON__('checkout', 1, 'checkout', 1) called
at /usr/share/perl5/HTML/Mason/Component.pm line 135
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x23bf90c0)', 'checkout', 1, 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 1278
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1268
HTML::Mason::Request::comp(undef, undef, undef, 'checkout', 1,
'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 937
HTML::Mason::Request::call_next('HTML::Mason::Request::ApacheHandler=HASH(0x23a81408)', 'checkout', 1) called at /usr/local/bricolage/comp/autohandler line 7
HTML::Mason::Commands::__ANON__('checkout', 1) called
at /usr/share/perl5/HTML/Mason/Component.pm line 135
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x23a8e918)', 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 1273
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1268
HTML::Mason::Request::comp(undef, undef, undef, 'checkout', 1) called
at /usr/share/perl5/HTML/Mason/Request.pm line 467
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 467
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 419
HTML::Mason::Request::exec('HTML::Mason::Request::ApacheHandler=HASH(0x23a81408)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 165
HTML::Mason::Request::ApacheHandler::exec('HTML::Mason::Request::ApacheHandler=HASH(0x23a81408)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 831
HTML::Mason::ApacheHandler::handle_request('HTML::Mason::ApacheHandler=HASH(0x230c5720)', 'Apache2::RequestRec=SCALAR(0x23bbeb98)') called at /usr/local/bricolage/lib/Bric/App/Handler.pm line 307
eval {...} at /usr/local/bricolage/lib/Bric/App/Handler.pm line 296
Bric::App::Handler::handler('Apache2::RequestRec=SCALAR(0x23bbeb98)')
called at -e line 0
eval {...} at -e line 0

---------------------

Attempting the same preview with password does this:

Error renaming '/usr/www/users/smell/arse/2010/07/06/yes/index.html.tmp'
to '/usr/www/users/smell/arse/2010/07/06/yes/index.html' on 'arse.ca'

4 SSH_FX_FAILURE

View Error Details
Error Details (hide)

Fault Class
Bric::Util::Fault::Exception::GEN
Description
General Exception
Timestamp
2010-07-09 07:45:42.000000
Package
Bric::Util::Trans::SFTP
Filename
/usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm
Line
177

Stack:

Trace begun at /usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm line 177
Bric::Util::Trans::SFTP::put_res('Bric::Util::Trans::SFTP',
'ARRAY(0x23cf7be0)', 'Bric::Dist::ServerType=HASH(0x23c4a908)') called
at /usr/local/bricolage/lib/Bric/Dist/Action/Mover.pm line 160
Bric::Dist::Action::Mover::do_it('Bric::Dist::Action::Mover=HASH(0x23c3f9e8)', 'ARRAY(0x23cf7be0)', 'Bric::Dist::ServerType=HASH(0x23c4a908)') called at /usr/local/bricolage/lib/Bric/Util/Job/Dist.pm line 227
Bric::Util::Job::Dist::_do_it('Bric::Util::Job::Dist=HASH(0x23c0a480)')
called at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1881
eval {...} at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1879
Bric::Util::Job::execute_me('Bric::Util::Job::Dist=HASH(0x23c0a480)')
called at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1800
Bric::Util::Job::save('Bric::Util::Job::Dist=HASH(0x23c0a480)') called
at /usr/local/bricolage/lib/Bric/Util/Burner.pm line 1036
eval {...} at /usr/local/bricolage/lib/Bric/Util/Burner.pm line 974
Bric::Util::Burner::preview('Bric::Util::Burner=HASH(0x23c16258)',
'Bric::Biz::Asset::Business::Story=HASH(0x23c3fb68)', 'story', 0, undef)
called at /usr/local/bricolage/lib/Bric/App/Callback/Publish.pm line 117
Bric::App::Callback::Publish::preview('Bric::App::Callback::Publish=HASH(0x23c3f788)') called at /usr/local/bricolage/comp/workflow/profile/preview/dhandler line 32
HTML::Mason::Commands::__ANON__('checkout', 1, 'checkout', 1) called
at /usr/share/perl5/HTML/Mason/Component.pm line 135
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x23c398c8)', 'checkout', 1, 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 1278
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1268
HTML::Mason::Request::comp(undef, undef, undef, 'checkout', 1,
'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 937
HTML::Mason::Request::call_next('HTML::Mason::Request::ApacheHandler=HASH(0x22adcae8)', 'checkout', 1) called at /usr/local/bricolage/comp/autohandler line 7
HTML::Mason::Commands::__ANON__('checkout', 1) called
at /usr/share/perl5/HTML/Mason/Component.pm line 135
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x23aa4760)', 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 1273
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1268
HTML::Mason::Request::comp(undef, undef, undef, 'checkout', 1) called
at /usr/share/perl5/HTML/Mason/Request.pm line 467
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 467
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 419
HTML::Mason::Request::exec('HTML::Mason::Request::ApacheHandler=HASH(0x22adcae8)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 165
HTML::Mason::Request::ApacheHandler::exec('HTML::Mason::Request::ApacheHandler=HASH(0x22adcae8)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 831
HTML::Mason::ApacheHandler::handle_request('HTML::Mason::ApacheHandler=HASH(0x230c5720)', 'Apache2::RequestRec=SCALAR(0x23be5508)') called at /usr/local/bricolage/lib/Bric/App/Handler.pm line 307
eval {...} at /usr/local/bricolage/lib/Bric/App/Handler.pm line 296
Bric::App::Handler::handler('Apache2::RequestRec=SCALAR(0x23be5508)')
called at -e line 0
eval {...} at -e line 0

This may be particular to my system. I haven't got SFTP working with the
old SFTP.pm either.

On Tue, 2010-07-06 at 13:45 -0700, David E. Wheeler wrote:
> Fellow Bricoleurs,
>
> While working on an SFTP bug for a client, I realized that the implementation of Bric::Util::Trans::SFTP did not do proper error handling. So I refactored it, fixing a few other issues along the way:
>
> http://github.com/bricoleurs/bricolage/commit/256e5509a54adb83253e190be26f948ca741efd1
>
> I would be grateful if a few of you on Bricolage 2.0 and doing distribution via SFTP would grab the refactored copy and restart Bricolage to be sure that it works as expected. I was not able to test it for the client, as they are using 1.10. You can grab the raw file here:
>
> http://github.com/bricoleurs/bricolage/raw/256e5509a54adb83253e190be26f948ca741efd1/lib/Bric/Util/Trans/SFTP.pm
>
> Please let me know if you get any unexpected errors or problems. Or if it just works.
>
> Thanks,
>
> David
>
>

--
Bret Dawson
Producer
Pectopah Productions Inc.
(416) 895-7635
bret [at] pectopah
www.pectopah.com

david at kineticode

Jul 9, 2010, 12:15 PM

Post #3 of 6 (954 views)
Permalink

Re: SFTP Refactor [In reply to]

On Jul 9, 2010, at 8:08 AM, Bret Dawson wrote:

> I tried it out, but no luck, I'm afraid. That may be because of
> something particular to my system (I haven't managed to get SFTP
> distribution working with the default one either), but I'm not sure.

Thanks Bret, much appreciated.

> I've put a comment on the commit in github, but I'll repeat it here for
> Google's sake.

I replied there, but will here, too. :-)

> There were a couple of typos that I fixed when trying it out, but I
> didn't get it to work, so it feels a little pointless to file a patch.
> Here they are:
>
> Line 269: method name should be "_connect", not "_connect_to" (github is
> hiding the initial underscores here)
> Line 275: "trow_gen" should be "throw_gen".

Fixed, thanks.

> Now, the errors:
>
> Attempting a preview to an SFTP destination, using key authentication
> yields this error:
>
> Error authenticating to 'arse.ca' via SSH2
>
> -37 LIBSSH2_ERROR_EAGAIN Would block requesting userauth list

Ah. Would try this patch, please?

--- a/lib/Bric/Util/Trans/SFTP.pm
+++ b/lib/Bric/Util/Trans/SFTP.pm
@@ -288,10 +288,14 @@ sub _connect_to {
) if SFTP_KEY_TYPE;

my $ret = $ssh2->auth(
- username => $server->get_login,
- password => $server->get_password,
- publickey => SFTP_PUBLIC_KEY_FILE,
- privatekey => SFTP_PRIVATE_KEY_FILE
+ ($server->get_login ? (
+ username => $server->get_login,
+ password => $server->get_password,
+ ) : ()),
+ (SFTP_PUBLIC_KEY_FILE ? (
+ publickey => SFTP_PUBLIC_KEY_FILE,
+ privatekey => SFTP_PRIVATE_KEY_FILE,
+ ) : ()),
);
throw_gen(
error => "Error authenticating to '$hn' via SSH2",

> Attempting the same preview with password does this:
>
> Error renaming '/usr/www/users/smell/arse/2010/07/06/yes/index.html.tmp'
> to '/usr/www/users/smell/arse/2010/07/06/yes/index.html' on 'arse.ca'
>
> 4 SSH_FX_FAILURE

Huh. No idea what that error means. Was the file in fact moved before the exception was thrown?

Thanks,

David

bret at pectopah

Jul 9, 2010, 9:44 PM

Post #4 of 6 (945 views)
Permalink

Re: SFTP Refactor [In reply to]

Hi again!

Sadly, the post-patch behaviour is just the same. Here's the error +
stack on the same preview attempt:

-37 LIBSSH2_ERROR_EAGAIN Would block requesting userauth list
View Error Details
Error Details (hide)
Fault Class
Bric::Util::Fault::Exception::GEN
Description
General Exception
Timestamp
2010-07-09 21:41:58.000000
Package
Bric::Util::Trans::SFTP
Filename
/usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm
Line
300

Stack:

Trace begun at /usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm line 300
Bric::Util::Trans::SFTP::_connect_to('Bric::Dist::Server=HASH(0x24be0fb8)') called at /usr/local/bricolage/lib/Bric/Util/Trans/SFTP.pm line 132
Bric::Util::Trans::SFTP::put_res('Bric::Util::Trans::SFTP', 'ARRAY(0x24bde3e8)', 'Bric::Dist::ServerType=HASH(0x24b41be8)') called at /usr/local/bricolage/lib/Bric/Dist/Action/Mover.pm line 160
Bric::Dist::Action::Mover::do_it('Bric::Dist::Action::Mover=HASH(0x24bda778)', 'ARRAY(0x24bde3e8)', 'Bric::Dist::ServerType=HASH(0x24b41be8)') called at /usr/local/bricolage/lib/Bric/Util/Job/Dist.pm line 227
Bric::Util::Job::Dist::_do_it('Bric::Util::Job::Dist=HASH(0x24b5d6d8)') called at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1881
eval {...} at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1879
Bric::Util::Job::execute_me('Bric::Util::Job::Dist=HASH(0x24b5d6d8)') called at /usr/local/bricolage/lib/Bric/Util/Job.pm line 1800
Bric::Util::Job::save('Bric::Util::Job::Dist=HASH(0x24b5d6d8)') called at /usr/local/bricolage/lib/Bric/Util/Burner.pm line 1036
eval {...} at /usr/local/bricolage/lib/Bric/Util/Burner.pm line 974
Bric::Util::Burner::preview('Bric::Util::Burner=HASH(0x246c8a90)', 'Bric::Biz::Asset::Business::Story=HASH(0x24382dc0)', 'story', 0, undef) called at /usr/local/bricolage/lib/Bric/App/Callback/Publish.pm line 117
Bric::App::Callback::Publish::preview('Bric::App::Callback::Publish=HASH(0x24b1b9c0)') called at /usr/local/bricolage/comp/workflow/profile/preview/dhandler line 32
HTML::Mason::Commands::__ANON__('checkout', 1, 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Component.pm line 135
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x24b1e818)', 'checkout', 1, 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 1278
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1268
HTML::Mason::Request::comp(undef, undef, undef, 'checkout', 1, 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 937
HTML::Mason::Request::call_next('HTML::Mason::Request::ApacheHandler=HASH(0x24b1c1b0)', 'checkout', 1) called at /usr/local/bricolage/comp/autohandler line 7
HTML::Mason::Commands::__ANON__('checkout', 1) called at /usr/share/perl5/HTML/Mason/Component.pm line 135
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x24b1ef18)', 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 1273
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 1268
HTML::Mason::Request::comp(undef, undef, undef, 'checkout', 1) called at /usr/share/perl5/HTML/Mason/Request.pm line 467
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 467
eval {...} at /usr/share/perl5/HTML/Mason/Request.pm line 419
HTML::Mason::Request::exec('HTML::Mason::Request::ApacheHandler=HASH(0x24b1c1b0)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 165
HTML::Mason::Request::ApacheHandler::exec('HTML::Mason::Request::ApacheHandler=HASH(0x24b1c1b0)') called at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 831
HTML::Mason::ApacheHandler::handle_request('HTML::Mason::ApacheHandler=HASH(0x246bd1d8)', 'Apache2::RequestRec=SCALAR(0x24ae3f58)') called at /usr/local/bricolage/lib/Bric/App/Handler.pm line 307
eval {...} at /usr/local/bricolage/lib/Bric/App/Handler.pm line 296
Bric::App::Handler::handler('Apache2::RequestRec=SCALAR(0x24ae3f58)') called at -e line 0
eval {...} at -e line 0

Hope this helps,

Bret

On Fri, 2010-07-09 at 12:15 -0700, David E. Wheeler wrote:
> On Jul 9, 2010, at 8:08 AM, Bret Dawson wrote:
>
> > I tried it out, but no luck, I'm afraid. That may be because of
> > something particular to my system (I haven't managed to get SFTP
> > distribution working with the default one either), but I'm not sure.
>
> Thanks Bret, much appreciated.
>
> > I've put a comment on the commit in github, but I'll repeat it here for
> > Google's sake.
>
> I replied there, but will here, too. :-)
>
> > There were a couple of typos that I fixed when trying it out, but I
> > didn't get it to work, so it feels a little pointless to file a patch.
> > Here they are:
> >
> > Line 269: method name should be "_connect", not "_connect_to" (github is
> > hiding the initial underscores here)
> > Line 275: "trow_gen" should be "throw_gen".
>
> Fixed, thanks.
>
> > Now, the errors:
> >
> > Attempting a preview to an SFTP destination, using key authentication
> > yields this error:
> >
> > Error authenticating to 'arse.ca' via SSH2
> >
> > -37 LIBSSH2_ERROR_EAGAIN Would block requesting userauth list
>
> Ah. Would try this patch, please?
>
> --- a/lib/Bric/Util/Trans/SFTP.pm
> +++ b/lib/Bric/Util/Trans/SFTP.pm
> @@ -288,10 +288,14 @@ sub _connect_to {
> ) if SFTP_KEY_TYPE;
>
> my $ret = $ssh2->auth(
> - username => $server->get_login,
> - password => $server->get_password,
> - publickey => SFTP_PUBLIC_KEY_FILE,
> - privatekey => SFTP_PRIVATE_KEY_FILE
> + ($server->get_login ? (
> + username => $server->get_login,
> + password => $server->get_password,
> + ) : ()),
> + (SFTP_PUBLIC_KEY_FILE ? (
> + publickey => SFTP_PUBLIC_KEY_FILE,
> + privatekey => SFTP_PRIVATE_KEY_FILE,
> + ) : ()),
> );
> throw_gen(
> error => "Error authenticating to '$hn' via SSH2",
>
> > Attempting the same preview with password does this:
> >
> > Error renaming '/usr/www/users/smell/arse/2010/07/06/yes/index.html.tmp'
> > to '/usr/www/users/smell/arse/2010/07/06/yes/index.html' on 'arse.ca'
> >
> > 4 SSH_FX_FAILURE
>
> Huh. No idea what that error means. Was the file in fact moved before the exception was thrown?
>
> Thanks,
>
> David
>
>

--
Bret Dawson
Producer
Pectopah Productions Inc.
(416) 895-7635
bret [at] pectopah
www.pectopah.com

mattrolf at me

Aug 1, 2010, 10:34 AM

Post #5 of 6 (883 views)
Permalink

Re: SFTP Refactor [In reply to]

Any progress on this error, David? Any new version you need tested?

-Matt

david at kineticode

Aug 1, 2010, 12:25 PM

Post #6 of 6 (871 views)
Permalink

Re: SFTP Refactor [In reply to]

On Aug 1, 2010, at 10:34 AM, Matthew Rolf wrote:

> Any progress on this error, David? Any new version you need tested?

No, the code is good. It's Net::SSH2 that's broken.

https://rt.cpan.org/Public/Bug/Display.html?id=59332

Best,

David

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads