Mailing List Archive: OpenSSH: Dev

spc restarts

Index | Next | Previous | View Threaded

mark at nosredna

Feb 26, 2012, 7:02 AM

Post #1 of 11 (463 views)
Permalink

spc restarts

I was wondering why scp didn't have a restart-in-the-middle
option when transferring a large file. Isn't that something
that is supported by the underlying ssh protocol? Is there
some other program that can give me this functionality ?

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

gert at greenie

Feb 26, 2012, 8:35 AM

Post #2 of 11 (450 views)
Permalink

Re: spc restarts [In reply to]

Hi,

On Sun, Feb 26, 2012 at 10:02:59AM -0500, Mark Anderson wrote:
> I was wondering why scp didn't have a restart-in-the-middle
> option when transferring a large file. Isn't that something
> that is supported by the underlying ssh protocol? Is there
> some other program that can give me this functionality ?

rsync (-over-ssh).

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert [at] greenie
fax: +49-89-35655025 gert [at] net
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

mouring at eviladmin

Feb 26, 2012, 9:09 AM

Post #3 of 11 (450 views)
Permalink

Re: spc restarts [In reply to]

On Feb 26, 2012, at 9:02 AM, Mark Anderson wrote:

> I was wondering why scp didn't have a restart-in-the-middle
> option when transferring a large file. Isn't that something
> that is supported by the underlying ssh protocol? Is there
> some other program that can give me this functionality ?

Scp came from rcp.. Which is a stupid simple(ish) protocol
more like doing: tar -cvf | ssh tar -xvf. So it really has nothing
to do with the ssh protocol itself as it just needs a streaming
socket to throw data at.

I suspect you're thinking sftp service, and that could be easily
extended to support it. I did a cheap proof of concept that
required adding an sha1 hash request extension to the server
so I could determine where I needed to pick up in the transfer.
It really was an attempt to implement a simplified rsync over sftp.

However, that code is long since dead and burned.

- Ben
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

mark at nosredna

Feb 26, 2012, 9:15 AM

Post #4 of 11 (445 views)
Permalink

Re: spc restarts [In reply to]

Yes, I was thinking of sftp but had forgotten because
I had wanted to ask this question on this list a while
ago but figured I ought to watch the list for a while
first to find out if it was appropriate.

I saw that the server side supported starting in the
middle and I figured that if you had part of the file
locally downloaded it would be easy to start where it
left off. The same capability that wget has for http
transfers.

I could use rsync but only if that server was running
on the other end, which is not as likely as sshd.

On 02/26/12 12:09, Ben Lindstrom wrote:
>
> On Feb 26, 2012, at 9:02 AM, Mark Anderson wrote:
>
>> I was wondering why scp didn't have a restart-in-the-middle
>> option when transferring a large file. Isn't that something
>> that is supported by the underlying ssh protocol? Is there
>> some other program that can give me this functionality ?
>
> Scp came from rcp.. Which is a stupid simple(ish) protocol
> more like doing: tar -cvf | ssh tar -xvf. So it really has nothing
> to do with the ssh protocol itself as it just needs a streaming
> socket to throw data at.
>
> I suspect you're thinking sftp service, and that could be easily
> extended to support it. I did a cheap proof of concept that
> required adding an sha1 hash request extension to the server
> so I could determine where I needed to pick up in the transfer.
> It really was an attempt to implement a simplified rsync over sftp.
>
> However, that code is long since dead and burned.
>
> - Ben
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

nkadel at gmail

Feb 26, 2012, 9:17 AM

Post #5 of 11 (445 views)
Permalink

Re: spc restarts [In reply to]

On Sun, Feb 26, 2012 at 11:35 AM, Gert Doering <gert [at] greenie> wrote:

> Hi,
>
> On Sun, Feb 26, 2012 at 10:02:59AM -0500, Mark Anderson wrote:
> > I was wondering why scp didn't have a restart-in-the-middle
> > option when transferring a large file. Isn't that something
> > that is supported by the underlying ssh protocol? Is there
> > some other program that can give me this functionality ?
>
> rsync (-over-ssh).
>
Specifically, review the documentation for and use the "-P" option for
rsync, which can use rsync automatically. Rsync over ssh actually has a
number of concrete advantages over scp: In particular, using the "-a" or
related options, it can copy symlinks as symlinks, and using the "-H"
option, it can copy hard links as hardlinks. The syntax take a bit of
practice, especially to handle directories.

For example, to replicate /etc from one host to a backup on a localhost:
Note the options used, and the trailing slashes.

rsync -avH root [at] srchos:/etc/ /backupdir/etc/
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

glenn at zewt

Feb 26, 2012, 9:26 AM

Post #6 of 11 (452 views)
Permalink

Re: spc restarts [In reply to]

On Sun, Feb 26, 2012 at 12:15 PM, Mark Anderson <mark [at] nosredna> wrote:

> I could use rsync but only if that server was running
> on the other end, which is not as likely as sshd.
>

You don't actually need a server running, rsync just needs to be available
on the other host. The client side of rsync will run rsync on the remote
end--that's all it's actually doing when it runs ssh (or rsh).

rsync defaults to ssh for me (not rsh), so I don't have to give it any
extra arguments. I'm not sure if that's the default or a Debian patch.
(If it's not the default upstream, it should be.)

--
Glenn Maynard
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

gert at greenie

Feb 26, 2012, 10:02 AM

Post #7 of 11 (454 views)
Permalink

Re: spc restarts [In reply to]

Hi,

On Sun, Feb 26, 2012 at 12:15:06PM -0500, Mark Anderson wrote:
> I could use rsync but only if that server was running
> on the other end, which is not as likely as sshd.

Use rsync-over-ssh, not rsync-to-rsyncd, then :-)

(Of course that requires rsync to be installed on the target machine,
which might not be the case, so I do see your point).

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert [at] greenie
fax: +49-89-35655025 gert [at] net
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

nkadel at gmail

Feb 26, 2012, 12:54 PM

Post #8 of 11 (445 views)
Permalink

Re: spc restarts [In reply to]

On Sun, Feb 26, 2012 at 1:02 PM, Gert Doering <gert [at] greenie> wrote:

> Hi,
>
> On Sun, Feb 26, 2012 at 12:15:06PM -0500, Mark Anderson wrote:
> > I could use rsync but only if that server was running
> > on the other end, which is not as likely as sshd.
>
> Use rsync-over-ssh, not rsync-to-rsyncd, then :-)
>
> (Of course that requires rsync to be installed on the target machine,
> which might not be the case, so I do see your point).

rsync is a pretty common system utility. It actually takes *work* to leave
it out of most common UNIX or Linux deployments, and it is very easy to
test. If the server is one of the commercial SFTP/FTP servers, which are
typically Windows based, it may not be available.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

sfandino at yahoo

Feb 27, 2012, 1:28 AM

Post #9 of 11 (446 views)
Permalink

Re: spc restarts [In reply to]

On 02/26/2012 06:09 PM, Ben Lindstrom wrote:
>
> On Feb 26, 2012, at 9:02 AM, Mark Anderson wrote:
>
>> I was wondering why scp didn't have a restart-in-the-middle
>> option when transferring a large file. Isn't that something
>> that is supported by the underlying ssh protocol? Is there
>> some other program that can give me this functionality ?
>
> Scp came from rcp.. Which is a stupid simple(ish) protocol
> more like doing: tar -cvf | ssh tar -xvf. So it really has nothing
> to do with the ssh protocol itself as it just needs a streaming
> socket to throw data at.
>
> I suspect you're thinking sftp service, and that could be easily
> extended to support it. I did a cheap proof of concept that
> required adding an sha1 hash request extension to the server
> so I could determine where I needed to pick up in the transfer.
> It really was an attempt to implement a simplified rsync over sftp.

You don't really need a hashing extension in order to support transfer
resuming. You can make a good heuristic just comparing remote and local
timestamps and file sizes. For instance, for "put":

if local_ts > remote_ts:
do full transfer
else:
if local_size <= remote_size: done
else: resume transfer from remote_size

That works reliably as far as the remote files are not modified in other
ways.

SFTP clients supporting that that I know are lftp and my perl module
Net::SFTP::Foreign.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

sfandino at yahoo

Feb 27, 2012, 1:28 AM

Post #10 of 11 (449 views)
Permalink

Re: spc restarts [In reply to]

jan.pechanec at oracle

Feb 27, 2012, 10:47 AM

Post #11 of 11 (450 views)
Permalink

Re: spc restarts [In reply to]

On Sun, 26 Feb 2012, Ben Lindstrom wrote:

>
>On Feb 26, 2012, at 9:02 AM, Mark Anderson wrote:
>
>> I was wondering why scp didn't have a restart-in-the-middle
>> option when transferring a large file. Isn't that something
>> that is supported by the underlying ssh protocol? Is there
>> some other program that can give me this functionality ?
>
>Scp came from rcp.. Which is a stupid simple(ish) protocol
>more like doing: tar -cvf | ssh tar -xvf. So it really has nothing
>to do with the ssh protocol itself as it just needs a streaming
>socket to throw data at.

hello, actually, "tar -cvf | ssh tar -xvf" can be much faster
then copying over scp, expecially with many files transferred over a
long distance link since the transfer of each file is confirmed before
the next one is sent. It is described here (section "Efficiency"):

https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works

J.

--
Jan Pechanec
http://blogs.oracle.com/janp
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads