How do I scp extremely large files

[Mike]

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!

 

[Robert Klemme]

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running.  I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy.  My question is how do I read in
chunks of these large files and write them out in chunks?  I cannot
hold the bytes in memory obviously.  Any help or pseudo code is
greatly appreaciated!

I am not sure I understand your question properly. Any Java library
which implements sftp or scp's protocols will have a means to copy
remote files or at least open remote files and obtain an InputStream
or Channel, from which you can read in chunks and store data locally.

If you need to do the transfer on a regularly basis maybe rsync is
better than cooking your own version of it.

Kind regards

robert

 

[Mike]

I am not sure I understand your question properly.  Any Java library
which implements sftp or scp's protocols will have a means to copy
remote files or at least open remote files and obtain an InputStream
or Channel, from which you can read in chunks and store data locally.

If you need to do the transfer on a regularly basis maybe rsync is
better than cooking your own version of it.

Kind regards

robert

Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. I believe you stated
just what I need to do... obtain a channel to read chunks from and
store locally. I'm just a little fuzzy on the details. Haven't done
much with channels or inputStreams for that matter.

 

[Lew]

Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. I believe you stated
just what I need to do... obtain a channel to read chunks from and
store locally. I'm just a little fuzzy on the details. Haven't done
much with channels or inputStreams for that matter.

I gather Java doesn't count as "outside software".

You can read these to get a start with streams, readers, channels and all that:

<http://docs.oracle.com/javase/tutorial/essential/io/index.html>
<http://docs.oracle.com/javase/tutorial/networking/sockets/index.html>

the tutorials being a great place to start
<http://docs.oracle.com/javase/tutorial/reallybigindex.html>
if you just need the basics.

Oracle has a ton more Java documentation besides the tutorials, of course, as
does IBM Developerworks.

 

[Eric Sosman]

[...]
Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. [...]

How much time have you allotted for writing your own JVM,
Java compiler, operating system, and BIOS?

 

[Robert Klemme]

Thanks for the response Robert. rsync is not an option.
Why?

I cannot use
any outside software, open-source or otherwise.

Why aren't you able to use "outside software"? Or did you mean you are
not allowed to?

I believe you stated
just what I need to do... obtain a channel to read chunks from and
store locally. I'm just a little fuzzy on the details. Haven't done
much with channels or inputStreams for that matter.

IO is pretty basic - in any language. I am surprised you haven't been
exposed to it yet. For the details please follow Lew's advice / links.

Kind regards

robert

 

[Arne Vajhøj]

[...]
Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. [...]

How much time have you allotted for writing your own JVM,
Java compiler, operating system, and BIOS?

Arne

 

[Arne Vajhøj]

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!

I would assume your SFTP code has some method that allows
reading of data as a stream which will allow you to write as
a stream too.

Arne

 

[Arne Vajhøj]

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!

Note that scp and sftp is not the same, so your first decision
is whether you want to use scp or sftp.

Arne

 

[Arne Vajhøj]

Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. I believe you stated
just what I need to do... obtain a channel to read chunks from and
store locally. I'm just a little fuzzy on the details. Haven't done
much with channels or inputStreams for that matter.

scp and sftp are non trivial to implement.

If reading chunks is a problem, then you will not be
able to implement them.

You need to get whoever made that rule give you
permission to use a library that support one of
those protocols.

Arne

 

[Eric Sosman]

[...]
Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. [...]

How much time have you allotted for writing your own JVM,
Java compiler, operating system, and BIOS?

Okay, my question was rather tongue-in-cheek. But the issue it's
intended to raise is the matter of "trust" in software: If the O.P. is
willing to trust an externally-supplied JVM, Java compiler, operating
system, and BIOS, the ban on "outside software" is obviously not
absolute. If it's not absolute, there must be some procedure,
somewhere, that declares "THIS outside software is acceptable; THAT
outside software is not." And (here's the exciting conclusion) if the
allowed/forbidden procedure bans rsync and its sixteen years of
development history in favor of a not-yet-written, not-yet-debugged,
not-even-designed half-hearted home-grown imitation, ... Well, the
procedure may have found an anatomically unlikely place to put its head.

 

[Arne Vajhøj]

On 2/6/2012 3:54 PM, Mike wrote:
[...]
Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. [...]

How much time have you allotted for writing your own JVM,
Java compiler, operating system, and BIOS?

Okay, my question was rather tongue-in-cheek. But the issue it's
intended to raise is the matter of "trust" in software: If the O.P. is
willing to trust an externally-supplied JVM, Java compiler, operating
system, and BIOS, the ban on "outside software" is obviously not
absolute. If it's not absolute, there must be some procedure,
somewhere, that declares "THIS outside software is acceptable; THAT
outside software is not." And (here's the exciting conclusion) if the
allowed/forbidden procedure bans rsync and its sixteen years of
development history in favor of a not-yet-written, not-yet-debugged,
not-even-designed half-hearted home-grown imitation, ... Well, the
procedure may have found an anatomically unlikely place to put its head.

I agree.

But weird rules are sometimes seen.

Arne

 

[Mike]

I gather Java doesn't count as "outside software".

You can read these to get a start with streams, readers, channels and allthat:

<http://docs.oracle.com/javase/tutorial/essential/io/index.html>
<http://docs.oracle.com/javase/tutorial/networking/sockets/index.html>

the tutorials being a great place to start
<http://docs.oracle.com/javase/tutorial/reallybigindex.html>
if you just need the basics.

Oracle has a ton more Java documentation besides the tutorials, of course, as
does IBM Developerworks.

Thanks for the advice Lew. You got me on the right track and I
accomplished what I needed. You were correct, Java was obviously
allowed just not 3rd party libraries. Sorry for the confusion I added
to this thread.

 

[Daniel Pitts]

Thanks for the advice Lew. You got me on the right track and I
accomplished what I needed. You were correct, Java was obviously
allowed just not 3rd party libraries. Sorry for the confusion I added
to this thread.

rsync is not a library, but a full, well supported, mature program. It
is used in many production environments for many different purposes.
Hopefully you're not overlooking it because of political nonsense.

 

[Roedy Green]

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running.

I know you have legal requirement not to use "outside" code.

Perhaps a VPN or WebDav would pass.

http://mindprod.com/jgloss/vpn.html
http://mindprod.com/jgloss/webdav.html

 

[Roedy Green]

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!

If your roll your own system, you might want to consider bundling the
small files and zipping as the Replicator does. You would have to
run a script on the server to unpack them. The Replicator only uploads
files that have changed. See
http://mindprod.com/webstart/replicator.html

If the files have only minor changes, you might consider hosting a
version control system on the server. The server can then check out
files that have changed. You get the advantage of atomic updates,
something you will not get with an FTP system. You also can check in
files from many sources directly to the server.