How do I scp extremely large files

Discussion in 'Java' started by Mike, Feb 5, 2012.

  1. Mike

    Mike Guest

    I need to copy extremely large files (30-100G) from a remove server to
    a machine where my code will be running. I have code in place now
    that uses an sftp connection to scan the file directory watching for
    files to show up for me to copy. My question is how do I read in
    chunks of these large files and write them out in chunks? I cannot
    hold the bytes in memory obviously. Any help or pseudo code is
    greatly appreaciated!
     
    Mike, Feb 5, 2012
    #1
    1. Advertising

  2. On 5 Feb., 03:03, Mike <> wrote:
    > I need to copy extremely large files (30-100G) from a remove server to
    > a machine where my code will be running.  I have code in place now
    > that uses an sftp connection to scan the file directory watching for
    > files to show up for me to copy.  My question is how do I read in
    > chunks of these large files and write them out in chunks?  I cannot
    > hold the bytes in memory obviously.  Any help or pseudo code is
    > greatly appreaciated!


    I am not sure I understand your question properly. Any Java library
    which implements sftp or scp's protocols will have a means to copy
    remote files or at least open remote files and obtain an InputStream
    or Channel, from which you can read in chunks and store data locally.

    If you need to do the transfer on a regularly basis maybe rsync is
    better than cooking your own version of it.

    Kind regards

    robert
     
    Robert Klemme, Feb 6, 2012
    #2
    1. Advertising

  3. Mike

    Mike Guest

    On Feb 6, 2:54 am, Robert Klemme <> wrote:
    > I am not sure I understand your question properly.  Any Java library
    > which implements sftp or scp's protocols will have a means to copy
    > remote files or at least open remote files and obtain an InputStream
    > or Channel, from which you can read in chunks and store data locally.
    >
    > If you need to do the transfer on a regularly basis maybe rsync is
    > better than cooking your own version of it.
    >
    > Kind regards
    >
    > robert


    Thanks for the response Robert. rsync is not an option. I cannot use
    any outside software, open-source or otherwise. I believe you stated
    just what I need to do... obtain a channel to read chunks from and
    store locally. I'm just a little fuzzy on the details. Haven't done
    much with channels or inputStreams for that matter.
     
    Mike, Feb 6, 2012
    #3
  4. Mike

    Lew Guest

    Mike wrote:
    > Thanks for the response Robert. rsync is not an option. I cannot use
    > any outside software, open-source or otherwise. I believe you stated
    > just what I need to do... obtain a channel to read chunks from and
    > store locally. I'm just a little fuzzy on the details. Haven't done
    > much with channels or inputStreams for that matter.


    I gather Java doesn't count as "outside software".

    You can read these to get a start with streams, readers, channels and all that:

    <http://docs.oracle.com/javase/tutorial/essential/io/index.html>
    <http://docs.oracle.com/javase/tutorial/networking/sockets/index.html>

    the tutorials being a great place to start
    <http://docs.oracle.com/javase/tutorial/reallybigindex.html>
    if you just need the basics.

    Oracle has a ton more Java documentation besides the tutorials, of course, as
    does IBM Developerworks.

    --
    Lew
     
    Lew, Feb 7, 2012
    #4
  5. Mike

    Eric Sosman Guest

    On 2/6/2012 3:54 PM, Mike wrote:
    >[...]
    > Thanks for the response Robert. rsync is not an option. I cannot use
    > any outside software, open-source or otherwise. [...]


    How much time have you allotted for writing your own JVM,
    Java compiler, operating system, and BIOS?

    --
    Eric Sosman
    d
     
    Eric Sosman, Feb 7, 2012
    #5
  6. On 02/06/2012 09:54 PM, Mike wrote:
    > On Feb 6, 2:54 am, Robert Klemme<> wrote:
    >> I am not sure I understand your question properly. Any Java library
    >> which implements sftp or scp's protocols will have a means to copy
    >> remote files or at least open remote files and obtain an InputStream
    >> or Channel, from which you can read in chunks and store data locally.
    >>
    >> If you need to do the transfer on a regularly basis maybe rsync is
    >> better than cooking your own version of it.


    > Thanks for the response Robert. rsync is not an option.


    Why?

    > I cannot use
    > any outside software, open-source or otherwise.


    Why aren't you able to use "outside software"? Or did you mean you are
    not allowed to?

    > I believe you stated
    > just what I need to do... obtain a channel to read chunks from and
    > store locally. I'm just a little fuzzy on the details. Haven't done
    > much with channels or inputStreams for that matter.


    IO is pretty basic - in any language. I am surprised you haven't been
    exposed to it yet. For the details please follow Lew's advice / links.

    Kind regards

    robert
     
    Robert Klemme, Feb 7, 2012
    #6
  7. Mike

    Arne Vajhøj Guest

    On 2/7/2012 8:12 AM, Eric Sosman wrote:
    > On 2/6/2012 3:54 PM, Mike wrote:
    >> [...]
    >> Thanks for the response Robert. rsync is not an option. I cannot use
    >> any outside software, open-source or otherwise. [...]

    >
    > How much time have you allotted for writing your own JVM,
    > Java compiler, operating system, and BIOS?


    :)

    Arne
     
    Arne Vajhøj, Feb 7, 2012
    #7
  8. Mike

    Arne Vajhøj Guest

    On 2/4/2012 9:03 PM, Mike wrote:
    > I need to copy extremely large files (30-100G) from a remove server to
    > a machine where my code will be running. I have code in place now
    > that uses an sftp connection to scan the file directory watching for
    > files to show up for me to copy. My question is how do I read in
    > chunks of these large files and write them out in chunks? I cannot
    > hold the bytes in memory obviously. Any help or pseudo code is
    > greatly appreaciated!


    I would assume your SFTP code has some method that allows
    reading of data as a stream which will allow you to write as
    a stream too.

    Arne
     
    Arne Vajhøj, Feb 7, 2012
    #8
  9. Mike

    Arne Vajhøj Guest

    On 2/4/2012 9:03 PM, Mike wrote:
    > I need to copy extremely large files (30-100G) from a remove server to
    > a machine where my code will be running. I have code in place now
    > that uses an sftp connection to scan the file directory watching for
    > files to show up for me to copy. My question is how do I read in
    > chunks of these large files and write them out in chunks? I cannot
    > hold the bytes in memory obviously. Any help or pseudo code is
    > greatly appreaciated!


    Note that scp and sftp is not the same, so your first decision
    is whether you want to use scp or sftp.

    Arne
     
    Arne Vajhøj, Feb 7, 2012
    #9
  10. Mike

    Arne Vajhøj Guest

    On 2/6/2012 3:54 PM, Mike wrote:
    > On Feb 6, 2:54 am, Robert Klemme<> wrote:
    >> I am not sure I understand your question properly. Any Java library
    >> which implements sftp or scp's protocols will have a means to copy
    >> remote files or at least open remote files and obtain an InputStream
    >> or Channel, from which you can read in chunks and store data locally.
    >>
    >> If you need to do the transfer on a regularly basis maybe rsync is
    >> better than cooking your own version of it.

    >
    > Thanks for the response Robert. rsync is not an option. I cannot use
    > any outside software, open-source or otherwise. I believe you stated
    > just what I need to do... obtain a channel to read chunks from and
    > store locally. I'm just a little fuzzy on the details. Haven't done
    > much with channels or inputStreams for that matter.


    scp and sftp are non trivial to implement.

    If reading chunks is a problem, then you will not be
    able to implement them.

    You need to get whoever made that rule give you
    permission to use a library that support one of
    those protocols.

    Arne
     
    Arne Vajhøj, Feb 7, 2012
    #10
  11. Mike

    Eric Sosman Guest

    On 2/7/2012 5:02 PM, Arne Vajhøj wrote:
    > On 2/7/2012 8:12 AM, Eric Sosman wrote:
    >> On 2/6/2012 3:54 PM, Mike wrote:
    >>> [...]
    >>> Thanks for the response Robert. rsync is not an option. I cannot use
    >>> any outside software, open-source or otherwise. [...]

    >>
    >> How much time have you allotted for writing your own JVM,
    >> Java compiler, operating system, and BIOS?

    >
    > :)


    Okay, my question was rather tongue-in-cheek. But the issue it's
    intended to raise is the matter of "trust" in software: If the O.P. is
    willing to trust an externally-supplied JVM, Java compiler, operating
    system, and BIOS, the ban on "outside software" is obviously not
    absolute. If it's not absolute, there must be some procedure,
    somewhere, that declares "THIS outside software is acceptable; THAT
    outside software is not." And (here's the exciting conclusion) if the
    allowed/forbidden procedure bans rsync and its sixteen years of
    development history in favor of a not-yet-written, not-yet-debugged,
    not-even-designed half-hearted home-grown imitation, ... Well, the
    procedure may have found an anatomically unlikely place to put its head.

    --
    Eric Sosman
    d
     
    Eric Sosman, Feb 8, 2012
    #11
  12. Mike

    Arne Vajhøj Guest

    On 2/7/2012 8:47 PM, Eric Sosman wrote:
    > On 2/7/2012 5:02 PM, Arne Vajhøj wrote:
    >> On 2/7/2012 8:12 AM, Eric Sosman wrote:
    >>> On 2/6/2012 3:54 PM, Mike wrote:
    >>>> [...]
    >>>> Thanks for the response Robert. rsync is not an option. I cannot use
    >>>> any outside software, open-source or otherwise. [...]
    >>>
    >>> How much time have you allotted for writing your own JVM,
    >>> Java compiler, operating system, and BIOS?

    >>
    >> :)

    >
    > Okay, my question was rather tongue-in-cheek. But the issue it's
    > intended to raise is the matter of "trust" in software: If the O.P. is
    > willing to trust an externally-supplied JVM, Java compiler, operating
    > system, and BIOS, the ban on "outside software" is obviously not
    > absolute. If it's not absolute, there must be some procedure,
    > somewhere, that declares "THIS outside software is acceptable; THAT
    > outside software is not." And (here's the exciting conclusion) if the
    > allowed/forbidden procedure bans rsync and its sixteen years of
    > development history in favor of a not-yet-written, not-yet-debugged,
    > not-even-designed half-hearted home-grown imitation, ... Well, the
    > procedure may have found an anatomically unlikely place to put its head.


    I agree.

    But weird rules are sometimes seen.

    Arne
     
    Arne Vajhøj, Feb 8, 2012
    #12
  13. Mike

    Mike Guest

    On Feb 6, 8:21 pm, Lew <> wrote:
    > Mike wrote:
    > > Thanks for the response Robert. rsync is not an option.  I cannot use
    > > any outside software, open-source or otherwise.  I believe you stated
    > > just what I need to do... obtain a channel to read chunks from and
    > > store locally.  I'm just a little fuzzy on the details.  Haven't done
    > > much with channels or inputStreams for that matter.

    >
    > I gather Java doesn't count as "outside software".
    >
    > You can read these to get a start with streams, readers, channels and allthat:
    >
    > <http://docs.oracle.com/javase/tutorial/essential/io/index.html>
    > <http://docs.oracle.com/javase/tutorial/networking/sockets/index.html>
    >
    > the tutorials being a great place to start
    > <http://docs.oracle.com/javase/tutorial/reallybigindex.html>
    > if you just need the basics.
    >
    > Oracle has a ton more Java documentation besides the tutorials, of course, as
    > does IBM Developerworks.
    >
    > --
    > Lew


    Thanks for the advice Lew. You got me on the right track and I
    accomplished what I needed. You were correct, Java was obviously
    allowed just not 3rd party libraries. Sorry for the confusion I added
    to this thread.
     
    Mike, Feb 9, 2012
    #13
  14. Mike

    Daniel Pitts Guest

    On 2/8/12 4:29 PM, Mike wrote:
    > On Feb 6, 8:21 pm, Lew<> wrote:
    >> Mike wrote:
    >>> Thanks for the response Robert. rsync is not an option. I cannot use
    >>> any outside software, open-source or otherwise. I believe you stated
    >>> just what I need to do... obtain a channel to read chunks from and
    >>> store locally. I'm just a little fuzzy on the details. Haven't done
    >>> much with channels or inputStreams for that matter.

    >>
    >> I gather Java doesn't count as "outside software".
    >>
    >> You can read these to get a start with streams, readers, channels and all that:
    >>
    >> <http://docs.oracle.com/javase/tutorial/essential/io/index.html>
    >> <http://docs.oracle.com/javase/tutorial/networking/sockets/index.html>
    >>
    >> the tutorials being a great place to start
    >> <http://docs.oracle.com/javase/tutorial/reallybigindex.html>
    >> if you just need the basics.
    >>
    >> Oracle has a ton more Java documentation besides the tutorials, of course, as
    >> does IBM Developerworks.
    >>
    >> --
    >> Lew

    >
    > Thanks for the advice Lew. You got me on the right track and I
    > accomplished what I needed. You were correct, Java was obviously
    > allowed just not 3rd party libraries. Sorry for the confusion I added
    > to this thread.

    rsync is not a library, but a full, well supported, mature program. It
    is used in many production environments for many different purposes.
    Hopefully you're not overlooking it because of political nonsense.
     
    Daniel Pitts, Feb 9, 2012
    #14
  15. Mike

    Roedy Green Guest

    On Sat, 4 Feb 2012 18:03:22 -0800 (PST), Mike <>
    wrote, quoted or indirectly quoted someone who said :

    >I need to copy extremely large files (30-100G) from a remove server to
    >a machine where my code will be running.


    I know you have legal requirement not to use "outside" code.

    Perhaps a VPN or WebDav would pass.

    http://mindprod.com/jgloss/vpn.html
    http://mindprod.com/jgloss/webdav.html
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    One of the most useful comments you can put in a program is
    "If you change this, remember to change ?XXX? too".
     
    Roedy Green, Feb 9, 2012
    #15
  16. Mike

    Roedy Green Guest

    On Sat, 4 Feb 2012 18:03:22 -0800 (PST), Mike <>
    wrote, quoted or indirectly quoted someone who said :

    >I need to copy extremely large files (30-100G) from a remove server to
    >a machine where my code will be running. I have code in place now
    >that uses an sftp connection to scan the file directory watching for
    >files to show up for me to copy. My question is how do I read in
    >chunks of these large files and write them out in chunks? I cannot
    >hold the bytes in memory obviously. Any help or pseudo code is
    >greatly appreaciated!


    If your roll your own system, you might want to consider bundling the
    small files and zipping as the Replicator does. You would have to
    run a script on the server to unpack them. The Replicator only uploads
    files that have changed. See
    http://mindprod.com/webstart/replicator.html

    If the files have only minor changes, you might consider hosting a
    version control system on the server. The server can then check out
    files that have changed. You get the advantage of atomic updates,
    something you will not get with an FTP system. You also can check in
    files from many sources directly to the server.

    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    One of the most useful comments you can put in a program is
    "If you change this, remember to change ?XXX? too".
     
    Roedy Green, Feb 9, 2012
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mikester
    Replies:
    7
    Views:
    614
    mikester
    Dec 25, 2003
  2. krabhi
    Replies:
    1
    Views:
    9,462
    Marco Meschieri
    Aug 9, 2006
  3. Robert Kochem
    Replies:
    1
    Views:
    378
    Arne Vajhøj
    Apr 29, 2010
  4. Une Bévue
    Replies:
    0
    Views:
    434
    Une Bévue
    Jul 22, 2008
  5. Husam Senussi

    downloading large file with NET::SCP

    Husam Senussi, Aug 4, 2009, in forum: Ruby
    Replies:
    0
    Views:
    116
    Husam Senussi
    Aug 4, 2009
Loading...

Share This Page