Large file uploads using Java

Discussion in 'Java' started by heather.fraser@gmail.com, Oct 14, 2007.

  1. Guest

    I am well aware of several file upload servlets & libraries (including
    the one from Apache/Jakarta).

    However, how does one go about creating a web-based file upload
    similar to Whalemail (www.whalemail.com) or many of the AJAX web2.0
    sites I see where they show the status of the upload (in KB and %age)
    as the upload progresses ?

    Also I now find myself with the need to upload files as large as 500
    MB over a Local Area Network - is the servlet going to be able to
    handle this or will it try to cache the uploaded data in memory and
    therefore run into memory limits on the server ?

    Thank you for any advice,

    Heather
    , Oct 14, 2007
    #1
    1. Advertising

  2. wrote:
    > I am well aware of several file upload servlets & libraries (including
    > the one from Apache/Jakarta).
    >
    > However, how does one go about creating a web-based file upload
    > similar to Whalemail (www.whalemail.com) or many of the AJAX web2.0
    > sites I see where they show the status of the upload (in KB and %age)
    > as the upload progresses ?
    >
    > Also I now find myself with the need to upload files as large as 500
    > MB over a Local Area Network - is the servlet going to be able to
    > handle this or will it try to cache the uploaded data in memory and
    > therefore run into memory limits on the server ?


    500 MB in memory should be OK on todays HW.

    But let us assume DVD sise instead of CD size.

    I don't think Jakarta Common FileUpload can stream directly from
    request to final destination.

    But I think Com Oreilly Servlet can.

    I have some code somewhere that can show uploaded bytes
    being incremented in a separate window (a refreshing window - this
    was made before AJAX became popular).

    Arne
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=, Oct 14, 2007
    #2
    1. Advertising

  3. Guest

    Thank you for the input, Arne.

    You're right actually - we WILL need to be able to upload DVD-sized
    objects at some stage. Architecturally, one of the key considerations
    has been whether we should use the web browser (and HTTP) for file-
    upload/transfer; or FTP separately & then regularly ingest the files
    on the server - but this introduces the problem of synchronizing the
    uploaded file with the user & the adding of meta data. In the long
    term, we believe all things are going the web-route and it is easier
    for the end users.

    We already use the Oreilly servlet actually so we'll see what happens
    with massive gargantuan file uploads and whether the JVM & Tomcat
    choke. However, how does one show & dynamically update the percentage
    uploaded with this?

    Thanks again,

    Heather



    On Oct 14, 12:35 pm, Arne Vajhøj <> wrote:
    > wrote:
    > > I am well aware of several file upload servlets & libraries (including
    > > the one from Apache/Jakarta).

    >
    > > However, how does one go about creating a web-based file upload
    > > similar to Whalemail (www.whalemail.com) or many of the AJAX web2.0
    > > sites I see where they show the status of the upload (in KB and %age)
    > > as the upload progresses ?

    >
    > > Also I now find myself with the need to upload files as large as 500
    > > MB over a Local Area Network - is the servlet going to be able to
    > > handle this or will it try to cache the uploaded data in memory and
    > > therefore run into memory limits on the server ?

    >
    > 500 MB in memory should be OK on todays HW.
    >
    > But let us assume DVD sise instead of CD size.
    >
    > I don't think Jakarta Common FileUpload can stream directly from
    > request to final destination.
    >
    > But I think Com Oreilly Servlet can.
    >
    > I have some code somewhere that can show uploaded bytes
    > being incremented in a separate window (a refreshing window - this
    > was made before AJAX became popular).
    >
    > Arne
    , Oct 14, 2007
    #3
  4. Roedy Green Guest

    On Sun, 14 Oct 2007 04:17:21 -0000, wrote,
    quoted or indirectly quoted someone who said :

    >Also I now find myself with the need to upload files as large as 500
    >MB over a Local Area Network - is the servlet going to be able to
    >handle this or will it try to cache the uploaded data in memory and
    >therefore run into memory limits on the server ?


    See the code for download at
    http://mindprod.com/products1.html#FILETRANSFER

    It uses use a read method that reads in chunks. You could attach your
    progress bar to that to jump x ticks for every byte read in the chunk.
    You have to change the read to a write, but the skeletons you need are
    there.

    see http://mindprod.com/jgloss/progress.html

    You need something on the server side to decide when to accept files.
    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Oct 14, 2007
    #4
  5. wrote:
    > We already use the Oreilly servlet actually so we'll see what happens
    > with massive gargantuan file uploads and whether the JVM & Tomcat
    > choke. However, how does one show & dynamically update the percentage
    > uploaded with this?


    If you read directly from network stream to file stream, then it should
    not require much memory.

    Just save percentages in session and have a refreshing window that
    calls something that retrieves that value.

    Arne
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=, Oct 14, 2007
    #5
  6. Roedy Green Guest

    On Sun, 14 Oct 2007 09:29:10 GMT, Roedy Green
    <> wrote, quoted or indirectly quoted
    someone who said :

    >See the code for download at
    >http://mindprod.com/products1.html#FILETRANSFER


    for giant files, you need ether to be able to restart part way through
    or you need to transfer in chunks, perhaps a meg each. Then combine
    them when all done. You also want to compress them.
    --
    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
    Roedy Green, Oct 15, 2007
    #6
  7. Guest

    > >See the code for download at
    > >http://mindprod.com/products1.html#FILETRANSFER


    Thank you. I shall take a look.

    > for giant files, you need ether to be able to restart part way through


    Yes, that's a good point. This is turning out to be not such a trivial
    upload question after all.

    > or you need to transfer in chunks, perhaps a meg each. Then combine
    > them when all done. You also want to compress them.


    Giant files tend to be binaries so I'm not sure the compression is
    going to help that much, is it ?

    Thanks for the help,

    Heather.
    , Oct 15, 2007
    #7
  8. Guest

    On Oct 15, 5:51 am, Arne Vajhøj <> wrote:
    > wrote:
    > > We already use the Oreilly servlet actually so we'll see what happens
    > > with massive gargantuan file uploads and whether the JVM & Tomcat
    > > choke. However, how does one show & dynamically update the percentage
    > > uploaded with this?

    >
    > If you read directly from network stream to file stream, then it should
    > not require much memory.
    >
    > Just save percentages in session and have a refreshing window that
    > calls something that retrieves that value.


    Wow, that's a fantastic idea and so simple. Thank you so much, Arne,
    I always thought it was so much more complicated than that :)
    , Oct 15, 2007
    #8
  9. Lew Guest

    wrote:
    > Giant files tend to be binaries so I'm not sure the compression is
    > going to help that much, is it ?


    Binary files compress. Typical binaries, like visual images or program code,
    display enough regularity for compression to be effective.

    Encrypted or already-compressed files are another matter.

    --
    Lew
    Lew, Oct 15, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rajeev Tipnis

    Handling Errors in Large File Uploads

    Rajeev Tipnis, Nov 10, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    419
    Rajeev Tipnis
    Nov 10, 2004
  2. Replies:
    0
    Views:
    286
  3. James Jones

    Large File Uploads

    James Jones, Sep 28, 2006, in forum: ASP General
    Replies:
    1
    Views:
    120
    Cowboy \(Gregory A. Beamer\)
    Sep 28, 2006
  4. Kal
    Replies:
    10
    Views:
    584
    David Wang
    Sep 17, 2008
  5. JMB
    Replies:
    10
    Views:
    201
Loading...

Share This Page