Uploading big files

Discussion in 'ASP .Net Web Controls' started by ThunderMusic, Sep 26, 2006.

  1. ThunderMusic

    ThunderMusic Guest

    Hi,
    I'm currently building a little web site for me and my friends where we
    would like to be able to post big files (10Mb to 250Mb approx.) Right now
    the ASP.Net UploadFile control can't do it as we are limited 1) by the
    MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
    crashes because it only uses memory to cache the upload...

    Is there an easy way to develop an HTTP Module to handle the upload (links,
    tutorials, ...)? I've seen components to do this on the net, but they are
    too expensive for what we need to do and the budget we have (near 0.00$)...
    I know we could use FTP (there are plenty of free server softwares out
    there), but we could not use all the other features an asp.net has to offer
    such as the ease of use and data binding and everything... right now I
    would like to be able to put these files into a database, which is possible
    with the FileUpload control, but because of the "big files" requirement, I
    just can't use it...

    Anybody have ideas?

    Thanks

    ThunderMusic
    ThunderMusic, Sep 26, 2006
    #1
    1. Advertising

  2. The MaxRequestLength limit can be changed in the web.config file.

    <configuration>
    <system.web>
    <httpRuntime maxRequestLength="40960" />
    </system.web>
    </configuration>

    Please see this document:
    http://msdn.microsoft.com/library/d...-us/cpgenref/html/gngrfHttpRuntimeSection.asp

    "ThunderMusic" <> wrote in message
    news:%...
    > Hi,
    > I'm currently building a little web site for me and my friends where we
    > would like to be able to post big files (10Mb to 250Mb approx.) Right now
    > the ASP.Net UploadFile control can't do it as we are limited 1) by the
    > MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
    > crashes because it only uses memory to cache the upload...
    >
    > Is there an easy way to develop an HTTP Module to handle the upload
    > (links, tutorials, ...)? I've seen components to do this on the net, but
    > they are too expensive for what we need to do and the budget we have (near
    > 0.00$)... I know we could use FTP (there are plenty of free server
    > softwares out there), but we could not use all the other features an
    > asp.net has to offer such as the ease of use and data binding and
    > everything... right now I would like to be able to put these files into a
    > database, which is possible with the FileUpload control, but because of
    > the "big files" requirement, I just can't use it...
    >
    > Anybody have ideas?
    >
    > Thanks
    >
    > ThunderMusic
    >
    Gabriele G. Ponti, Sep 26, 2006
    #2
    1. Advertising

  3. ThunderMusic

    ThunderMusic Guest

    I know this part... We changed it, but there are 2 problems with this 1) it
    opens the server for Denial Of Service attacks and 2)even if I put it to 2Gb
    and upload a file of 200Mb, the server crashes because the FileUpload
    control caches the request entirely in memory...

    Thanks

    ThunderMusic


    "Gabriele G. Ponti" <ggponti.at.hotmail.com> wrote in message
    news:...
    > The MaxRequestLength limit can be changed in the web.config file.
    >
    > <configuration>
    > <system.web>
    > <httpRuntime maxRequestLength="40960" />
    > </system.web>
    > </configuration>
    >
    > Please see this document:
    > http://msdn.microsoft.com/library/d...-us/cpgenref/html/gngrfHttpRuntimeSection.asp
    >
    > "ThunderMusic" <> wrote in message
    > news:%...
    >> Hi,
    >> I'm currently building a little web site for me and my friends where we
    >> would like to be able to post big files (10Mb to 250Mb approx.) Right now
    >> the ASP.Net UploadFile control can't do it as we are limited 1) by the
    >> MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
    >> crashes because it only uses memory to cache the upload...
    >>
    >> Is there an easy way to develop an HTTP Module to handle the upload
    >> (links, tutorials, ...)? I've seen components to do this on the net, but
    >> they are too expensive for what we need to do and the budget we have
    >> (near 0.00$)... I know we could use FTP (there are plenty of free server
    >> softwares out there), but we could not use all the other features an
    >> asp.net has to offer such as the ease of use and data binding and
    >> everything... right now I would like to be able to put these files into
    >> a database, which is possible with the FileUpload control, but because of
    >> the "big files" requirement, I just can't use it...
    >>
    >> Anybody have ideas?
    >>
    >> Thanks
    >>
    >> ThunderMusic
    >>

    >
    >
    ThunderMusic, Sep 26, 2006
    #3
  4. I know you said your budget was $0 but I use a component called aspupload
    that works great. I think the single server license is $150.00.


    "ThunderMusic" <> wrote in message
    news:%...
    > Hi,
    > I'm currently building a little web site for me and my friends where we
    > would like to be able to post big files (10Mb to 250Mb approx.) Right now
    > the ASP.Net UploadFile control can't do it as we are limited 1) by the
    > MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
    > crashes because it only uses memory to cache the upload...
    >
    > Is there an easy way to develop an HTTP Module to handle the upload
    > (links, tutorials, ...)? I've seen components to do this on the net, but
    > they are too expensive for what we need to do and the budget we have (near
    > 0.00$)... I know we could use FTP (there are plenty of free server
    > softwares out there), but we could not use all the other features an
    > asp.net has to offer such as the ease of use and data binding and
    > everything... right now I would like to be able to put these files into a
    > database, which is possible with the FileUpload control, but because of
    > the "big files" requirement, I just can't use it...
    >
    > Anybody have ideas?
    >
    > Thanks
    >
    > ThunderMusic
    >
    Brian P. Hammer, Sep 26, 2006
    #4
  5. ThunderMusic

    ThunderMusic Guest

    it could be interesting... I'll look for it on google... (or maybe you have
    to link to the home page og the publisher?)

    I'm currently trying a free solution that is under the LGPL called :
    NeatUpload... I found it a little before noon (Eastern time USA) I'll see if
    it does the job... if it does, I'll use it, if it doesn't, I'll look for
    your aspupload... ;)

    thanks a lot...

    ThunderMusic


    "Brian P. Hammer" <> wrote in message
    news:...
    >I know you said your budget was $0 but I use a component called aspupload
    >that works great. I think the single server license is $150.00.
    >
    >
    > "ThunderMusic" <> wrote in message
    > news:%...
    >> Hi,
    >> I'm currently building a little web site for me and my friends where we
    >> would like to be able to post big files (10Mb to 250Mb approx.) Right now
    >> the ASP.Net UploadFile control can't do it as we are limited 1) by the
    >> MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
    >> crashes because it only uses memory to cache the upload...
    >>
    >> Is there an easy way to develop an HTTP Module to handle the upload
    >> (links, tutorials, ...)? I've seen components to do this on the net, but
    >> they are too expensive for what we need to do and the budget we have
    >> (near 0.00$)... I know we could use FTP (there are plenty of free server
    >> softwares out there), but we could not use all the other features an
    >> asp.net has to offer such as the ease of use and data binding and
    >> everything... right now I would like to be able to put these files into
    >> a database, which is possible with the FileUpload control, but because of
    >> the "big files" requirement, I just can't use it...
    >>
    >> Anybody have ideas?
    >>
    >> Thanks
    >>
    >> ThunderMusic
    >>

    >
    >
    ThunderMusic, Sep 26, 2006
    #5
  6. http://www.aspupload.com/ is the site. I think their main site is at
    http://www.persits.com/

    HTH,
    Brian

    "ThunderMusic" <> wrote in message
    news:%23Hj0%...
    > it could be interesting... I'll look for it on google... (or maybe you
    > have to link to the home page og the publisher?)
    >
    > I'm currently trying a free solution that is under the LGPL called :
    > NeatUpload... I found it a little before noon (Eastern time USA) I'll see
    > if it does the job... if it does, I'll use it, if it doesn't, I'll look
    > for your aspupload... ;)
    >
    > thanks a lot...
    >
    > ThunderMusic
    >
    >
    > "Brian P. Hammer" <> wrote in message
    > news:...
    >>I know you said your budget was $0 but I use a component called aspupload
    >>that works great. I think the single server license is $150.00.
    >>
    >>
    >> "ThunderMusic" <> wrote in message
    >> news:%...
    >>> Hi,
    >>> I'm currently building a little web site for me and my friends where we
    >>> would like to be able to post big files (10Mb to 250Mb approx.) Right
    >>> now the ASP.Net UploadFile control can't do it as we are limited 1) by
    >>> the MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the
    >>> server crashes because it only uses memory to cache the upload...
    >>>
    >>> Is there an easy way to develop an HTTP Module to handle the upload
    >>> (links, tutorials, ...)? I've seen components to do this on the net, but
    >>> they are too expensive for what we need to do and the budget we have
    >>> (near 0.00$)... I know we could use FTP (there are plenty of free server
    >>> softwares out there), but we could not use all the other features an
    >>> asp.net has to offer such as the ease of use and data binding and
    >>> everything... right now I would like to be able to put these files into
    >>> a database, which is possible with the FileUpload control, but because
    >>> of the "big files" requirement, I just can't use it...
    >>>
    >>> Anybody have ideas?
    >>>
    >>> Thanks
    >>>
    >>> ThunderMusic
    >>>

    >>
    >>

    >
    >
    Brian P. Hammer, Sep 26, 2006
    #6
  7. ThunderMusic

    mcrose Guest

    I was working on a component to allow a "multi-post upload"; I had to
    set it down for the moment in order to take care of more urgent work,
    but I intend to get back on it next week.

    It is a fairly simple server side script but it does require a client
    side component that I currently have coded as a VB Application and
    needs to be recoded as a downloadable .net component (it would be even
    if this was a java applet, for cross browser compatability)

    It was working, and the basic design is in place, but it could use some
    code review and exception handling. It is pretty simple and should
    allow for arbitrary small posts, and abitrarly large uploads, limitted
    only by disk availability.

    If anyone uses this, or improves it, please let me know so I can
    benefit from the code review.

    Basic concept:
    1. client creates a connection and gets a hash from server
    2. client reads file and post "postSize" bytes - server side opens file
    associated with hash, seek() to offset sent by client and writes the
    data sent. -repeat
    3. client sends a completed signal.

    There could be some optimization if you kept the file open between
    posts, but that brings up the

    The files are:
    gigaUploadClient.vb - a class that implements the client
    UploadServlet.aspx
    UploadServlet.aspx.vb - implements the server side storage
    uploadTester.vb - The code behind a form demonstrating how to use the
    client.

    uploadTester.vb:
    Public Class uploadTester

    Private _uploader As gigaUploadClient

    Private Sub btnSend_Click(ByVal sender As System.Object, ByVal e As
    System.EventArgs) Handles btnSend.Click
    btnSend.Enabled = False
    Dim postSize As Integer
    If Not Integer.TryParse(txtPostSizeKB.Text, postSize) Then
    MessageBox.Show("<" + txtPostSizeKB.Text + "> is not a
    valid integer!")
    Return
    End If
    _uploader = New gigaUploadClient(txtFile.Text, txtUrl.Text,
    postSize)
    Dim upldThrd As New Threading.Thread(AddressOf
    _uploader.SendFile)
    upldThrd.Start()
    End Sub

    Private Sub btnUpdate_Click(ByVal sender As System.Object, ByVal e
    As System.EventArgs) Handles btnUpdate.Click
    lblProgress.Text = _uploader.GetProgress + " : " +
    Math.Round(_uploader.GetProgressFraction * 100).ToString + "%"
    btnSend.Enabled = Not _uploader.IsRunning()
    End Sub
    End Class

    ********************************************************************************************
    UploadServlet.aspx:
    <%@ Page Language="VB" AutoEventWireup="false"
    CodeFile="UploadServlet.aspx.vb" Inherits="_UploadServlet" %>

    ********************************************************************************************
    Imports System
    Imports System.IO

    Partial Class _UploadServlet
    Inherits System.Web.UI.Page
    Private savePath As String
    Private Const readBlockSize As Integer = 10240

    Public Sub New()
    Dim settings As Configuration.AppSettingsReader = New
    Configuration.AppSettingsReader()
    Try
    savePath = settings.GetValue("savePath", GetType(String))
    Catch
    savePath = "c:\temp\uploader"
    End Try
    Try
    If Not Directory.Exists(savePath) Then
    Directory.CreateDirectory(savePath)
    End If
    Catch ex As Exception

    End Try
    End Sub

    Protected Sub Page_Load(ByVal sender As Object, ByVal e As
    System.EventArgs) Handles Me.Load
    ProcessUploadRequest()
    End Sub

    Private Sub restartUpload(ByVal fpath As String)
    '/* Find the seek position */
    Dim seekPos As Long = 0
    Try
    seekPos = Long.Parse(Request("offset"))
    Catch nex As Exception
    Response.Write("-1")
    Return
    End Try

    Dim fout As FileStream
    If seekPos > 0 Then
    '* Open a RandomAccessFile so that we can seek on it.
    fout = File.Create(fpath, FileMode.OpenOrCreate,
    FileOptions.RandomAccess)
    'this seek call takes significant time when the file is
    big, so only do
    'it if we get a seekposition which suggest a resume upload
    fout.Seek(seekPos, IO.SeekOrigin.Begin)
    'truncate the file to the restart position
    fout.Close()
    Response.Output.Write(seekPos.ToString)
    Return
    Else
    Response.Write("-1")
    Return
    End If
    End Sub

    Private Sub SpoolFile(ByVal fpath As String)

    Dim instr As Stream = Request.InputStream

    If (Not Directory.Exists(Path.GetDirectoryName(fpath))) Then
    Try
    Directory.CreateDirectory(Path.GetDirectoryName(fpath))
    Catch ex As Exception
    End Try
    End If

    Dim fout As FileStream
    If (File.Exists(fpath)) Then
    fout = File.OpenWrite(fpath)
    fout.Seek(0, SeekOrigin.End)
    Else
    fout = File.Create(fpath, FileMode.CreateNew)
    End If


    Dim b(readBlockSize) As Byte
    Dim i As Integer

    Do
    i = Request.InputStream.Read(b, 0, readBlockSize)
    fout.Write(b, 0, i)
    Diagnostics.Debug.Print(Text.Encoding.ASCII.GetString(b, 0,
    i))
    Loop While (i = readBlockSize)
    'Response.Write(fout.Length)
    fout.Close()

    End Sub

    Public Sub ProcessUploadRequest()
    Try
    Response.ContentType = "text/html;charset=UTF-8"

    Dim hash As String = Request("hash")
    Dim cmd As String = Request("cmd")
    Dim fname As String = Request("fname")
    If "".Equals(fname) Or fname Is Nothing Then fname =
    "downloadfile.dat"

    Diagnostics.Debug.Print("cmd ::" + cmd)
    Diagnostics.Debug.Print("fname ::" + fname)

    If hash Is Nothing Then hash = "0" +
    Guid.NewGuid().ToString + "1"
    If ("").Equals(hash) Then hash = "0" +
    Guid.NewGuid().ToString + "1"
    Dim fpath As String = Path.Combine(Path.Combine(savePath,
    hash), fname)

    Diagnostics.Debug.Print("hash ::" + hash)

    Select Case cmd

    Case "getconnection"
    Response.Output.Write(hash)
    Response.Output.Close()
    Return

    Case "getstatus"
    'this is and extra step for the current
    configuration
    'but it is useful if we want to implement a resume
    upload.
    'this is not currently implemented.
    Diagnostics.Debug.Print("writing file to " + fpath)

    If (File.Exists(fpath)) Then
    '* The file exists on the server. A previous
    upload has failed.
    '* report back how many bytes have been written
    Response.Output.Write(New
    FileInfo(fpath).Length())
    Else
    '* report back that zero bytes have been
    written
    Response.Output.Write("0")
    Response.Output.Close()
    End If

    Case "completeupload"
    ' this is where we could do any clean up or file
    moving that we want

    Response.Output.Write("Success. Bytes written: " +
    FileLen(fpath).ToString)
    Return

    Case "restart"
    restartUpload(fpath)
    Response.Output.Close()
    Return
    Case "spoolfile"
    'this is the case of spooling the file
    If (hash Is Nothing) Then
    Response.Output.Write("null hash, null cmd is
    not valid")
    End If
    SpoolFile(fpath)
    Response.Output.Close()
    Return
    End Select

    Catch e As Exception
    Response.Output.Write("failed")
    Response.Output.Write(e.Message)
    Diagnostics.Debug.Print(e.Message)
    End Try
    End Sub
    End Class


    ***********************************************************************************************

    Imports System
    Imports System.IO
    Imports System.Net

    Public Class gigaUploadClient
    Private _cli As WebClient
    Private _hash As String
    Private _fileSize As Long
    Private _reader As IO.Stream
    Private _uploadUrl As String
    Private _sourceFile As String
    Private _postSize As Long
    Private _curPos As Long = 0
    Private _lockObj As New Object
    Private _isRunning As Boolean = False

    Public Sub New(ByVal sourceFile As String, ByVal uploadUrl As
    String, ByVal postSize As Integer)
    _uploadUrl = uploadUrl
    _sourceFile = sourceFile
    _postSize = postSize
    End Sub


    Public Sub SendFile()
    _isRunning = True
    _cli = New WebClient()

    _cli.QueryString.Add("cmd", "getconnection")
    'cli.QueryString.Add("hash", "")
    _hash = _cli.DownloadString(_uploadUrl)

    'begin hack
    'trim off any whitespace. think this was because I used a
    WriteLine() on the servlet. duh!
    _hash.TrimEnd(Constants.vbCrLf.ToCharArray)
    If _hash.EndsWith(Constants.vbCrLf) Then
    _hash = _hash.Substring(0, _hash.Length - 2)
    End If
    'end hack

    Dim fileOffset As Long = getStatus()
    If fileOffset > 0 Then
    restartTransfer(fileOffset)
    End If
    spoolFile()
    _isRunning = False
    End Sub

    Private Function getStatus() As Long
    _cli.QueryString.Clear()
    _cli.QueryString.Add("cmd", "getstatus")
    _cli.QueryString.Add("hash", _hash)
    _cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))

    Dim fileOffsetStr As String = _cli.DownloadString(_uploadUrl)
    Dim fileOffset As Long
    If Long.TryParse(fileOffsetStr, fileOffset) Then
    Return fileOffset
    Else
    Return 0
    End If
    End Function

    Private Sub restartTransfer(ByVal offset As Long)
    _cli.QueryString.Clear()
    _cli.QueryString.Add("cmd", "restart")
    _cli.QueryString.Add("hash", _hash)
    _cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))

    Dim fileOffsetStr As String = _cli.DownloadString(_uploadUrl)
    SyncLock _lockObj
    _curPos = offset
    End SyncLock
    End Sub

    Private Sub confirmFileFinished()
    _cli.QueryString.Clear()
    _cli.QueryString.Add("cmd", "completeupload")
    _cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))
    _cli.QueryString.Add("hash", _hash)
    Dim resp As String = _cli.DownloadString(_uploadUrl)
    End Sub

    Private Sub spoolFile()
    Dim out As IO.Stream
    _fileSize = FileLen(_sourceFile)

    Const readBufferSize As Integer = 4096
    Dim b(readBufferSize) As Byte
    Dim postSize As Integer = _postSize * 1024
    Dim i As Integer
    Dim bytesSpooled As Long = 0

    _reader = File.OpenRead(_sourceFile)
    If _curPos > 0 Then
    _reader.Seek(_curPos, SeekOrigin.Begin)
    End If

    Do
    _cli.QueryString.Clear()
    _cli.QueryString.Add("cmd", "spoolfile")
    _cli.QueryString.Add("hash", _hash)
    _cli.QueryString.Add("fname",
    Path.GetFileName(_sourceFile))
    _cli.Headers.Add("Content-Type",
    "application/octet-stream")
    'don't write offset unless we need to resume an upload
    'cli.QueryString.Add("offset", curPos.ToString)
    out = _cli.OpenWrite(_uploadUrl)
    Dim bytesSpooledThisPost As Integer = 0
    Do
    i = _reader.Read(b, 0, readBufferSize)
    out.Write(b, 0, i)
    bytesSpooledThisPost += i
    bytesSpooled += i
    SyncLock _lockObj
    _curPos = bytesSpooled
    End SyncLock
    Loop While (i = readBufferSize And bytesSpooledThisPost <
    postSize)
    out.Close()
    Loop While (i = readBufferSize) ' i will be less then
    readBufferSize when we reach the end of the file
    End Sub

    Public Function GetProgressFraction() As Double
    SyncLock _lockObj
    Return _curPos / _fileSize
    End SyncLock
    End Function

    Public Function GetProgress() As String
    SyncLock _lockObj
    Return Math.Round(_curPos / 1024).ToString + "kb of " +
    Math.Round(_fileSize / 1024).ToString + "kb transfered"
    End SyncLock
    End Function

    Public Function IsRunning() As Boolean
    Return _isRunning
    End Function
    End Class
    mcrose, Sep 27, 2006
    #7
  8. ThunderMusic

    mcrose Guest

    Ok, piss-poor editing on that last post.
    If anyone wants the files, let me know, and I can email them.
    mcrose, Sep 27, 2006
    #8
  9. Please pass them along... bhammer at leas dot com



    Thanks,
    Brian

    "mcrose" <> wrote in message
    news:...
    > Ok, piss-poor editing on that last post.
    > If anyone wants the files, let me know, and I can email them.
    >
    Brian P. Hammer, Sep 27, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ThunderMusic

    Uploading big files

    ThunderMusic, Sep 26, 2006, in forum: ASP .Net
    Replies:
    8
    Views:
    519
    Brian P. Hammer
    Sep 27, 2006
  2. Shaguf
    Replies:
    0
    Views:
    320
    Shaguf
    Dec 24, 2008
  3. Shaguf
    Replies:
    0
    Views:
    422
    Shaguf
    Dec 26, 2008
  4. Shaguf
    Replies:
    0
    Views:
    207
    Shaguf
    Dec 26, 2008
  5. Shaguf
    Replies:
    0
    Views:
    193
    Shaguf
    Dec 24, 2008
Loading...

Share This Page