How to find uploaded data size from content-length

Discussion in 'Perl Misc' started by Asterbing, Apr 19, 2006.

  1. Asterbing

    Asterbing Guest

    Considering a multipart/form-data form, is there a formula to calculate
    the size of an uploaded file from within $ENV{'CONTENT_LENGTH'} which
    represent the size of the entire data in STDIN (including all name/value
    pairs, boundary between parts, content-disposition & type lines, some
    blank lines) ?
     
    Asterbing, Apr 19, 2006
    #1
    1. Advertising

  2. Asterbing

    Guest

    Asterbing <> wrote:
    > Considering a multipart/form-data form, is there a formula to calculate
    > the size of an uploaded file from within $ENV{'CONTENT_LENGTH'} which
    > represent the size of the entire data in STDIN (including all name/value
    > pairs, boundary between parts, content-disposition & type lines, some
    > blank lines) ?


    No.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 19, 2006
    #2
    1. Advertising

  3. Asterbing

    John Bokma Guest

    Asterbing <> wrote:

    > Considering a multipart/form-data form, is there a formula to calculate
    > the size of an uploaded file from within $ENV{'CONTENT_LENGTH'} which
    > represent the size of the entire data in STDIN (including all name/value
    > pairs, boundary between parts, content-disposition & type lines, some
    > blank lines) ?


    Why not -s filename assuming you store it in a file system?

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
     
    John Bokma, Apr 19, 2006
    #3
  4. Asterbing

    Asterbing Guest

    In article <Xns97AA6D15AEC7Fcastleamber@130.133.1.4>,
    says...
    > Why not -s filename assuming you store it in a file system?
    >


    Because I would like to avoid too big files
     
    Asterbing, Apr 19, 2006
    #4
  5. Asterbing

    Asterbing Guest

    In article <20060419114637.129$>,
    says...
    >
    > No.
    >


    Why ?
     
    Asterbing, Apr 19, 2006
    #5
  6. Asterbing

    John Bokma Guest

    Asterbing <> wrote:

    > In article <Xns97AA6D15AEC7Fcastleamber@130.133.1.4>,
    > says...
    >> Why not -s filename assuming you store it in a file system?

    >
    > Because I would like to avoid too big files


    Ok, so your question actually is: how do I avoid the uploading of big
    files:

    $CGI::pOST_MAX

    If set to a non-negative integer, this variable puts a ceiling on the
    size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
    the ceiling, it will immediately exit with an error message. This value
    will affect both ordinary POSTs and multipart POSTs, meaning that it
    limits the maximum size of file uploads as well. You should set this to a
    reasonably high value, such as 1 megabyte.


    (documentation of CGI.pm, Avoiding Denial of Service Attacks)

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
     
    John Bokma, Apr 19, 2006
    #6
  7. Asterbing

    Guest

    John Bokma <> wrote:
    > Asterbing <> wrote:
    >
    > > In article <Xns97AA6D15AEC7Fcastleamber@130.133.1.4>,
    > > says...
    > >> Why not -s filename assuming you store it in a file system?

    > >
    > > Because I would like to avoid too big files

    >
    > Ok, so your question actually is: how do I avoid the uploading of big
    > files:
    >
    > $CGI::pOST_MAX
    >
    > If set to a non-negative integer, this variable puts a ceiling on the
    > size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
    > the ceiling, it will immediately exit with an error message.


    That should probaby be changed. It does not immediately exit with
    an error message. It first reads (and discards) the entire contents
    of the POST, which apparently the OP doesn't want to happen althought it
    is kind of hard to figure out exactly what he wants. Also, it doesn't
    exit at all. It causes param to return an empty param list, and it causes
    cgi_error to return an error message--no exiting involved.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 19, 2006
    #7
  8. Asterbing

    Guest

    Asterbing <> wrote:
    > In article <20060419114637.129$>,
    > says...
    > >
    > > No.
    > >

    >
    > Why ?


    Why isn't the moon made out of green cheese?

    You pretty much covered why in the post. You can't figure out how the
    sizes of the different parts of the post (which sum up to CONTENT_LENGTH)
    are distributed until you read all but one of them. Since you can't
    gaurantee that the part which contains the file upload is the last part,
    you can't figure out how big that part is until you have read it.

    I have no idea why you think you need to do this.....

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 19, 2006
    #8
  9. Asterbing

    Asterbing Guest

    In article <20060419144430.442$>,
    says...
    > I have no idea why you think you need to do this.....
    >


    Because when CONTENT_LENGTH has been informed by server itself after
    receiving of entire POST, I can effectively read STDIN, extract uploaded
    data and do the substraction.

    But, when CONTENT_LENGTH comes from original request, how to proceed ?

    Say, I wish to avoid any upload (I mean write on server, since data may
    already be in STDIN) above 1MB, how to proceed ?
     
    Asterbing, Apr 19, 2006
    #9
  10. Asterbing

    John Bokma Guest

    wrote:

    > John Bokma <> wrote:
    >> Asterbing <> wrote:
    >>
    >> > In article <Xns97AA6D15AEC7Fcastleamber@130.133.1.4>,
    >> > says...
    >> >> Why not -s filename assuming you store it in a file system?
    >> >
    >> > Because I would like to avoid too big files

    >>
    >> Ok, so your question actually is: how do I avoid the uploading of big
    >> files:
    >>
    >> $CGI::pOST_MAX
    >>
    >> If set to a non-negative integer, this variable puts a ceiling on
    >> the
    >> size of POSTings, in bytes. If CGI.pm detects a POST that is greater
    >> than the ceiling, it will immediately exit with an error message.

    >
    > That should probaby be changed. It does not immediately exit with
    > an error message. It first reads (and discards) the entire contents
    > of the POST, which apparently the OP doesn't want to happen althought
    > it is kind of hard to figure out exactly what he wants. Also, it
    > doesn't exit at all. It causes param to return an empty param list,
    > and it causes cgi_error to return an error message--no exiting
    > involved.


    You mean it reads until it detects that the max size has been reached?
    Yes, that's what I expect, since one can keep streaming bytes, and only
    when the max has been reached the script knows it was too much.

    The second part: if the documentation is wrong, it must be fixed. Did you
    submit this?

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
     
    John Bokma, Apr 19, 2006
    #10
  11. Asterbing

    Asterbing Guest

    In article <Xns97AA8023D74CDcastleamber@130.133.1.4>,
    says...
    > $CGI::pOST_MAX
    >
    > If set to a non-negative integer, this variable puts a ceiling on the
    > size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
    > the ceiling, it will immediately exit with an error message. This value
    > will affect both ordinary POSTs and multipart POSTs, meaning that it
    > limits the maximum size of file uploads as well. You should set this to a
    > reasonably high value, such as 1 megabyte.
    >


    Without CGI.pm !

    However, reading what you posted from CGI.pm help, it seems that CGI.pm
    doesn't really limit the file size precisely but rather the entire
    POSTed data. So, how if I want to be free to choose the file limit not
    as 1MB but as 5KB ?
     
    Asterbing, Apr 19, 2006
    #11
  12. Asterbing

    Asterbing Guest

    In article <20060419143435.067$>,
    says...
    > John Bokma <> wrote:
    > > Asterbing <> wrote:
    > >
    > > > In article <Xns97AA6D15AEC7Fcastleamber@130.133.1.4>,
    > > > says...
    > > >> Why not -s filename assuming you store it in a file system?
    > > >
    > > > Because I would like to avoid too big files

    > >
    > > Ok, so your question actually is: how do I avoid the uploading of big
    > > files:
    > >
    > > $CGI::pOST_MAX
    > >
    > > If set to a non-negative integer, this variable puts a ceiling on the
    > > size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
    > > the ceiling, it will immediately exit with an error message.

    >
    > That should probaby be changed. It does not immediately exit with
    > an error message. It first reads (and discards) the entire contents
    > of the POST, which apparently the OP doesn't want to happen althought it
    > is kind of hard to figure out exactly what he wants. Also, it doesn't
    > exit at all. It causes param to return an empty param list, and it causes
    > cgi_error to return an error message--no exiting involved.
    >
    > Xho
    >
    >


    OK, so it doesn't specifically limit the size of uploadable file (but
    size of entire POST) and it doesn't avoid sending of entire POST data
    from web browser to server's STDIN ?
     
    Asterbing, Apr 19, 2006
    #12
  13. Asterbing

    Asterbing Guest

    In article <Xns97AA916DBEDB4castleamber@130.133.1.4>,
    says...
    > You mean it reads until it detects that the max size has been reached?
    > Yes, that's what I expect, since one can keep streaming bytes, and only
    > when the max has been reached the script knows it was too much.
    >


    When CONTENT_LENGTH not given during request, server has to compute it
    from real data received in STDIN : so, in this case how does CGI.pm is
    able to stop receiving to STDIN precisely when $CGI::pOST_MAX is reached
    ?

    Not sure it's able to do that. What do you think ?
     
    Asterbing, Apr 19, 2006
    #13
  14. Asterbing

    John Bokma Guest

    Asterbing <> wrote:

    > In article <Xns97AA8023D74CDcastleamber@130.133.1.4>,
    > says...
    >> $CGI::pOST_MAX
    >>
    >> If set to a non-negative integer, this variable puts a ceiling on
    >> the
    >> size of POSTings, in bytes. If CGI.pm detects a POST that is greater
    >> than the ceiling, it will immediately exit with an error message.
    >> This value will affect both ordinary POSTs and multipart POSTs,
    >> meaning that it limits the maximum size of file uploads as well. You
    >> should set this to a reasonably high value, such as 1 megabyte.
    >>

    >
    > Without CGI.pm !


    Why? Look how "they" do it, and copy it.

    > However, reading what you posted from CGI.pm help, it seems that
    > CGI.pm doesn't really limit the file size precisely but rather the
    > entire POSTed data. So, how if I want to be free to choose the file
    > limit not as 1MB but as 5KB ?


    Good question, set it to 50KB, and check the actual filesize.

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
     
    John Bokma, Apr 19, 2006
    #14
  15. Asterbing

    Denver Guest

    Asterbing wrote:
    > xhoster wrote:
    >> John Bokma wrote:
    >> > Ok, so your question actually is: how do I avoid the uploading of big
    >> > files:
    >> >
    >> > $CGI::pOST_MAX
    >> >
    >> > If set to a non-negative integer, this variable puts a ceiling on the
    >> > size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
    >> > the ceiling, it will immediately exit with an error message.

    >>
    >> That should probaby be changed. It does not immediately exit with
    >> an error message. It first reads (and discards) the entire contents
    >> of the POST, which apparently the OP doesn't want to happen


    It would make sense to me to send an error message, close the connection, and exit.
    Otherwise, a DoS attack indeed keeps streaming bits to the server.


    >> it doesn't exit at all. It causes param to return an empty param list, and it causes
    >> cgi_error to return an error message--no exiting involved.

    >
    > OK, so it doesn't specifically limit the size of uploadable file
    > (but size of entire POST)
    > and it doesn't avoid sending of entire POST data from web browser to server's STDIN ?


    STDIN isn't the right term.
    Apparently the incoming data continue filling the TCP receive buffer for that socket.
     
    Denver, Apr 19, 2006
    #15
  16. Asterbing

    Guest

    John Bokma <> wrote:
    > wrote:
    >
    > > John Bokma <> wrote:
    > >> Asterbing <> wrote:
    > >>
    > >> > In article <Xns97AA6D15AEC7Fcastleamber@130.133.1.4>,
    > >> > says...
    > >> >> Why not -s filename assuming you store it in a file system?
    > >> >
    > >> > Because I would like to avoid too big files
    > >>
    > >> Ok, so your question actually is: how do I avoid the uploading of big
    > >> files:
    > >>
    > >> $CGI::pOST_MAX
    > >>
    > >> If set to a non-negative integer, this variable puts a ceiling on
    > >> the
    > >> size of POSTings, in bytes. If CGI.pm detects a POST that is greater
    > >> than the ceiling, it will immediately exit with an error message.

    > >
    > > That should probaby be changed. It does not immediately exit with
    > > an error message. It first reads (and discards) the entire contents
    > > of the POST, which apparently the OP doesn't want to happen althought
    > > it is kind of hard to figure out exactly what he wants. Also, it
    > > doesn't exit at all. It causes param to return an empty param list,
    > > and it causes cgi_error to return an error message--no exiting
    > > involved.

    >
    > You mean it reads until it detects that the max size has been reached?


    No. It compares CONTENT_LENGTH to POST_MAX. If CONTENT_LENGTH is greater
    than POST_MAX, then it reads the entire post, but doesn't do anything with
    it (doesn't parse it, doesn't save it to disk, just reads and throws away
    in chunks.) If CONTENT_LENGTH is less than POST_MAX, it reads the post
    and parses it (saving parts of it to disk, if appropriate.)


    > Yes, that's what I expect, since one can keep streaming bytes, and only
    > when the max has been reached the script knows it was too much.


    But AFAICT, it doesn't do it this way. It seems to trust CONTENT_LENGTH
    absolutely, and if CONTENT_LENGTH is wrong it does nothing to detect this.

    > The second part: if the documentation is wrong, it must be fixed. Did you
    > submit this?


    I haven't yet. I'm not sure how I would recommend changing it. The
    read-but-discard thing may be an (only sometimes important) implementation
    detail that does not need to be documented. The "immediate exit" is
    clarified later on the documention, and I'm not sure how to clarify here
    also without causing bloat.


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 19, 2006
    #16
  17. Asterbing

    Guest

    Asterbing <> wrote:
    > In article <Xns97AA8023D74CDcastleamber@130.133.1.4>,
    > says...
    > > $CGI::pOST_MAX
    > >
    > > If set to a non-negative integer, this variable puts a ceiling on
    > > the size of POSTings, in bytes. If CGI.pm detects a POST that is
    > > greater than the ceiling, it will immediately exit with an error
    > > message. This value will affect both ordinary POSTs and multipart
    > > POSTs, meaning that it limits the maximum size of file uploads as well.
    > > You should set this to a reasonably high value, such as 1 megabyte.
    > >

    >
    > Without CGI.pm !
    >
    > However, reading what you posted from CGI.pm help, it seems that CGI.pm
    > doesn't really limit the file size precisely but rather the entire
    > POSTed data. So, how if I want to be free to choose the file limit not
    > as 1MB but as 5KB ?


    Do you really consider 5KB to be a denial of service attack?

    Set one limit on the whole post of, say, 10MB, to prevent DoS attacks.
    If the size is over 10MB, send a nasty message about DoS not being welcome.
    If it is under 10MB, process it, save the upload to a temp file, and
    check its size. If the size is more than 5KB, send a polite message
    saying the file was a tad too big, and delete the temp file. Otherwise,
    do whatever it is you want to do.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 19, 2006
    #17
  18. Asterbing

    Guest

    Asterbing <> wrote:
    > In article <20060419144430.442$>,
    > says...
    > > I have no idea why you think you need to do this.....
    > >

    >
    > Because when CONTENT_LENGTH has been informed by server itself after
    > receiving of entire POST, I can effectively read STDIN, extract uploaded
    > data and do the substraction.


    Once you've extracted the data, why would you need to do the subtraction?
    Once you've extracted the data, you already have the data, just ask the
    data how big it is. I thought the whole point was to avoid extracting the
    data in the first place.

    >
    > But, when CONTENT_LENGTH comes from original request, how to proceed ?
    >
    > Say, I wish to avoid any upload (I mean write on server, since data may
    > already be in STDIN) above 1MB, how to proceed ?


    You have enough memory to load a 1 MB of non-file-upload form-data into
    memory, but not enough disk to temporarily save 1 MB of file-upload data?
    That just doesn't make sense. If you computer will break with 1MB of
    posted data, don't allow that size of post, whether it is file upload or
    not. If your computer won't break, then what's the problem? Uploaded it,
    ask how big each form-part is, and do the appropriate thing.

    Having said that, you could hack/subclass the read_multipart method
    of CGI.pm to have it, at a certain size, stop copying the data into the
    file, something like:

    while (defined($data = $buffer->read)) {
    if (defined $self->{'.upload_hook'})
    {
    $totalbytes += length($data);
    &{$self->{'.upload_hook'}}($filename ,$data, $totalbytes,
    $self->{'.upload_data'});
    }
    ##print $filehandle $data;
    print $filehandle $data unless $totalbytes > $FOO::Asterbing_size;
    }

    Of course, you would have to set a flag or something so that you know that
    this has happened.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 19, 2006
    #18
  19. Asterbing <> writes:

    > Because when CONTENT_LENGTH has been informed by server itself after
    > receiving of entire POST, I can effectively read STDIN, extract uploaded
    > data and do the substraction.


    Don't read from stdin. Read from the filehandle you get from CGI.pm.

    > Say, I wish to avoid any upload (I mean write on server, since data may
    > already be in STDIN) above 1MB, how to proceed ?


    Wish for something that's possible.

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
     
    Sherm Pendley, Apr 19, 2006
    #19
  20. Asterbing <> writes:

    > In article <Xns97AA916DBEDB4castleamber@130.133.1.4>,
    > says...
    >> You mean it reads until it detects that the max size has been reached?
    >> Yes, that's what I expect, since one can keep streaming bytes, and only
    >> when the max has been reached the script knows it was too much.
    >>

    >
    > When CONTENT_LENGTH not given during request, server has to compute it
    > from real data received in STDIN


    The server isn't receiving data from stdin, it's receiving it from a net-
    work connection.

    > : so, in this case how does CGI.pm is
    > able to stop receiving to STDIN precisely when $CGI::pOST_MAX is reached


    It doesn't make the client stop sending data, it just ignores anything
    beyond POST_MAX.

    > Not sure it's able to do that. What do you think ?


    It can and does stop receiving data. Your confusion is the result of
    thinking that it *also* causes the client to stop *sending* data.

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
     
    Sherm Pendley, Apr 19, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt G
    Replies:
    1
    Views:
    1,198
    Deepak Kumar Vasudevan
    Aug 22, 2003
  2. SamIAm
    Replies:
    0
    Views:
    418
    SamIAm
    Feb 9, 2004
  3. SamIAm
    Replies:
    0
    Views:
    375
    SamIAm
    Feb 10, 2004
  4. Replies:
    2
    Views:
    2,750
  5. Replies:
    2
    Views:
    345
    Christian Tismer
    Feb 23, 2004
Loading...

Share This Page