resume upload wsgi script

Discussion in 'Python' started by gert, Aug 9, 2009.

  1. gert

    gert Guest

    I working on a resume upload script and encountered the following
    problems

    sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    '\ufffd\ufff
    d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    Content-Disposition: form-data; name="Filename"

    DSC00013.JPG
    ------------Ef1

    when I dont use SUBSTR or COALESCE and || it works

    def application(environ, response):
    range=environ.get('HTTP_RANGE','bytes=0-').replace
    ('bytes=','').split(',')
    offset=[]
    for r in range: offset.append(r.split('-'))
    db.execute('UPDATE users SET picture=SUBSTR(picture,?) WHERE
    uid=?', (offset[0][0],s.UID))

    chunk=environ['wsgi.input'].read(int(environ
    ['CONTENT_LENGTH'])).decode('latin1')
    db.execute('UPDATE users SET picture=COALESCE(picture,?)||? WHERE
    uid=?', (''.encode('latin1'),chunk.encode('latin1'), s.UID))

    #while True:
    # chunk=environ['wsgi.input'].read(8192).decode('latin1')
    # if not chunk: break
    # db.execute("UPDATE users SET picture=COALESCE
    (picture,'')||? WHERE uid=?", (chunk, s.UID))

    the following is the POST clean up when PUT is not used, does anyone
    know how to do this without loading the complete blob into memory ?

    db.execute('SELECT picture FROM users WHERE uid=?',(s.UID,))
    f=db.fetch()
    b=environ['CONTENT_TYPE'].split('boundary=')[1]
    p=search(b+r'.*?Content-Type: application/octet-stream\r\n\r\n(.*?)
    \r\n--'+b,f[0][0].decode('latin1'),DOTALL).group(1)
    db.execute('UPDATE users SET picture=? WHERE uid=?', (p.encode
    ('latin1'), s.UID))
     
    gert, Aug 9, 2009
    #1
    1. Advertising

  2. gert schrieb:
    > I working on a resume upload script and encountered the following
    > problems
    >
    > sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    > '\ufffd\ufff
    > d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    > Content-Disposition: form-data; name="Filename"


    You are treating a binary data column as if it were a string. That's
    bogus, you need to use a blob column.

    Also I wouldn't combine the uplodaded chunks until the full upload is
    finished - and even then only if I actually need the data.

    Diez
     
    Diez B. Roggisch, Aug 9, 2009
    #2
    1. Advertising

  3. gert

    gert Guest

    On Aug 9, 3:17 pm, "Diez B. Roggisch" <> wrote:
    > gert schrieb:
    >
    > > I working on a resume upload script and encountered the following
    > > problems

    >
    > > sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    > > '\ufffd\ufff
    > > d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    > > Content-Disposition: form-data; name="Filename"

    >
    > You are treating a binary data column as if it were a string. That's
    > bogus, you need to use a blob column.
    >
    > Also I wouldn't combine the uplodaded chunks until the full upload is
    > finished - and even then only if I actually need the data.
    >
    > Diez


    And the best solution would be to use TEXT instead or some sort of
    SUBBIN that i do not know of in sqlite ?
     
    gert, Aug 9, 2009
    #3
  4. gert schrieb:
    > On Aug 9, 3:17 pm, "Diez B. Roggisch" <> wrote:
    >> gert schrieb:
    >>
    >>> I working on a resume upload script and encountered the following
    >>> problems
    >>> sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    >>> '\ufffd\ufff
    >>> d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    >>> Content-Disposition: form-data; name="Filename"

    >> You are treating a binary data column as if it were a string. That's
    >> bogus, you need to use a blob column.
    >>
    >> Also I wouldn't combine the uplodaded chunks until the full upload is
    >> finished - and even then only if I actually need the data.
    >>
    >> Diez

    >
    > And the best solution would be to use TEXT instead or some sort of
    > SUBBIN that i do not know of in sqlite ?
    >


    No, the best solution would be to use "BLOB", and no SUB*-stuff, but
    instead a 1:n-relation of chunks, that when the upload is finished you
    can easily combine in python to one large blob, storing that in the DB
    again.

    Diez
     
    Diez B. Roggisch, Aug 9, 2009
    #4
  5. gert

    gert Guest

    On Aug 9, 4:42 pm, "Diez B. Roggisch" <> wrote:
    > gert schrieb:
    >
    >
    >
    > > On Aug 9, 3:17 pm, "Diez B. Roggisch" <> wrote:
    > >> gert schrieb:

    >
    > >>> I working on a resume upload script and encountered the following
    > >>> problems
    > >>> sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    > >>> '\ufffd\ufff
    > >>> d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    > >>> Content-Disposition: form-data; name="Filename"
    > >> You are treating a binary data column as if it were a string. That's
    > >> bogus, you need to use a blob column.

    >
    > >> Also I wouldn't combine the uplodaded chunks until the full upload is
    > >> finished - and even then only if I actually need the data.

    >
    > >> Diez

    >
    > > And the best solution would be to use TEXT instead or some sort of
    > > SUBBIN that i do not know of in sqlite ?

    >
    > No, the best solution would be to use "BLOB", and no SUB*-stuff, but
    > instead a 1:n-relation of chunks, that when the upload is finished you
    > can easily combine in python to one large blob, storing that in the DB
    > again.
    >
    > Diez


    so one table of chunks

    CREATE TABLE temp (
    file_id VARCHAR(64),
    chunK_id INTEGER,
    chunk BLOB,
    PRIMARY KEY(file_id,chunk_id)
    );

    SELECT chunk FROM temp WHERE file_id = 'file'
    concatenating result in python
    update blob
    delete temp

    How do I concatenate results that do not fit into memory ?
     
    gert, Aug 9, 2009
    #5
  6. gert schrieb:
    > On Aug 9, 4:42 pm, "Diez B. Roggisch" <> wrote:
    >> gert schrieb:
    >>
    >>
    >>
    >>> On Aug 9, 3:17 pm, "Diez B. Roggisch" <> wrote:
    >>>> gert schrieb:
    >>>>> I working on a resume upload script and encountered the following
    >>>>> problems
    >>>>> sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    >>>>> '\ufffd\ufff
    >>>>> d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    >>>>> Content-Disposition: form-data; name="Filename"
    >>>> You are treating a binary data column as if it were a string. That's
    >>>> bogus, you need to use a blob column.
    >>>> Also I wouldn't combine the uplodaded chunks until the full upload is
    >>>> finished - and even then only if I actually need the data.
    >>>> Diez
    >>> And the best solution would be to use TEXT instead or some sort of
    >>> SUBBIN that i do not know of in sqlite ?

    >> No, the best solution would be to use "BLOB", and no SUB*-stuff, but
    >> instead a 1:n-relation of chunks, that when the upload is finished you
    >> can easily combine in python to one large blob, storing that in the DB
    >> again.
    >>
    >> Diez

    >
    > so one table of chunks
    >
    > CREATE TABLE temp (
    > file_id VARCHAR(64),
    > chunK_id INTEGER,
    > chunk BLOB,
    > PRIMARY KEY(file_id,chunk_id)
    > );
    >
    > SELECT chunk FROM temp WHERE file_id = 'file'
    > concatenating result in python
    > update blob
    > delete temp
    >
    > How do I concatenate results that do not fit into memory ?


    By writing them into one file? If files were to large for your memory,
    all the substring-stuff wouldn't help either - the sqlite works in the
    same memory as your program...

    But how many gigabytes of uploads do you expect?

    Diez
     
    Diez B. Roggisch, Aug 9, 2009
    #6
  7. gert

    gert Guest

    On Aug 9, 8:25 pm, "Diez B. Roggisch" <> wrote:
    > gert schrieb:
    >
    >
    >
    > > On Aug 9, 4:42 pm, "Diez B. Roggisch" <> wrote:
    > >> gert schrieb:

    >
    > >>> On Aug 9, 3:17 pm, "Diez B. Roggisch" <> wrote:
    > >>>> gert schrieb:
    > >>>>> I working on a resume upload script and encountered the following
    > >>>>> problems
    > >>>>> sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
    > >>>>> '\ufffd\ufff
    > >>>>> d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
    > >>>>> Content-Disposition: form-data; name="Filename"
    > >>>> You are treating a binary data column as if it were a string. That's
    > >>>> bogus, you need to use a blob column.
    > >>>> Also I wouldn't combine the uplodaded chunks until the full upload is
    > >>>> finished - and even then only if I actually need the data.
    > >>>> Diez
    > >>> And the best solution would be to use TEXT instead or some sort of
    > >>> SUBBIN that i do not know of in sqlite ?
    > >> No, the best solution would be to use "BLOB", and no SUB*-stuff, but
    > >> instead a 1:n-relation of chunks, that when the upload is finished you
    > >> can easily combine in python to one large blob, storing that in the DB
    > >> again.

    >
    > >> Diez

    >
    > > so one table of chunks

    >
    > > CREATE TABLE temp (
    > >     file_id  VARCHAR(64),
    > >     chunK_id  INTEGER,
    > >     chunk BLOB,
    > >     PRIMARY KEY(file_id,chunk_id)
    > > );

    >
    > > SELECT chunk FROM temp WHERE file_id = 'file'
    > > concatenating result in python
    > > update blob
    > > delete temp

    >
    > > How do I concatenate results that do not fit into memory ?

    >
    > By writing them into one file? If files were to large for your memory,
    > all the substring-stuff wouldn't help either - the sqlite works in the
    > same memory as your program...
    >
    > But how many gigabytes of uploads do you expect?
    >


    250KB :)
    Its just HTTP1.1 has everything for making ftp like file transfers
    possible.
    When I write it to a file then I am back at square one because I still
    need to load it completely to get it into a blob.
    So there is no way to concatenate BLOB's without loading it completely
    into memory ?
     
    gert, Aug 9, 2009
    #7
  8. >
    > 250KB :)


    So why do you bother?

    > Its just HTTP1.1 has everything for making ftp like file transfers
    > possible.
    > When I write it to a file then I am back at square one because I still
    > need to load it completely to get it into a blob.


    Well, the blob is nothing but datat in the file-system. If you are
    *really* concerned about that, then don't use the db, but use the
    filesystem, appending to the file until it's finished - and storing a
    reference to it to the DB. We do that as well, because otherwise the db
    will become unmanagable anyway (dumping, backups).

    > So there is no way to concatenate BLOB's without loading it completely
    > into memory ?


    In theory, the DB might offer special stream-based methods for these
    kinds of tasks, but the db-api lacks them. Some DB-adapters might offer
    some non-standard-extensions, but I don't think sqlite does.

    Diez
     
    Diez B. Roggisch, Aug 10, 2009
    #8
  9. gert

    gert Guest

    On Aug 10, 10:39 am, "Diez B. Roggisch" <> wrote:
    > > 250KB :)

    >
    > So why do you bother?
    >
    > > Its just HTTP1.1 has everything for making ftp like file transfers
    > > possible.
    > > When I write it to a file then I am back at square one because I still
    > > need to load it completely to get it into a blob.

    >
    > Well, the blob is nothing but datat in the file-system. If you are
    > *really* concerned about that, then don't use the db, but use the
    > filesystem, appending to the file until it's finished - and storing a
    > reference to it to the DB. We do that as well, because otherwise the db
    > will become unmanagable anyway (dumping, backups).
    >


    I also hate debugging sql that contains blob data. So if using files
    how do you clean the http post stuff?
    (b+r'.*?Content-Type: application/octet-stream\r\n\r\n(.*?)\r
    \n--'+b,file,DOTALL)
    using as litlle memory as possible ?
    I can not use PUT upload only because flash does not support PUT.
     
    gert, Aug 10, 2009
    #9
  10. gert schrieb:
    > On Aug 10, 10:39 am, "Diez B. Roggisch" <> wrote:
    >>> 250KB :)

    >> So why do you bother?
    >>
    >>> Its just HTTP1.1 has everything for making ftp like file transfers
    >>> possible.
    >>> When I write it to a file then I am back at square one because I still
    >>> need to load it completely to get it into a blob.

    >> Well, the blob is nothing but datat in the file-system. If you are
    >> *really* concerned about that, then don't use the db, but use the
    >> filesystem, appending to the file until it's finished - and storing a
    >> reference to it to the DB. We do that as well, because otherwise the db
    >> will become unmanagable anyway (dumping, backups).
    >>

    >
    > I also hate debugging sql that contains blob data. So if using files
    > how do you clean the http post stuff?
    > (b+r'.*?Content-Type: application/octet-stream\r\n\r\n(.*?)\r
    > \n--'+b,file,DOTALL)
    > using as litlle memory as possible ?
    > I can not use PUT upload only because flash does not support PUT.


    PUT or POST has nothing todo with that.

    It's the question if you need formencoded data or not. If not, you can
    simply upload the data directly, no nead to cleanup anything.

    For decoding formencoded data, you will need to write your own parser if
    you insist on these very debatable memory-constraints of yours.

    Diez
     
    Diez B. Roggisch, Aug 13, 2009
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. KGuru
    Replies:
    4
    Views:
    5,882
    Thomas Weidenfeller
    Mar 23, 2006
  2. Robert Brewer

    Re: Snakelets and WSGI

    Robert Brewer, Oct 12, 2004, in forum: Python
    Replies:
    2
    Views:
    359
    Paul Boddie
    Oct 13, 2004
  3. Robert Brewer

    RE: Snakelets and WSGI

    Robert Brewer, Oct 13, 2004, in forum: Python
    Replies:
    3
    Views:
    363
    Irmen de Jong
    Oct 13, 2004
  4. Jxzzy78

    resume upload

    Jxzzy78, Jul 5, 2005, in forum: Python
    Replies:
    1
    Views:
    446
    Jeff Epler
    Jul 5, 2005
  5. Hidura

    Upload files with wsgi

    Hidura, Sep 28, 2010, in forum: Python
    Replies:
    3
    Views:
    1,524
    Richard Thomas
    Sep 29, 2010
Loading...

Share This Page