resume upload wsgi script

G

gert

I working on a resume upload script and encountered the following
problems

sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
'\ufffd\ufff
d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
Content-Disposition: form-data; name="Filename"

DSC00013.JPG
------------Ef1

when I dont use SUBSTR or COALESCE and || it works

def application(environ, response):
range=environ.get('HTTP_RANGE','bytes=0-').replace
('bytes=','').split(',')
offset=[]
for r in range: offset.append(r.split('-'))
db.execute('UPDATE users SET picture=SUBSTR(picture,?) WHERE
uid=?', (offset[0][0],s.UID))

chunk=environ['wsgi.input'].read(int(environ
['CONTENT_LENGTH'])).decode('latin1')
db.execute('UPDATE users SET picture=COALESCE(picture,?)||? WHERE
uid=?', (''.encode('latin1'),chunk.encode('latin1'), s.UID))

#while True:
# chunk=environ['wsgi.input'].read(8192).decode('latin1')
# if not chunk: break
# db.execute("UPDATE users SET picture=COALESCE
(picture,'')||? WHERE uid=?", (chunk, s.UID))

the following is the POST clean up when PUT is not used, does anyone
know how to do this without loading the complete blob into memory ?

db.execute('SELECT picture FROM users WHERE uid=?',(s.UID,))
f=db.fetch()
b=environ['CONTENT_TYPE'].split('boundary=')[1]
p=search(b+r'.*?Content-Type: application/octet-stream\r\n\r\n(.*?)
\r\n--'+b,f[0][0].decode('latin1'),DOTALL).group(1)
db.execute('UPDATE users SET picture=? WHERE uid=?', (p.encode
('latin1'), s.UID))
 
D

Diez B. Roggisch

gert said:
I working on a resume upload script and encountered the following
problems

sql: Could not decode to UTF-8 column 'SUBSTR(picture,?)' with text
'\ufffd\ufff
d\ufffd\ufffd↑!Exif------------Ef1gL6KM7Ij5ae0gL6KM7cH2cH2GI3
Content-Disposition: form-data; name="Filename"

You are treating a binary data column as if it were a string. That's
bogus, you need to use a blob column.

Also I wouldn't combine the uplodaded chunks until the full upload is
finished - and even then only if I actually need the data.

Diez
 
G

gert

You are treating a binary data column as if it were a string. That's
bogus, you need to use a blob column.

Also I wouldn't combine the uplodaded chunks until the full upload is
finished - and even then only if I actually need the data.

Diez

And the best solution would be to use TEXT instead or some sort of
SUBBIN that i do not know of in sqlite ?
 
D

Diez B. Roggisch

gert said:
And the best solution would be to use TEXT instead or some sort of
SUBBIN that i do not know of in sqlite ?

No, the best solution would be to use "BLOB", and no SUB*-stuff, but
instead a 1:n-relation of chunks, that when the upload is finished you
can easily combine in python to one large blob, storing that in the DB
again.

Diez
 
G

gert

gert schrieb:





No, the best solution would be to use "BLOB", and no SUB*-stuff, but
instead a 1:n-relation of chunks, that when the upload is finished you
can easily combine in python to one large blob, storing that in the DB
again.

Diez

so one table of chunks

CREATE TABLE temp (
file_id VARCHAR(64),
chunK_id INTEGER,
chunk BLOB,
PRIMARY KEY(file_id,chunk_id)
);

SELECT chunk FROM temp WHERE file_id = 'file'
concatenating result in python
update blob
delete temp

How do I concatenate results that do not fit into memory ?
 
D

Diez B. Roggisch

gert said:
so one table of chunks

CREATE TABLE temp (
file_id VARCHAR(64),
chunK_id INTEGER,
chunk BLOB,
PRIMARY KEY(file_id,chunk_id)
);

SELECT chunk FROM temp WHERE file_id = 'file'
concatenating result in python
update blob
delete temp

How do I concatenate results that do not fit into memory ?

By writing them into one file? If files were to large for your memory,
all the substring-stuff wouldn't help either - the sqlite works in the
same memory as your program...

But how many gigabytes of uploads do you expect?

Diez
 
G

gert

gert schrieb:








By writing them into one file? If files were to large for your memory,
all the substring-stuff wouldn't help either - the sqlite works in the
same memory as your program...

But how many gigabytes of uploads do you expect?

250KB :)
Its just HTTP1.1 has everything for making ftp like file transfers
possible.
When I write it to a file then I am back at square one because I still
need to load it completely to get it into a blob.
So there is no way to concatenate BLOB's without loading it completely
into memory ?
 
D

Diez B. Roggisch


So why do you bother?
Its just HTTP1.1 has everything for making ftp like file transfers
possible.
When I write it to a file then I am back at square one because I still
need to load it completely to get it into a blob.

Well, the blob is nothing but datat in the file-system. If you are
*really* concerned about that, then don't use the db, but use the
filesystem, appending to the file until it's finished - and storing a
reference to it to the DB. We do that as well, because otherwise the db
will become unmanagable anyway (dumping, backups).
So there is no way to concatenate BLOB's without loading it completely
into memory ?

In theory, the DB might offer special stream-based methods for these
kinds of tasks, but the db-api lacks them. Some DB-adapters might offer
some non-standard-extensions, but I don't think sqlite does.

Diez
 
G

gert

So why do you bother?


Well, the blob is nothing but datat in the file-system. If you are
*really* concerned about that, then don't use the db, but use the
filesystem, appending to the file until it's finished - and storing a
reference to it to the DB. We do that as well, because otherwise the db
will become unmanagable anyway (dumping, backups).

I also hate debugging sql that contains blob data. So if using files
how do you clean the http post stuff?
(b+r'.*?Content-Type: application/octet-stream\r\n\r\n(.*?)\r
\n--'+b,file,DOTALL)
using as litlle memory as possible ?
I can not use PUT upload only because flash does not support PUT.
 
D

Diez B. Roggisch

gert said:
I also hate debugging sql that contains blob data. So if using files
how do you clean the http post stuff?
(b+r'.*?Content-Type: application/octet-stream\r\n\r\n(.*?)\r
\n--'+b,file,DOTALL)
using as litlle memory as possible ?
I can not use PUT upload only because flash does not support PUT.

PUT or POST has nothing todo with that.

It's the question if you need formencoded data or not. If not, you can
simply upload the data directly, no nead to cleanup anything.

For decoding formencoded data, you will need to write your own parser if
you insist on these very debatable memory-constraints of yours.

Diez
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top