Image decompression, eruby

[Belorion]

I am working on a website with a MySQL backend. The site allows
users to upload files, such as jpg, gif, etc. I'm having issues
loading images from the database and getting them to display in the
webbrowser.

Since the raw image encoding contains characters MySQL doesn't like
when doing an INSERT, I encoded the image using Base64.encode64().

So, when I pulled the data back out, I do a Base64.decode64().

I also do a:

print cgi.header( "type=>image/jpg" ) so I have the correct header,
and then I dump the data by simply using <%= data %> as the only
information that is sent to the browser other than print cgi.header.

However, when I load the image, I get browser errors. In Firefox, it
knows it is image/jpg, but I get "The image (image path) cannot be
displayed, because it contains errors." And IE6 completely barfs,
asking to save the file because it doesn't know what type it is (and
saving the file as a jpg and then opening that up does not work)

What am I doing wrong? Does encoding/decoding in Base64 totally screw
this up? Any other recommended approaches? As I mentioned, Firefox is
aware of the fact it is image/jpg, but IE does not. In either case,
the raw data is not a properly encoded jpeg.

 

[Carlos]

I am working on a website with a MySQL backend. The site allows
users to upload files, such as jpg, gif, etc. I'm having issues
loading images from the database and getting them to display in the
webbrowser.

Since the raw image encoding contains characters MySQL doesn't like
when doing an INSERT, I encoded the image using Base64.encode64().

So, when I pulled the data back out, I do a Base64.decode64().

I also do a:

print cgi.header( "type=>image/jpg" ) so I have the correct header,

Shouldn't that be cgi.header("type"=>"image/jpg") ?

and then I dump the data by simply using <%= data %> as the only
information that is sent to the browser other than print cgi.header.

Just a guess: check for blanks/new lines around the <%= data %>.

...

What am I doing wrong? Does encoding/decoding in Base64 totally screw
this up? Any other recommended approaches? As I mentioned, Firefox is
aware of the fact it is image/jpg, but IE does not. In either case,
the raw data is not a properly encoded jpeg.

The other possibility is to add a header "Content-Transfer-Encoding: base64"
and send the base64 encoded photo directly.

Good luck.

 

[Kaspar Schiess]

(In response to by Belorion)

Since the raw image encoding contains characters MySQL doesn't like
when doing an INSERT, I encoded the image using Base64.encode64().

I recommend storing the binary data in BLOB's or whatever that data type is
called with mysql. Then you can insert data by doing
insert into mytable (blobcol) values (?)

The question mark is a placeholder for the data. You can bind your binary
data to it by calling DBI#execute as follows:
dbh.execute( stmt, data )

Note that there are some RDBMS' out there that disallow BLOB inserts. If
this is the case, you need to first insert all non-blob columns and then go
back and UPDATE your table's blob columns (selecting the id's you have just
inserted).

This data then comes back from a SELECT as normal binary data which is
exactly (and without detour) your image. It can be sent to the browser by
the following code (untested):

# image contained in +data+
require 'cgi'
c = CGI.new
c.print c.header( 'type' => 'image/gif', 'length' => data.length )
STDOUT.binmode if PLATFORM =~ /32/
c.print data

The binmode part is UGLY, but there seems to be no other way to do this.

happy hacking !
kaspar

nb: This is a part of the DBI manual:

execute( stmt, *bindvars ) {|statement_handle| aBlock}

Immediately executes the SQL statement stmt after binding the values
in bindvars to the placeholders in the statement.

kaspar

hand manufactured code - www.tua.ch/ruby

 

[Belorion]

Shouldn't that be cgi.header("type"=>"image/jpg") ?

Yes, it should, and it is in my code

Just a guess: check for blanks/new lines around the <%= data %>.

Nope, can't find any blank lines....

The other possibility is to add a header "Content-Transfer-Encoding: base64"
and send the base64 encoded photo directly.

Good idea. Gave it a shot, but couldn't get it to work. Same errors.

I recommend storing the binary data in BLOB's or whatever that data type is
called with mysql. Then you can insert data by doing
insert into mytable (blobcol) values (?)

The question mark is a placeholder for the data. You can bind your binary
data to it by calling DBI#execute as follows:
dbh.execute( stmt, data )

I hadn't seen that before. I gave it a shot. Seems to work fine for
getting the browser to display an image. I take that back, dbh.do(
stmt, data ) works fine, I get an error with dbh.execute. However,
the image is coming back with glaring errors. After further
investigation it appears that the raw data in the MySQL table has
issues, though I can't tell if it's from the way the data is inserted,
or if is corrupted from the actual upload.

So my first question regarding displaying the image via eruby in my
browser has been answered (thanks!), but maybe someone has an idea as
to why my data is being corrupted (the image comes back with a ton of
artifacts, especially with strange bars of color).

The way I am uploading my file is derived from a script I found at
http://www.zytrax.com/tech/lang/ruby/#upload.

Specifically, my HTML for uploading looks like:

<form name='fileupload' enctype="multipart/form-data"
action='attach.rhtml' method='post'>
<input type=hidden name=index value=<%= index %>>
<input type='file' name="myfile" size="35">
<input type='submit' value="Upload">

And my upload specific ruby in attach.rhtml looks like:

fromfile = cgi.params['myfile'].first
fromfile.binmode # Tried both with and without this line
dbh.do( "INSERT INTO attachments VALUES( ? )", [ index,
fromfile.original_filename, fromfile.read] )

So, somewhere in that process the data gets corrupted, and the image
comes out half garbled. Any ideas? Plain text files, which are
handled using the exact same script, come out just fine.

 

[Carlos]

Specifically, my HTML for uploading looks like:

<form name='fileupload' enctype="multipart/form-data"
action='attach.rhtml' method='post'>
<input type=hidden name=index value=<%= index %>>
<input type='file' name="myfile" size="35">
<input type='submit' value="Upload">

And my upload specific ruby in attach.rhtml looks like:

fromfile = cgi.params['myfile'].first
fromfile.binmode # Tried both with and without this line
dbh.do( "INSERT INTO attachments VALUES( ? )", [ index,
fromfile.original_filename, fromfile.read] )

So, somewhere in that process the data gets corrupted, and the image
comes out half garbled. Any ideas? Plain text files, which are
handled using the exact same script, come out just fine.

I don't have any idea about dbi or mysql...

Attacking the problem by the web side , maybe the upload got interrupted
and you are getting a truncated file.

If you add a parameter like

<input type=hidden name=MARK value=MARK>

at the end of your form, you can check it to know if all the previous fields
arrived whole. At least, if that's not the problem, you'll have a variable
less to ponder (and it is a good general tip, too .

 

[Belorion]

I don't have any idea about dbi or mysql...

Attacking the problem by the web side , maybe the upload got interrupted
and you are getting a truncated file.

If you add a parameter like

<input type=hidden name=MARK value=MARK>

at the end of your form, you can check it to know if all the previous fields
arrived whole. At least, if that's not the problem, you'll have a variable
less to ponder (and it is a good general tip, too .

I just tried dumping fromfile.read to a file and then viewing the
file. It turns out that the data at that point in time is *fine*.
So I know the data is arriving whole. So, it has something to do
specifically with inserting the binary data into MySQL, which I
suppose is beyond the normal scope of this list (though I won't
discourage any more suggestions!) I am going to do some digging on
binary data in MySQL, but if anyone has any more suggestions I'd love
to hear them.

 

[Roland Schmitt]

Hi,

I just tried dumping fromfile.read to a file and then viewing the
file. It turns out that the data at that point in time is *fine*.
So I know the data is arriving whole. So, it has something to do
specifically with inserting the binary data into MySQL, which I
suppose is beyond the normal scope of this list (though I won't
discourage any more suggestions!) I am going to do some digging on
binary data in MySQL, but if anyone has any more suggestions I'd love
to hear them.

you can try to convert the binary data to base64 and then store the
base64-data in the database:

require "base64"
...
b64_data = Base64.encode64(binary_data)
# store to db
...
# read from db
binary_data = Base64.decode64(b64_data)
...

Obviously you must convert back after reading the data from the database
;-)

For me base64-formatted data is easier to handle when looking for
differences between two files.

Regards,
Roland

 

[Belorion]

Hi,
you can try to convert the binary data to base64 and then store the
base64-data in the database:

require "base64"
...
b64_data = Base64.encode64(binary_data)
# store to db
...
# read from db
binary_data = Base64.decode64(b64_data)
...

Obviously you must convert back after reading the data from the database
;-)

For me base64-formatted data is easier to handle when looking for
differences between two files.

Regards,
Roland

I've tried that once with no success, but maybe I'll give it a shot again.

Although, it occurrs to me... my "test" which told me my MySQL data
was bad may have been incorrect. What, precisely, is the best/correct
way to write out binary data? Basically, what I was doing was:

sth = dbh.prepare( "SELECT content FROM attachment WHERE index=41" )
sth.execute
data = sth.fetch[0]

f = File.new( "test.jpg", "w+" )
f << data
f.close

Is that correct way to write out binary data?

 

[Belorion]

I feel dumb. the Base64 was one of the first approaches I used. I
just tried it again and it worked. I'm not sure what I am doing right
this time that I was doing wrong the last... but it works now! Thanks
everyone for the help!

And, for anyone searching Ruby talk for a binary file upload solution
using MySQL, here ya go:

HTML:

<form name='fileupload' enctype="multipart/form-data"
action='attach.rhtml' method='post'>
<input type='file' name="myfile" size="35">
<input type='submit' value="Upload">

In attach.rhtml, the relavent code for insertion into MySQL is:

require 'cgi'
require 'base64'
require 'dbi'
require 'stringio' # not sure if you need this

myfile = cgi.params['myfile'].first
dbh = DBI.connect( "DBI:Mysql:userdb", "user", "psswd" )
dbh.do( "INSERT INTO table VALUES( ? )", Base64.encode64(myfile.read) ] )
dbh.disconnect

And then, to view the file using eruby:

<%
require 'cgi'
require 'dbi'
require 'base64'

cgi = CGI.new

dbh = DBI.connect( "DBI:Mysql:userdb", "user", "psswd" )
sth = dbh.prepare( " (statement to select out your data) "
sth.execute
data = sth.fetch[0] # assuming your binary data is the only thing returned

print cgi.header("type"=>"image/jpg", "length"=>data.length)
cgi.print Base64.decode64(data)
%>

 

[Osuka Adartse]

Shouldn't that be cgi.header("type"=>"image/jpg") ?

Yes, it should, and it is in my code

Just a guess: check for blanks/new lines around the <%= data %>.

Nope, can't find any blank lines....

The other possibility is to add a header "Content-Transfer-Encoding: base64"
and send the base64 encoded photo directly.

Good idea. Gave it a shot, but couldn't get it to work. Same errors.

I recommend storing the binary data in BLOB's or whatever that data type is
called with mysql. Then you can insert data by doing
insert into mytable (blobcol) values (?)

The question mark is a placeholder for the data. You can bind your binary
data to it by calling DBI#execute as follows:
dbh.execute( stmt, data )

I hadn't seen that before. I gave it a shot. Seems to work fine for
getting the browser to display an image. I take that back, dbh.do(
stmt, data ) works fine, I get an error with dbh.execute. However,
the image is coming back with glaring errors. After further
investigation it appears that the raw data in the MySQL table has
issues, though I can't tell if it's from the way the data is inserted,
or if is corrupted from the actual upload.

So my first question regarding displaying the image via eruby in my
browser has been answered (thanks!), but maybe someone has an idea as
to why my data is being corrupted (the image comes back with a ton of
artifacts, especially with strange bars of color).

The way I am uploading my file is derived from a script I found at
http://www.zytrax.com/tech/lang/ruby/#upload.

Specifically, my HTML for uploading looks like:

<form name='fileupload' enctype="multipart/form-data"
action='attach.rhtml' method='post'>
<input type=hidden name=index value=<%= index %>>
<input type='file' name="myfile" size="35">
<input type='submit' value="Upload">

And my upload specific ruby in attach.rhtml looks like:

fromfile = cgi.params['myfile'].first
fromfile.binmode # Tried both with and without this line
dbh.do( "INSERT INTO attachments VALUES( ? )", [ index,
fromfile.original_filename, fromfile.read] )

So, somewhere in that process the data gets corrupted, and the image
comes out half garbled. Any ideas? Plain text files, which are
handled using the exact same script, come out just fine.

let me guess! The image starts OK but then horizontal lines start to
appear with degenerating colors? if so the data in DB is probably OK as
long as you use MySql::quote(img_data) at insert time.

Ok try this, select the image data and write it to disk,just an idea
since I had once a similar problem.

[email protected]("select pik from test where idpiks=1;")
res.each{|r| img=r[0]} #well r[0] it's the actual image.
File.open("e:/temp/test.jpg","wb").write(img)

now compare it, if it's OK then no upload problems or in db but at
"browser display". If not make sure the #quote method is being used.

how I displayed the image
imager.rb
...
[email protected]("select pik from test where idpiks=#{cgi.params['CID']};")
res.each{|r| img=r[0]}
puts "Content-Type: image/jpeg"
puts "Content-Length: #{img.size}"
puts "Content-Title:MyTest.jpg"
puts
$stdout.binmode << img #this is the culprit in windows, notice 'wb'
mode in File#Write so #binmode, it drove me nuts back then, the answer
was obvious...
}

hope it helps
Adartse

 

[Belorion]

let me guess! The image starts OK but then horizontal lines start to

appear with degenerating colors? if so the data in DB is probably OK as
long as you use MySql::quote(img_data) at insert time.

That was precisely the problem!

Ok try this, select the image data and write it to disk,just an idea
since I had once a similar problem.

[email protected]("select pik from test where idpiks=1;")
res.each{|r| img=r[0]} #well r[0] it's the actual image.
File.open("e:/temp/test.jpg","wb").write(img)

now compare it, if it's OK then no upload problems or in db but at
"browser display". If not make sure the #quote method is being used.

how I displayed the image
imager.rb
...
[email protected]("select pik from test where idpiks=#{cgi.params['CID']};")
res.each{|r| img=r[0]}
puts "Content-Type: image/jpeg"
puts "Content-Length: #{img.size}"
puts "Content-Title:MyTest.jpg"
puts
$stdout.binmode << img #this is the culprit in windows, notice 'wb'
mode in File#Write so #binmode, it drove me nuts back then, the answer
was obvious...
}

hope it helps
Adartse

Thanks for that solution as well. I currently have it working using
Base64 encoding, so I may not switch it over, but thanks anyway.

And thanks to all for their help.

 

[Kaspar Schiess]

(In response to by
Belorion)

So my first question regarding displaying the image via eruby in my
browser has been answered (thanks!), but maybe someone has an idea as
to why my data is being corrupted (the image comes back with a ton of
artifacts, especially with strange bars of color).

I remember having the same effect and would like to follow up on your
'solution'-post by saying that it is definitely possible to store images
1:1 in a suited mysql datatype.

I remember having to tweak different things:
- Upload, for it was only sending filename first. This is common,
and related to the form not being 'multipart' or so.

- Storage, related to not using ? placeholders at first (like you)
and then related to mysql dropping the connection after a number of bytes
transferred which need to be set to a more reasonable (=REAL BIG) value.
This is probably happening to you. This issue surfaces both ways, sending
data to mysql (solution here is placeholders) and receiving, so either
the INSERT inserts not all data, or the SELECT does not fetch it
entirely.
Note that I don't really recommend the use of #quote on such data, since
it has been known to fail and violate the DBI abstraction. Placeholders
are really the way to go.

- Output, the usual binary/non-binary output CRUFT that windows
forces you to occupy your mind with. I think the solution to that problem
has been posted several times.

The b) part of the Storage Problem can be solved by fiddling with
LongReadLen, LongTruncOk options on the DBI handle. More details on this
in the Perl (yes I said the word) manual page on DBI. (Perl documentation
rocks, perhaps because the language is not as easy to read back.)

But of course once you got it working, I don't expect you to change your
solution again. This is more for posteriority.

my best regards,
kaspar

hand manufactured code - www.tua.ch/ruby

 

[Belorion]

- Output, the usual binary/non-binary output CRUFT that windows

forces you to occupy your mind with. I think the solution to that problem
has been posted several times.

# image contained in +data+
require 'cgi'
c = CGI.new
c.print c.header( 'type' => 'image/gif', 'length' => data.length )
STDOUT.binmode if PLATFORM =~ /32/
c.print data>

Okay ... so everything was working fine and dandy in my Firefox
browser on windows. However, I still cannot convince IE6 that the
file is of type jpeg. When I click on the link for the image, IE6
doesn't know the file type (it asks to save the file of unknown type).
I've tried this with and without STDOUT.binmode. Firefox works in
either case, properly identifying it as content type: image/jpg... but
windows does not.

The really funny part is if I screw up and put a typo in my cgi
script, I get a server error (as expected), but when I hit refresh
after fixing the typo the image pops up just fine. But, that is the
*only* time I can get IE6 to display the image. Any suggestions?

 

[Kaspar Schiess]

(In response to by Belorion)

The really funny part is if I screw up and put a typo in my cgi
script, I get a server error (as expected), but when I hit refresh
after fixing the typo the image pops up just fine. But, that is the
*only* time I can get IE6 to display the image. Any suggestions?

Yes, although this is not a 'solution'-type suggestion but rather a 'how to
progress from here'-type suggestion:

Get yourself a decent debugging proxy (LiveHTTPHeaders for Firefox,
httpdebug.pl in Perl, any other) that allows you to inspect headers sent
from and to IE when accessing REAL images. Then compare with what you send
and make up for the lack of headers.

Sometimes it has helped to create cgi's that are called cgi.jpg, cgi.png,
etc.. to fool browsers with a simpler logic into believing the image. This
can be hard to get to work with your webserver though.

Tell me if this works out, I may have the time to look at this in more
detail.

kaspar

hand manufactured code - www.tua.ch/ruby

 

[Belorion]

Thanks for the suggestion. My problem, it turns out, was much
simpler. I was using a content type of 'image/jpg' when really IE6
wants to see 'image/jpeg'. Which I suppose makes sense, since jpeg is
the correct mime notation... thanks for the help though!