piping out binaries properly

A

Andy Leszczynski

I have got following program:

import sys
import binascii
from string import *
sys.stdout.write(binascii.unhexlify("41410A4141"))


when I run under Unix I got:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0a 41 41

and under Windows/Cygwin following:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0d 0a 41 41
0000006


The question is how can I pipe out binary content properly and platform
independently?


A.
 
M

Mike Meyer

Andy Leszczynski said:
I have got following program:

import sys
import binascii
from string import *
sys.stdout.write(binascii.unhexlify("41410A4141"))


when I run under Unix I got:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0a 41 41

and under Windows/Cygwin following:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0d 0a 41 41
0000006

The question is how can I pipe out binary content properly and platform
independently?

It's not normal to write binary content to stdout - you normally write
it to a file. Open the file with open(name, 'wb') to write binaries.

There doesn't appear to be any way to retroactively change the mode on
a file. Which is probably a good thing.

<mike
 
A

Andy Leszczynski

Mike said:
It's not normal to write binary content to stdout - you normally write

Well, I grew up in the Unix world and it is normal over there.

I am still curious which layer adds that 0xd. Is it python, cygwin,
windows ...

Thx for reply, Andy
 
M

marduk

]
It's not normal to write binary content to stdout - you normally write
it to a file. Open the file with open(name, 'wb') to write binaries.

It is interesting that as a "Unix consultant" you should make that
claim. Especially since
<open file '<stdout>', mode 'w' at 0x2aaaaaac9198>
^^^^

Indeed there would be a lot of broken Unix systems out there if that
were not the case.

As for the OP, you may want to check how stdout is opened on your
system. In Windows there are two "write" modes for files, 'w', and 'wb'
where apparently 'w' is text mode and 'wb' is binary. You may want to
check what mode your stdout is in. I don't have a Windows box handy
right now to verify.
There doesn't appear to be any way to retroactively change the mode on
a file. Which is probably a good thing.

<mike--
Mike Meyer <[email protected]> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more
information.
 
M

Mike Meyer

Andy Leszczynski said:
Well, I grew up in the Unix world and it is normal over there.

I watched the Unix world grow up, and it ain't normal to me. I don't
think I've ever written a program that wrote binary data to standard
out, not in nearly 30 years of unix programming. I've written lots of
things whose standard out was designed specifically to be read by
another program, but never as binary data.
I am still curious which layer adds that 0xd. Is it python, cygwin,
windows ...

It's Python. You tell Python whether or not it needs to do
platform-dependent newline mangling when you open the file.

<mike
 
E

Erik Max Francis

Mike said:
I watched the Unix world grow up, and it ain't normal to me.

Since there's no distinction between a file opened in binary mode and in
text mode on Unix, there is no difference.
I don't
think I've ever written a program that wrote binary data to standard
out, not in nearly 30 years of unix programming. I've written lots of
things whose standard out was designed specifically to be read by
another program, but never as binary data.

Plenty of applications use that functionality and depend on it. See
cjpeg, djpeg, the pbmplus library, and so forth.
 
M

Mike Meyer

marduk said:
]
It's not normal to write binary content to stdout - you normally write
it to a file. Open the file with open(name, 'wb') to write binaries.

It is interesting that as a "Unix consultant" you should make that
claim. Especially since
<open file '<stdout>', mode 'w' at 0x2aaaaaac9198>
^^^^

Indeed there would be a lot of broken Unix systems out there if that
were not the case.

Ain't ambiguity wonderful? That we use the same word to donote a
collection of bytes on a disk and a software entity that one can write
bytes to allows for all kind of mistaken interpretations.

<mike
 
P

Paul Rubin

Erik Max Francis said:
Plenty of applications use that functionality and depend on it. See
cjpeg, djpeg, the pbmplus library, and so forth.

Also "tar cf files... | compress > tarball.Z" used to be a standard
idiom, now replaced by "tar cfz ..." which does about the same thing under
the covers.
 
F

Fredrik Lundh

Andy said:
when I run under Unix I got:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0a 41 41

and under Windows/Cygwin following:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0d 0a 41 41
0000006

The question is how can I pipe out binary content properly and platform
independently?

$ python -h
usage: python [option] ... [-c cmd | file | -] [arg] ...
Options and arguments (and corresponding environment variables):
....
-u : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
....

</F>
 
P

Paul Watson

Andy Leszczynski said:
I have got following program:

import sys
import binascii
from string import *
sys.stdout.write(binascii.unhexlify("41410A4141"))


when I run under Unix I got:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0a 41 41

and under Windows/Cygwin following:

$ python u.py > u.bin
$ od -t x1 u.bin
0000000 41 41 0d 0a 41 41
0000006


The question is how can I pipe out binary content properly and platform
independently?


A.

try:
import msvcrt, os
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
except:
pass
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top