piping out binaries properly

Discussion in 'Python' started by Andy Leszczynski, Oct 12, 2005.

  1. I have got following program:

    import sys
    import binascii
    from string import *
    sys.stdout.write(binascii.unhexlify("41410A4141"))


    when I run under Unix I got:

    $ python u.py > u.bin
    $ od -t x1 u.bin
    0000000 41 41 0a 41 41

    and under Windows/Cygwin following:

    $ python u.py > u.bin
    $ od -t x1 u.bin
    0000000 41 41 0d 0a 41 41
    0000006


    The question is how can I pipe out binary content properly and platform
    independently?


    A.
     
    Andy Leszczynski, Oct 12, 2005
    #1
    1. Advertising

  2. Andy Leszczynski

    Mike Meyer Guest

    Andy Leszczynski <leszczynscyATnospam.yahoo.com.nospam> writes:
    > I have got following program:
    >
    > import sys
    > import binascii
    > from string import *
    > sys.stdout.write(binascii.unhexlify("41410A4141"))
    >
    >
    > when I run under Unix I got:
    >
    > $ python u.py > u.bin
    > $ od -t x1 u.bin
    > 0000000 41 41 0a 41 41
    >
    > and under Windows/Cygwin following:
    >
    > $ python u.py > u.bin
    > $ od -t x1 u.bin
    > 0000000 41 41 0d 0a 41 41
    > 0000006
    >
    > The question is how can I pipe out binary content properly and platform
    > independently?


    It's not normal to write binary content to stdout - you normally write
    it to a file. Open the file with open(name, 'wb') to write binaries.

    There doesn't appear to be any way to retroactively change the mode on
    a file. Which is probably a good thing.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Oct 12, 2005
    #2
    1. Advertising

  3. Mike Meyer wrote:
    > It's not normal to write binary content to stdout - you normally write


    Well, I grew up in the Unix world and it is normal over there.

    I am still curious which layer adds that 0xd. Is it python, cygwin,
    windows ...

    Thx for reply, Andy
     
    Andy Leszczynski, Oct 12, 2005
    #3
  4. Andy Leszczynski

    marduk Guest

    On Wed, 2005-10-12 at 00:16 -0400, Mike Meyer wrote:
    [...]
    > It's not normal to write binary content to stdout - you normally write
    > it to a file. Open the file with open(name, 'wb') to write binaries.
    >


    It is interesting that as a "Unix consultant" you should make that
    claim. Especially since

    >>> sys.stdout

    <open file '<stdout>', mode 'w' at 0x2aaaaaac9198>
    ^^^^

    Indeed there would be a lot of broken Unix systems out there if that
    were not the case.

    As for the OP, you may want to check how stdout is opened on your
    system. In Windows there are two "write" modes for files, 'w', and 'wb'
    where apparently 'w' is text mode and 'wb' is binary. You may want to
    check what mode your stdout is in. I don't have a Windows box handy
    right now to verify.

    > There doesn't appear to be any way to retroactively change the mode on
    > a file. Which is probably a good thing.
    >
    > <mike--
    > Mike Meyer <>

    http://www.mired.org/home/mwm/
    > Independent WWW/Perforce/FreeBSD/Unix consultant, email for more

    information.
     
    marduk, Oct 12, 2005
    #4
  5. Andy Leszczynski

    Mike Meyer Guest

    Andy Leszczynski <leszczynscyATnospam.yahoo.com.nospam> writes:

    > Mike Meyer wrote:
    >> It's not normal to write binary content to stdout - you normally write

    >
    > Well, I grew up in the Unix world and it is normal over there.


    I watched the Unix world grow up, and it ain't normal to me. I don't
    think I've ever written a program that wrote binary data to standard
    out, not in nearly 30 years of unix programming. I've written lots of
    things whose standard out was designed specifically to be read by
    another program, but never as binary data.

    > I am still curious which layer adds that 0xd. Is it python, cygwin,
    > windows ...


    It's Python. You tell Python whether or not it needs to do
    platform-dependent newline mangling when you open the file.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Oct 12, 2005
    #5
  6. Mike Meyer wrote:

    > I watched the Unix world grow up, and it ain't normal to me.


    Since there's no distinction between a file opened in binary mode and in
    text mode on Unix, there is no difference.

    > I don't
    > think I've ever written a program that wrote binary data to standard
    > out, not in nearly 30 years of unix programming. I've written lots of
    > things whose standard out was designed specifically to be read by
    > another program, but never as binary data.


    Plenty of applications use that functionality and depend on it. See
    cjpeg, djpeg, the pbmplus library, and so forth.

    --
    Erik Max Francis && && http://www.alcyone.com/max/
    San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
    Every human being is a problem in search of a solution.
    -- Ashley Montagu
     
    Erik Max Francis, Oct 12, 2005
    #6
  7. Andy Leszczynski

    Mike Meyer Guest

    marduk <> writes:

    > On Wed, 2005-10-12 at 00:16 -0400, Mike Meyer wrote:
    > [...]
    >> It's not normal to write binary content to stdout - you normally write
    >> it to a file. Open the file with open(name, 'wb') to write binaries.
    >>

    >
    > It is interesting that as a "Unix consultant" you should make that
    > claim. Especially since
    >
    >>>> sys.stdout

    > <open file '<stdout>', mode 'w' at 0x2aaaaaac9198>
    > ^^^^
    >
    > Indeed there would be a lot of broken Unix systems out there if that
    > were not the case.


    Ain't ambiguity wonderful? That we use the same word to donote a
    collection of bytes on a disk and a software entity that one can write
    bytes to allows for all kind of mistaken interpretations.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Oct 12, 2005
    #7
  8. Andy Leszczynski

    Paul Rubin Guest

    Erik Max Francis <> writes:
    > > I've written lots of things whose standard out was designed
    > > specifically to be read by another program, but never as binary data.

    >
    > Plenty of applications use that functionality and depend on it. See
    > cjpeg, djpeg, the pbmplus library, and so forth.


    Also "tar cf files... | compress > tarball.Z" used to be a standard
    idiom, now replaced by "tar cfz ..." which does about the same thing under
    the covers.
     
    Paul Rubin, Oct 12, 2005
    #8
  9. Andy Leszczynski wrote:

    > when I run under Unix I got:
    >
    > $ python u.py > u.bin
    > $ od -t x1 u.bin
    > 0000000 41 41 0a 41 41
    >
    > and under Windows/Cygwin following:
    >
    > $ python u.py > u.bin
    > $ od -t x1 u.bin
    > 0000000 41 41 0d 0a 41 41
    > 0000006
    >
    > The question is how can I pipe out binary content properly and platform
    > independently?


    $ python -h
    usage: python [option] ... [-c cmd | file | -] [arg] ...
    Options and arguments (and corresponding environment variables):
    ....
    -u : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
    ....

    </F>
     
    Fredrik Lundh, Oct 12, 2005
    #9
  10. Andy Leszczynski

    Paul Watson Guest

    "Andy Leszczynski" <leszczynscyATnospam.yahoo.com.nospam> wrote in message
    news:eek:...
    >I have got following program:
    >
    > import sys
    > import binascii
    > from string import *
    > sys.stdout.write(binascii.unhexlify("41410A4141"))
    >
    >
    > when I run under Unix I got:
    >
    > $ python u.py > u.bin
    > $ od -t x1 u.bin
    > 0000000 41 41 0a 41 41
    >
    > and under Windows/Cygwin following:
    >
    > $ python u.py > u.bin
    > $ od -t x1 u.bin
    > 0000000 41 41 0d 0a 41 41
    > 0000006
    >
    >
    > The question is how can I pipe out binary content properly and platform
    > independently?
    >
    >
    > A.


    try:
    import msvcrt, os
    msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
    except:
    pass
     
    Paul Watson, Oct 12, 2005
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. bernd wegener

    Piping ping into perl-prog

    bernd wegener, Sep 15, 2004, in forum: Perl
    Replies:
    3
    Views:
    673
    Jim Gibson
    Sep 22, 2004
  2. Jeff Kish

    piping data

    Jeff Kish, Oct 15, 2004, in forum: Java
    Replies:
    1
    Views:
    538
    Thomas Fritsch
    Oct 15, 2004
  3. christopher diggins

    Piping between Functions

    christopher diggins, Jan 8, 2005, in forum: C++
    Replies:
    4
    Views:
    550
    christopher diggins
    Jan 10, 2005
  4. KevinSimonson
    Replies:
    17
    Views:
    689
    KevinSimonson
    Sep 28, 2009
  5. Caio Chassot

    piping input when shelling out

    Caio Chassot, Sep 20, 2006, in forum: Ruby
    Replies:
    12
    Views:
    207
    Rick DeNatale
    Sep 21, 2006
Loading...

Share This Page