Binary data handling ?

Discussion in 'Python' started by Bill Loren, Aug 28, 2003.

  1. Bill Loren

    Bill Loren Guest

    Hello ppl,

    I'm having difficulties to accomplish some simple chores with binary data.
    I'm having a string (?) which I received via an HTTP transactions which is a
    binary file.
    Problem is the webserver I'm communicating with injected a \x0D before every
    \x0A,
    and I need to remove those 0x0D characters from my buffer before saving it
    to disk.

    any ideas ?

    I tried the following without any success:
    string.replace("%c%c" % (13,10), "%c" % (10))
    string.replace("\x0d\x0a", "\0x0a")

    thx
    ~B
     
    Bill Loren, Aug 28, 2003
    #1
    1. Advertising

  2. Bill Loren

    Peter Hansen Guest

    Bill Loren wrote:
    >
    > Hello ppl,
    >
    > I'm having difficulties to accomplish some simple chores with binary data.
    > I'm having a string (?) which I received via an HTTP transactions which is a
    > binary file.
    > Problem is the webserver I'm communicating with injected a \x0D before every
    > \x0A,
    > and I need to remove those 0x0D characters from my buffer before saving it
    > to disk.


    This sounds wrong. I don't think a properly configured web server should
    be transmitting unencoded binary files with newline conversion.

    > any ideas ?
    >
    > I tried the following without any success:
    > string.replace("%c%c" % (13,10), "%c" % (10))
    > string.replace("\x0d\x0a", "\0x0a")


    Strings are immutable. Are you expecting the above to change the string
    (which doesn't happen) or to return a new string with the changes made?
    Assign the result of the replace() call to a new variable and it should
    work. (Except in the latter example you should have \x0a, not \0x0a.)

    -Peter
     
    Peter Hansen, Aug 28, 2003
    #2
    1. Advertising

  3. Bill Loren

    Peter Abel Guest

    "Bill Loren" <> wrote in message news:<>...
    > Hello ppl,
    >
    > I'm having difficulties to accomplish some simple chores with binary data.
    > I'm having a string (?) which I received via an HTTP transactions which is a
    > binary file.
    > Problem is the webserver I'm communicating with injected a \x0D before every
    > \x0A,
    > and I need to remove those 0x0D characters from my buffer before saving it
    > to disk.
    >
    > any ideas ?
    >
    > I tried the following without any success:
    > string.replace("%c%c" % (13,10), "%c" % (10))
    > string.replace("\x0d\x0a", "\0x0a")
    >
    > thx
    > ~B


    >>> # A short function to generate a string with 0x0D's
    >>> def make_bin():

    .... bytes=[]
    .... for i in range(20):
    .... bytes.append(chr(i))
    .... # every 5th char is a 0x0D
    .... if not i%5:
    .... bytes.append(chr(0x0D))
    .... return ''.join(bytes)
    ....
    >>> binary=make_bin()
    >>> binary

    '\x00\r\x01\x02\x03\x04\x05\r\x06\x07\x08\t\n\r\x0b\x0c\r\x0e\x0f\r\x10\x11\x12\x13'
    >>> # split the string at the 0x0D's and join it again
    >>> binary_without_x0D=''.join(binary.split(chr(0x0D)))
    >>> #et voila
    >>> binary_without_x0D

    '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\x0e\x0f\x10\x11\x12\x13'
    >>>


    Regards
    Peter
     
    Peter Abel, Aug 28, 2003
    #3
  4. Bill Loren

    Peter Hansen Guest

    Bill Loren wrote:
    >
    > about the code problem, I did fix that bug you mentioned, but it still
    > doesn't work.
    > the code is:
    > data = data.replace(...the two options I mentioned before...)
    > but alas... no replacement...


    Since the following clearly works, you must be confused about what
    is in the string called "data" prior to the replacement:

    C:\>python22
    Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> s = 'test 1\r\ntest 2\rtest 3\ntest4\r\n\r\ntest5'
    >>> s

    'test 1\r\ntest 2\rtest 3\ntest4\r\n\r\ntest5'
    >>> t = s.replace('\r\n', '\n')
    >>> u = s.replace('\x0d\x0a', '\x0a')
    >>> v = s.replace('%c%c' % (13, 10), '%c' % (10))
    >>> t

    'test 1\ntest 2\rtest 3\ntest4\n\ntest5'
    >>> u

    'test 1\ntest 2\rtest 3\ntest4\n\ntest5'
    >>> v

    'test 1\ntest 2\rtest 3\ntest4\n\ntest5'

    (Please post to the newsgroup/mailing list instead of mailing directly,
    so that others can benefit from or participate in the discussion.)

    -Peter
     
    Peter Hansen, Aug 28, 2003
    #4
  5. Bill Loren

    Tim Roberts Guest

    "Bill Loren" <> wrote:
    >
    >I'm having difficulties to accomplish some simple chores with binary data.
    >I'm having a string (?) which I received via an HTTP transactions which is a
    >binary file.
    >Problem is the webserver I'm communicating with injected a \x0D before every
    >\x0A,
    >and I need to remove those 0x0D characters from my buffer before saving it
    >to disk.
    >
    >any ideas ?


    I'll bet you real money that the problem is not in the web server. I'd
    wager that the string is correct when you receive it, but that you are
    writing it to file like this:
    file('out.txt','w').write(download)

    On a Windows system, that'll turn all the LFs into CR-LFs. Use this
    instead:
    file('out.txt','wb').write(download)
    --
    - Tim Roberts,
    Providenza & Boekelheide, Inc.
     
    Tim Roberts, Aug 30, 2003
    #5
  6. Bill Loren

    Bill Loren Guest

    Indeed !

    thanks !!!

    if I use the 'wb' will it work on a unix system, too ?

    ~B
    ----- Original Message -----
    From: "Tim Roberts" <>
    Newsgroups: comp.lang.python
    To: <>
    Sent: Saturday, August 30, 2003 7:11 AM
    Subject: Re: Binary data handling ?


    > "Bill Loren" <> wrote:
    > >
    > >I'm having difficulties to accomplish some simple chores with binary

    data.
    > >I'm having a string (?) which I received via an HTTP transactions which

    is a
    > >binary file.
    > >Problem is the webserver I'm communicating with injected a \x0D before

    every
    > >\x0A,
    > >and I need to remove those 0x0D characters from my buffer before saving

    it
    > >to disk.
    > >
    > >any ideas ?

    >
    > I'll bet you real money that the problem is not in the web server. I'd
    > wager that the string is correct when you receive it, but that you are
    > writing it to file like this:
    > file('out.txt','w').write(download)
    >
    > On a Windows system, that'll turn all the LFs into CR-LFs. Use this
    > instead:
    > file('out.txt','wb').write(download)
    > --
    > - Tim Roberts,
    > Providenza & Boekelheide, Inc.
    > --
    > http://mail.python.org/mailman/listinfo/python-list
     
    Bill Loren, Sep 1, 2003
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fangs
    Replies:
    3
    Views:
    9,802
    darshana
    Oct 26, 2008
  2. Marc Schellens
    Replies:
    8
    Views:
    3,024
    John Harrison
    Jul 15, 2003
  3. Bruce Lee

    Handling binary data in C - questions

    Bruce Lee, Jan 18, 2005, in forum: C Programming
    Replies:
    6
    Views:
    506
    Richard Bos
    Jan 19, 2005
  4. Replies:
    12
    Views:
    582
    Richard Heathfield
    Apr 8, 2007
  5. SG
    Replies:
    8
    Views:
    642
    James Kanze
    Dec 18, 2008
Loading...

Share This Page