recv_into(bytearray) complains about a "pinned buffer"

Discussion in 'Python' started by Andrew Dalke, Jan 31, 2010.

  1. Andrew Dalke

    Andrew Dalke Guest

    In Python 2.6 I can't socket.recv_into(a byte array instance). I get a
    TypeError which complains about a "pinned buffer". I have only an
    inkling of what that means. Since an array.array("b") works there, and
    since it works in Python 3.1.1, and since I thought the point of a
    bytearray was to make things like recv_into easier, I think this
    exception is a bug in Python 2.6.

    I want to double check before posting it to the tracker.

    Here's my reproducibles:

    Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
    [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import socket
    >>> sock = socket.socket()
    >>> sock.connect( ("python.org", 80) )
    >>> sock.send(b"GET / HTTP/1.0\r\n\r\n")

    18
    >>> buf = bytearray(b" " * 10)
    >>> sock.recv_into(buf)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    TypeError: recv_into() argument 1 must be pinned buffer, not bytearray
    >>>


    I expected a bytearray to work there. In fact, I thought the point of
    bytearray was to allow this to work.

    By comparison, an array of bytes does work:

    >>> import array
    >>> arr = array.array("b")
    >>> arr.extend(map(ord, "This is a test"))
    >>> len(arr)

    14
    >>> sock.recv_into(arr)

    14
    >>> arr

    array('b', [72, 84, 84, 80, 47, 49, 46, 49, 32, 51, 48, 50, 32, 70])
    >>> "".join(map(chr, arr))

    'HTTP/1.1 302 F'

    I don't even know what a "pinned buffer" means, and searching
    python.org isn't helpful.

    Using a bytearray in Python 3.1.1 *does* work:

    Python 3.1.1 (r311:74480, Jan 31 2010, 23:07:16)
    [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import socket
    >>> sock = socket.socket()
    >>> sock.connect( ("python.org", 80) )
    >>> sock.send(b"GET / HTTP/1.0\r\n\r\n")

    18
    >>> buf = bytearray(b" " * 10)
    >>> sock.recv_into(buf)

    10
    >>> buf

    bytearray(b'HTTP/1.1 3')
    >>>


    Is this a bug in Python 2.6 or a deliberate choice regarding
    implementation concerns I don't know about?

    If it's a bug, I'll add it to the tracker.

    Andrew Dalke
     
    Andrew Dalke, Jan 31, 2010
    #1
    1. Advertising

  2. Hello Andrew,

    > I don't even know what a "pinned buffer" means, and searching python.org
    > isn't helpful.
    >
    > Using a bytearray in Python 3.1.1 *does* work:
    > [...]


    Agreed, the error message is cryptic.
    The problem is that socket.recv_into() in 2.6 doesn't recognize the new
    buffer API which is needed to accept bytearray objects.
    (it does in 3.1, because the old buffer API doesn't exist anymore there)

    You could open an issue on the bug tracker for this.

    Thank you

    Antoine.
     
    Antoine Pitrou, Feb 1, 2010
    #2
    1. Advertising

  3. Andrew Dalke

    Andrew Dalke Guest

    On Feb 1, 1:04 am, Antoine Pitrou <> wrote:
    > The problem is that socket.recv_into() in 2.6 doesn't recognize the new
    > buffer API which is needed to accept bytearray objects.
    > (it does in 3.1, because the old buffer API doesn't exist anymore there)


    That's about what I thought it was, but I don't know if this was a
    deliberate choice or accidental.

    BTW, 2.7 (freshly built from version control) also has the same
    exception.

    > You could open an issue on the bug tracker for this.


    I've done that. It's http://bugs.python.org/issue7827 .

    Cheers!
    Andrew
     
    Andrew Dalke, Feb 1, 2010
    #3
  4. > In Python 2.6 I can't socket.recv_into(a byte array instance). I get a
    > TypeError which complains about a "pinned buffer". I have only an
    > inkling of what that means.


    A pinned buffer is one that cannot move in memory, even if another
    thread tries to behind your back. Typically, resizable containers
    are not inherently pinned, and "a user" (i.e. the API function) must
    explicitly pin it, which recv_into fails to do.

    > Is this a bug in Python 2.6 or a deliberate choice regarding
    > implementation concerns I don't know about?


    It's actually a bug also that you pass an array; doing so *should*
    give the very same error.

    It may be that the bugs surrounding buffers will never get fully
    resolved in the lifetime of Python 2.x, so I would probably just ignore
    recv_into.

    Regards,
    Martin
     
    Martin v. Loewis, Feb 1, 2010
    #4
  5. Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a écrit :
    >
    >> Is this a bug in Python 2.6 or a deliberate choice regarding
    >> implementation concerns I don't know about?

    >
    > It's actually a bug also that you pass an array; doing so *should* give
    > the very same error.


    Well, if you can give neither an array nor a bytearray to recv_into(),
    what *could* you give it?

    recv_into() should simply be fixed to use the new buffer API, as it does
    in 3.x.
     
    Antoine Pitrou, Feb 1, 2010
    #5
  6. Antoine Pitrou wrote:
    > Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a écrit :
    >>> Is this a bug in Python 2.6 or a deliberate choice regarding
    >>> implementation concerns I don't know about?

    >> It's actually a bug also that you pass an array; doing so *should* give
    >> the very same error.

    >
    > Well, if you can give neither an array nor a bytearray to recv_into(),
    > what *could* you give it?


    My recommendation would be to not use recv_into in 2.x, but only in 3.x.

    > recv_into() should simply be fixed to use the new buffer API, as it does
    > in 3.x.


    I don't think that's the full solution. The array module should also
    implement the new buffer API, so that it would also fail with the old
    recv_into.

    Regards,
    Martin
     
    Martin v. Loewis, Feb 1, 2010
    #6
  7. Andrew Dalke

    Andrew Dalke Guest

    On Feb 2, 12:12 am, Martin v. Loewis wrote:
    > My recommendation would be to not use recv_into in 2.x, but only in 3.x.


    > I don't think that's the full solution. The array module should also
    > implement the new buffer API, so that it would also fail with the old
    > recv_into.


    Okay. But recv_into was added in 2.5 and the test case in
    2.6's test_socket.py clearly allows an array there:


    def testRecvInto(self):
    buf = array.array('c', ' '*1024)
    nbytes = self.cli_conn.recv_into(buf)
    self.assertEqual(nbytes, len(MSG))
    msg = buf.tostring()[:len(MSG)]
    self.assertEqual(msg, MSG)

    Checking koders and Google Code search engines, I found one project
    which used recv_into, with the filename bmpreceiver.py . It
    uses a array.array("B", [0] * length) .

    Clearly it was added to work with an array, and it's
    being used with an array. Why shouldn't people use it
    with Python 2.x?

    Andrew
     
    Andrew Dalke, Feb 2, 2010
    #7
  8. Le Tue, 02 Feb 2010 00:12:34 +0100, Martin v. Loewis a écrit :
    >> recv_into() should simply be fixed to use the new buffer API, as it
    >> does in 3.x.

    >
    > I don't think that's the full solution. The array module should also
    > implement the new buffer API, so that it would also fail with the old
    > recv_into.


    There was a patch for this in http://bugs.python.org/issue6071 , but the
    bug was closed when hashlib was "fixed".
     
    Antoine Pitrou, Feb 2, 2010
    #8
  9. > Clearly it was added to work with an array, and it's
    > being used with an array. Why shouldn't people use it
    > with Python 2.x?


    Because it's not thread-safe; it may crash the interpreter if used
    incorrectly.

    Of course, if you don't share the array across threads, it can be safe
    to use.

    Regards,
    Martin
     
    Martin v. Loewis, Feb 2, 2010
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?c2lycGVsaWRvcg==?=

    debugger complains: auto-attach to process 'aspnet_wp.exe'

    =?Utf-8?B?c2lycGVsaWRvcg==?=, Aug 29, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    2,167
    =?Utf-8?B?c2lycGVsaWRvcg==?=
    Aug 29, 2005
  2. Old Wolf
    Replies:
    4
    Views:
    347
  3. Replies:
    1
    Views:
    2,544
    Jonathan N. Little
    Jun 22, 2006
  4. Elaine Jackson

    an ingrate newbie complains

    Elaine Jackson, Feb 4, 2004, in forum: Python
    Replies:
    10
    Views:
    519
    Peter Otten
    Feb 7, 2004
  5. mrstephengross
    Replies:
    0
    Views:
    363
    mrstephengross
    Mar 10, 2006
Loading...

Share This Page