recv_into(bytearray) complains about a "pinned buffer"

A

Andrew Dalke

In Python 2.6 I can't socket.recv_into(a byte array instance). I get a
TypeError which complains about a "pinned buffer". I have only an
inkling of what that means. Since an array.array("b") works there, and
since it works in Python 3.1.1, and since I thought the point of a
bytearray was to make things like recv_into easier, I think this
exception is a bug in Python 2.6.

I want to double check before posting it to the tracker.

Here's my reproducibles:

Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):

I expected a bytearray to work there. In fact, I thought the point of
bytearray was to allow this to work.

By comparison, an array of bytes does work:
import array
arr = array.array("b")
arr.extend(map(ord, "This is a test"))
len(arr) 14
sock.recv_into(arr) 14
arr array('b', [72, 84, 84, 80, 47, 49, 46, 49, 32, 51, 48, 50, 32, 70])
"".join(map(chr, arr))
'HTTP/1.1 302 F'

I don't even know what a "pinned buffer" means, and searching
python.org isn't helpful.

Using a bytearray in Python 3.1.1 *does* work:

Python 3.1.1 (r311:74480, Jan 31 2010, 23:07:16)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Is this a bug in Python 2.6 or a deliberate choice regarding
implementation concerns I don't know about?

If it's a bug, I'll add it to the tracker.

Andrew Dalke
(e-mail address removed)
 
A

Antoine Pitrou

Hello Andrew,
I don't even know what a "pinned buffer" means, and searching python.org
isn't helpful.

Using a bytearray in Python 3.1.1 *does* work:
[...]

Agreed, the error message is cryptic.
The problem is that socket.recv_into() in 2.6 doesn't recognize the new
buffer API which is needed to accept bytearray objects.
(it does in 3.1, because the old buffer API doesn't exist anymore there)

You could open an issue on the bug tracker for this.

Thank you

Antoine.
 
A

Andrew Dalke

The problem is that socket.recv_into() in 2.6 doesn't recognize the new
buffer API which is needed to accept bytearray objects.
(it does in 3.1, because the old buffer API doesn't exist anymore there)

That's about what I thought it was, but I don't know if this was a
deliberate choice or accidental.

BTW, 2.7 (freshly built from version control) also has the same
exception.
You could open an issue on the bug tracker for this.

I've done that. It's http://bugs.python.org/issue7827 .

Cheers!
Andrew
(e-mail address removed)
 
M

Martin v. Loewis

In Python 2.6 I can't socket.recv_into(a byte array instance). I get a
TypeError which complains about a "pinned buffer". I have only an
inkling of what that means.

A pinned buffer is one that cannot move in memory, even if another
thread tries to behind your back. Typically, resizable containers
are not inherently pinned, and "a user" (i.e. the API function) must
explicitly pin it, which recv_into fails to do.
Is this a bug in Python 2.6 or a deliberate choice regarding
implementation concerns I don't know about?

It's actually a bug also that you pass an array; doing so *should*
give the very same error.

It may be that the bugs surrounding buffers will never get fully
resolved in the lifetime of Python 2.x, so I would probably just ignore
recv_into.

Regards,
Martin
 
A

Antoine Pitrou

Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a écrit :
It's actually a bug also that you pass an array; doing so *should* give
the very same error.

Well, if you can give neither an array nor a bytearray to recv_into(),
what *could* you give it?

recv_into() should simply be fixed to use the new buffer API, as it does
in 3.x.
 
M

Martin v. Loewis

Antoine said:
Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a écrit :

Well, if you can give neither an array nor a bytearray to recv_into(),
what *could* you give it?

My recommendation would be to not use recv_into in 2.x, but only in 3.x.
recv_into() should simply be fixed to use the new buffer API, as it does
in 3.x.

I don't think that's the full solution. The array module should also
implement the new buffer API, so that it would also fail with the old
recv_into.

Regards,
Martin
 
A

Andrew Dalke

My recommendation would be to not use recv_into in 2.x, but only in 3.x.
I don't think that's the full solution. The array module should also
implement the new buffer API, so that it would also fail with the old
recv_into.

Okay. But recv_into was added in 2.5 and the test case in
2.6's test_socket.py clearly allows an array there:


def testRecvInto(self):
buf = array.array('c', ' '*1024)
nbytes = self.cli_conn.recv_into(buf)
self.assertEqual(nbytes, len(MSG))
msg = buf.tostring()[:len(MSG)]
self.assertEqual(msg, MSG)

Checking koders and Google Code search engines, I found one project
which used recv_into, with the filename bmpreceiver.py . It
uses a array.array("B", [0] * length) .

Clearly it was added to work with an array, and it's
being used with an array. Why shouldn't people use it
with Python 2.x?

Andrew
(e-mail address removed)
 
A

Antoine Pitrou

Le Tue, 02 Feb 2010 00:12:34 +0100, Martin v. Loewis a écrit :
I don't think that's the full solution. The array module should also
implement the new buffer API, so that it would also fail with the old
recv_into.

There was a patch for this in http://bugs.python.org/issue6071 , but the
bug was closed when hashlib was "fixed".
 
M

Martin v. Loewis

Clearly it was added to work with an array, and it's
being used with an array. Why shouldn't people use it
with Python 2.x?

Because it's not thread-safe; it may crash the interpreter if used
incorrectly.

Of course, if you don't share the array across threads, it can be safe
to use.

Regards,
Martin
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top