How to convert arbitrary objects directly to base64 without initial string conversion?

[Russell Warren]

I've got a case where I want to convert binary blocks of data (various
ctypes objects) to base64 strings.

The conversion calls in the base64 module expect strings as input, so
right now I'm converting the binary blocks to strings first, then
converting the resulting string to base64. This seems highly
inefficient and I'd like to just go straight from binary to a base64
string.

Here is the conversion we're using from object to string...

import ctypes
def ObjAsString(obj):
sz = ctypes.sizeof(obj)
charArray = ctypes.c_char * sz
buf = charArray.from_address(ctypes.addressof(obj))
return buf.raw[:sz]

The returned string can then be sent to base64 for conversion (although
we're actually using xmlrpc.Binary), but there is obviously some waste
in here.

import base64
b64 = base64.b64encode(ObjAsString(foo))

Is there a canned/pre-existing way to convert a block of memory to a
base64 string more efficiently? I'd like to avoid writing my own
base64 conversion routine if possible. Anyone have any good ideas?
Even a mroe efficient/less clunky way of conevrting an arbitrary object
to a string would be appreciated.

Thanks,
Russ

 

[Thomas Heller]

I've got a case where I want to convert binary blocks of data (various
ctypes objects) to base64 strings.

The conversion calls in the base64 module expect strings as input, so
right now I'm converting the binary blocks to strings first, then
converting the resulting string to base64. This seems highly
inefficient and I'd like to just go straight from binary to a base64
string. [...]
Is there a canned/pre-existing way to convert a block of memory to a
base64 string more efficiently? I'd like to avoid writing my own
base64 conversion routine if possible. Anyone have any good ideas?
Even a mroe efficient/less clunky way of conevrting an arbitrary object
to a string would be appreciated.

Many functions that operate on strings also accept buffer objects as parameters,
this seems also be the case for the base64.encodestring function. ctypes objects
support the buffer interface.

So, base64.b64encode(buffer(ctypes_instance)) should work efficiently.

Thomas

 

[Russell Warren]

Many functions that operate on strings also accept buffer objects as parameters,

this seems also be the case for the base64.encodestring function. ctypes objects
support the buffer interface.

So, base64.b64encode(buffer(ctypes_instance)) should work efficiently.

Thanks! I have never used (or even heard of) the buffer objects. I'll
check it out.

Russ

 

[Russell Warren]

After some digging around it appears there is not a tonne of
documentation on buffer objects, although they are clearly core and
ancient... been sifting through some hits circa 1999, long before my
python introduction.

What I can find says that buffer is deprecated (Python in a Nutshell),
or non-essential/for-older-versions (Python documentation).

At least it no longer seems terribly weird to me that I never noticed
this built-in before... I got this from the python docs in reference to
buffer and others:

"Python programmers, trainers, students and bookwriters should feel
free to bypass these functions without concerns about missing something
important".

Is buffer safe to use? Is there an alternative?

ctypes objects support the buffer interface

How can you tell what objects support the buffer interface? Is
anything visible at the python level, or do you need to dig into the C
source?

Regarding documentation, I assume the C PyBufferObject is the
underlying thing for the python-level buffer? If so, is the best place
for docs on this ancient object to glean what I can from this link:
http://www.python.org/doc/1.5.2p2/api/bufferObjects.html ?

Any help is appreciated... I'd like to understand what I can about this
object if I'm to use it... I'm wary of nasty surprises.

Russ

 

[Stefan Behnel]

How can you tell what objects support the buffer interface? Is
anything visible at the python level, or do you need to dig into the C
source?

At the C level, there is a function for testing:

int PyObject_CheckReadBuffer(PyObject* o)

http://docs.python.org/dev/api/abstract-buffer.html

According to

http://docs.python.org/dev/lib/non-essential-built-in-funcs.html

a Python call to buffer(obj) will either return a buffer object or (tested)
raise a TypeError:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: buffer object expected

Stefan