Long strings as function parameters

[onlyonemc]

I would like to have functions that operate on long strings, 10-100 MB.
In C I would of course pass a pointer to the string for a quick
function call. What is an efficient way to do this in python?
Cheers,
-mark

 

[Irmen de Jong]

I would like to have functions that operate on long strings, 10-100 MB.
In C I would of course pass a pointer to the string for a quick
function call. What is an efficient way to do this in python?

Err, just pass the string to the function?
In Python, all function arguments are passed by (object)reference.
So if you are afraid that Python passes your 50Mb string object
/by value/ (thus creating a copy): it doesn't.

--Irmen

 

[Dan Bishop]

I would like to have functions that operate on long strings, 10-100 MB.
In C I would of course pass a pointer to the string for a quick
function call. What is an efficient way to do this in python?
Cheers,

In Python, *every* expression is a pointer. This fact is clearest when
you look a C-implemented Python functions (which have parameter types
and return types of PyObject*), or when using mutable types.
.... def __init__(self):
.... self.foo = 0
....17

Compare this to the C code:

typedef struct {int foo;} Spam;
....
Spam *x, *y;
....
y = x;
y->foo = 17;
printf("%d\n", x->foo);

 

[Jeremy Bowers]

I would like to have functions that operate on long strings, 10-100 MB. In
C I would of course pass a pointer to the string for a quick function
call. What is an efficient way to do this in python? Cheers,
-mark

Others have pointed out that there is no particular inefficient way to
pass a string.

However that only solves the immediate problem of passing. Once inside the
function you should be aware that all string operations that change the
string will create a new string, including slicing and such.

This may not be a problem if you are examining it in small chunks, but if
you routinely taking distinct multi-megabyte hunks out of it with slices,
you will be constructing and destructing a lot of strings.

I had thought there was an obvious class in the standard library to assist
with this, but I must have been wrong. I think you can load it as an array
from the array module (as long as it is an array of bytes and not an
encoding like UTF-8 or something), or you might be able to use the mmap
module if the string comes from a file. Both of those techniques can also
load the info directly from a file so there is only one copy needed.
(Wasn't there a "buffer" kind of class at one point, that you could slice
into and get something back that didn't make any copies of strings?)

Again, this is only an issue depending on usage, and you should probably
prototype it while ignoring these issues and see if it is fast enough. But
if it isn't, there are options.

 

[Terry Reedy]

In Python, *every* expression is a pointer.

Minor but to me important nit: this should start "In the CPython
implementation of Python" ...

Humans can understand and execute Python code without knowing about
computer memory addresses (pointers). Other computer languages can also do
something slightly different from CPython, as I believe is the case with
Jython.

The Python language is defined by sematics, not by mechanisms, especially
low-level interpreter-specific mechanisms. This is a large part of what
makes it readable.

To the OP: a Python function call binds argument objects to either local
parameter names or slots within a named catchall list or dict. Like other
binding (assignment) operations, there is no copying.

Terry J. Reedy

 

[Nick Coghlan]

I had thought there was an obvious class in the standard library to assist
with this, but I must have been wrong.

buffer is the closest current contender, but I believe it's on the outer due to
some problems with its implementation.

I think the intention is to eventually have a 'bytes' type, but I don't recall
if that was going to be mutable or immutable.

Cheers,
Nick.