Long strings as function parameters

Discussion in 'Python' started by onlyonemc@gmail.com, Jan 9, 2005.

  1. Guest

    I would like to have functions that operate on long strings, 10-100 MB.
    In C I would of course pass a pointer to the string for a quick
    function call. What is an efficient way to do this in python?
    Cheers,
    -mark
    , Jan 9, 2005
    #1
    1. Advertising

  2. wrote:
    > I would like to have functions that operate on long strings, 10-100 MB.
    > In C I would of course pass a pointer to the string for a quick
    > function call. What is an efficient way to do this in python?


    Err, just pass the string to the function?
    In Python, all function arguments are passed by (object)reference.
    So if you are afraid that Python passes your 50Mb string object
    /by value/ (thus creating a copy): it doesn't.

    --Irmen
    Irmen de Jong, Jan 9, 2005
    #2
    1. Advertising

  3. Dan Bishop Guest

    wrote:
    > I would like to have functions that operate on long strings, 10-100

    MB.
    > In C I would of course pass a pointer to the string for a quick
    > function call. What is an efficient way to do this in python?
    > Cheers,


    In Python, *every* expression is a pointer. This fact is clearest when
    you look a C-implemented Python functions (which have parameter types
    and return types of PyObject*), or when using mutable types.

    >>> class Spam:

    .... def __init__(self):
    .... self.foo = 0
    ....
    >>> x = Spam()
    >>> y = x
    >>> y.foo = 17
    >>> x.foo

    17

    Compare this to the C code:

    typedef struct {int foo;} Spam;
    ....
    Spam *x, *y;
    ....
    y = x;
    y->foo = 17;
    printf("%d\n", x->foo);
    Dan Bishop, Jan 9, 2005
    #3
  4. On Sun, 09 Jan 2005 08:36:04 -0800, onlyonemc wrote:

    > I would like to have functions that operate on long strings, 10-100 MB. In
    > C I would of course pass a pointer to the string for a quick function
    > call. What is an efficient way to do this in python? Cheers,
    > -mark



    Others have pointed out that there is no particular inefficient way to
    pass a string.

    However that only solves the immediate problem of passing. Once inside the
    function you should be aware that all string operations that change the
    string will create a new string, including slicing and such.

    This may not be a problem if you are examining it in small chunks, but if
    you routinely taking distinct multi-megabyte hunks out of it with slices,
    you will be constructing and destructing a lot of strings.

    I had thought there was an obvious class in the standard library to assist
    with this, but I must have been wrong. I think you can load it as an array
    from the array module (as long as it is an array of bytes and not an
    encoding like UTF-8 or something), or you might be able to use the mmap
    module if the string comes from a file. Both of those techniques can also
    load the info directly from a file so there is only one copy needed.
    (Wasn't there a "buffer" kind of class at one point, that you could slice
    into and get something back that didn't make any copies of strings?)

    Again, this is only an issue depending on usage, and you should probably
    prototype it while ignoring these issues and see if it is fast enough. But
    if it isn't, there are options.
    Jeremy Bowers, Jan 10, 2005
    #4
  5. Terry Reedy Guest

    "Dan Bishop" <> wrote in message
    news:...
    > In Python, *every* expression is a pointer.


    Minor but to me important nit: this should start "In the CPython
    implementation of Python" ...

    Humans can understand and execute Python code without knowing about
    computer memory addresses (pointers). Other computer languages can also do
    something slightly different from CPython, as I believe is the case with
    Jython.

    The Python language is defined by sematics, not by mechanisms, especially
    low-level interpreter-specific mechanisms. This is a large part of what
    makes it readable.

    To the OP: a Python function call binds argument objects to either local
    parameter names or slots within a named catchall list or dict. Like other
    binding (assignment) operations, there is no copying.

    Terry J. Reedy
    Terry Reedy, Jan 10, 2005
    #5
  6. Nick Coghlan Guest

    Jeremy Bowers wrote:
    > I had thought there was an obvious class in the standard library to assist
    > with this, but I must have been wrong.


    buffer is the closest current contender, but I believe it's on the outer due to
    some problems with its implementation.

    I think the intention is to eventually have a 'bytes' type, but I don't recall
    if that was going to be mutable or immutable.

    Cheers,
    Nick.

    --
    Nick Coghlan | | Brisbane, Australia
    ---------------------------------------------------------------
    http://boredomandlaziness.skystorm.net
    Nick Coghlan, Jan 10, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. George Marsaglia

    Assigning unsigned long to unsigned long long

    George Marsaglia, Jul 8, 2003, in forum: C Programming
    Replies:
    1
    Views:
    674
    Eric Sosman
    Jul 8, 2003
  2. Daniel Rudy

    unsigned long long int to long double

    Daniel Rudy, Sep 19, 2005, in forum: C Programming
    Replies:
    5
    Views:
    1,186
    Peter Shaggy Haywood
    Sep 20, 2005
  3. Mathieu Dutour

    long long and long

    Mathieu Dutour, Jul 17, 2007, in forum: C Programming
    Replies:
    4
    Views:
    473
    santosh
    Jul 24, 2007
  4. Bart C

    Use of Long and Long Long

    Bart C, Jan 9, 2008, in forum: C Programming
    Replies:
    27
    Views:
    797
    Peter Nilsson
    Jan 15, 2008
  5. veryhotsausage
    Replies:
    1
    Views:
    1,799
    veryhotsausage
    Jul 4, 2008
Loading...

Share This Page