marshal locks program even in thread

Discussion in 'Python' started by Zac Burns, Nov 8, 2008.

  1. Zac Burns

    Zac Burns Guest

    Greetings,

    It seems that marshal.load will lock the problem if the file object
    (in this case a pipe) is not ready to be read from - even if it's done
    in a thread.

    The use case here is in writing a scripting interface for perforce
    using the -G option
    (http://www.perforce.com/perforce/doc.072/manuals/cmdref/o.gopts.html#1040647).
    We are using the sub process module to communicate through pipes to
    avoid using the hard-drive. Furthermore, because the machine is using
    windows we need threads for each pipe (stdin, stdout, stderr)
    initialized from the beginning to keep from buffer lock.

    If marshal.load would accept a cStringIO object, I would have a
    workaround - marshal.load(cStringIO.cStringIO(pipe.read())) because
    cStringIO does not have the same problem and I'm not as worried about
    the extra memory as I am the hard-drive use.

    I can't read the C code yet, but if I had to make a guess I would say
    python is grabbing the GIL and then attempting to read from the pipe,
    instead of the other way around.

    Could someone help me to understand what is going on and suggest a
    workaround? I hope I've been clear about the nature of the problem.
    Thanks,
     
    Zac Burns, Nov 8, 2008
    #1
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.