pythonic exec* spawn*

D

Daniel Nogradi

Is it possible to pass a python object to a python program as argument?

In my program I would like to start executing an other python program
and don't wait until that finishes, only launch it and keep running
the original program. The obvious way I can think of it is using the
exec* or spawn* function families. However, these are generic
functions to start any external program and I was wondering if there
was a special way to start a python program (as opposed to an
arbitrary program), especially because I would like to pass data to
this other program in the form of python objects.

While the original program is running it creates all sorts of data in
a nice pythonic way, for example as a class instance with lot of
member data, and then I wouldn't want to convert all of this into
command line arguments because that would create a huge list. I would
like to pass the whole object at once to the second python program
which should start doing its thing with it while the original program
should keep running.

I was looking around for threading and forking but couldn't come up
with a clear solution.
 
K

kmkz89

You could possibly pickle an object and send it, but that could be a
bit messy...
 
D

dwelch

You could possibly pickle an object and send it, but that could be a
bit messy...
pickle it (or shelve?) and then pass the pickle/shleve filename to the
other process as a command line parameter?

Not sure if this would work or not, but that's where I would start.

-Don
 
R

Rene Pijlman

Daniel Nogradi:
I would like to pass the whole object at once to the second python
program which should start doing its thing with it while the original
program should keep running.

os.fork() does that (on Mac and Unix).
 
D

Daniel Nogradi

I would like to pass the whole object at once to the second python
os.fork() does that (on Mac and Unix).

Okay, but how? It seems to me that if the process which issued
os.fork() ends, then the forked process also ends. And what I would
need is that regardless of the execution time of the original program,
the second one which the original starts (or forks) should finish what
it supposed to do. Roughly I have the following in mind:

-------------------------------------------------------------------------------
def longer( data ):
# do some stuff, that takes long


# do something to collect data

data = something

# here call the function longer( data ) in a way
# that execution of the code doesn't stop, so
# we get to the following print and then to the end

print "okay, the first program finished"
------------------------------------------------------------------------------

But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.
 
R

Rene Pijlman

Daniel Nogradi:
Okay, but how?

Sorry, fork() is implemented strictly on a 'need to know' basis :)
It seems to me that if the process which issued os.fork() ends, then
the forked process also ends.

No, no, they're not a quantum mechanic photon pair. Every process decides
for itself if and when to end. As long as it's not signalled/killed, that
is.
But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.

You can search for "daemonize".

Here's a recipe:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012

The concept is best explained in this book:
http://www.kohala.com/start/apue.html
 
D

Daniel Nogradi

os.fork() does that (on Mac and Unix).
Sorry, fork() is implemented strictly on a 'need to know' basis :)


No, no, they're not a quantum mechanic photon pair. Every process decides
for itself if and when to end. As long as it's not signalled/killed, that
is.


You can search for "daemonize".

Here's a recipe:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012

The concept is best explained in this book:
http://www.kohala.com/start/apue.html

Okay, thanks a lot, I'll look into that.
 
N

Nick Craig-Wood

Daniel Nogradi said:
Is it possible to pass a python object to a python program as
argument?

Yes - you can pickle it.

http://www.python.org/doc/2.4.2/lib/module-cPickle.html

You can pickle just about anything into a string and unpickle it back
into python objects. You shouldn't accept pickles from untrusted
sources as this may reperesent a security problem.
In my program I would like to start executing an other python program
and don't wait until that finishes, only launch it and keep running
the original program. The obvious way I can think of it is using the
exec* or spawn* function families. However, these are generic
functions to start any external program and I was wondering if there
was a special way to start a python program (as opposed to an
arbitrary program), especially because I would like to pass data to
this other program in the form of python objects.

While the original program is running it creates all sorts of data in
a nice pythonic way, for example as a class instance with lot of
member data, and then I wouldn't want to convert all of this into
command line arguments because that would create a huge list. I would
like to pass the whole object at once to the second python program
which should start doing its thing with it while the original program
should keep running.

I was looking around for threading and forking but couldn't come up
with a clear solution.

If you want to fork(), then I would use the subprocess module to
create a child process with a pipe to it. I would then pass python
objects as pickles back and forth across that pipe.

http://www.python.org/doc/2.4.2/lib/module-subprocess.html

You'll probably find threading easier though.

http://www.python.org/doc/2.4.2/lib/module-threading.html

But it will use your multiple CPUs less efficiently than fork()-ing.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,905
Latest member
Kristy_Poole

Latest Threads

Top