Reading variables from a forked child (Python C/API)

[Donn Cave]

Quoth MrEntropy <[email protected]>:

| I'm having a little trouble getting an idea running. I am writing a C
| program which is really a frontend to a Python program. Now, my C
| program starts up, does some initialisation like initialisation of it's
| variables and Py_Initialize() and then it fork()s. After forking the
| child launches the python program with Py_Main() and with the parent I
| want to be able to read the variables of the Python program. I have
| tried many ways but cannot find a solution to do this.

That's because there is no solution. After a fork, the only effect
observable in the parent process is the return value of fork, which
will have a non-zero value. Subsequent execution in the child process
occurs completely independently from the parent, and leaves no trace
whatever. So you can't read variables from memory, that were set by
the child.

Past that, I deleted the rest of your post because it sort of avoids
the question of what you're really trying to accomplish and what errors
you actually got, but note that just for the sake of avoiding segmentation
faults etc., it's a good idea when writing in C to check return values:

obj = PyMapping_GetItemString(dict, "foo");
if (obj) {
...
} else {
...
}

Anyway, if you really need a subprocess, you're going to have to
communicate with it via some sort of I/O, like UNIX pipes or temporary
files or something. You probably don't need to call Python from C,
may as well just invoke python (cf. os.spawnv)

Donn Cave, (e-mail address removed)

 

[Jonathan Conrad]

#ifdef def

Quoth MrEntropy <[email protected]>:

| I'm having a little trouble getting an idea running. I am writing a C
| program which is really a frontend to a Python program. Now, my C
| program starts up, does some initialisation like initialisation of it's
| variables and Py_Initialize() and then it fork()s. After forking the
| child launches the python program with Py_Main() and with the parent I
| want to be able to read the variables of the Python program. I have
| tried many ways but cannot find a solution to do this.

That's because there is no solution. After a fork, the only effect
observable in the parent process is the return value of fork, which
will have a non-zero value. Subsequent execution in the child process
occurs completely independently from the parent, and leaves no trace
whatever. So you can't read variables from memory, that were set by
the child.

Though there is probably some solution involving a third party
such as POSIX threads, Linux has a clone(2) procedure which
fork(2) is a macro of. It is tricky to use, but allows for
having the child process have the same pid, memory, memory
mappings, file tables, etc. BSD mmap(2) is more portable and can
be used for interprocess sharing. The solution which Don Cave
provided is probably the one that Mr. Entropy needs to start
with. More specifically, ...

#else

int main (int argc, char **argv)
{
int fd0, fd1, fd2;
{
int fd_in [2], fd_out [2], fd_err [2];
pipe (fd_in); fd0 = fd_in [1];
pipe (fd_out); fd1 = fd_out [0];
pipe (fd_err); fd2 = fd_err [0];
if (!fork ()) {
dup2 (fd_in [0], 0); close (fd0);
dup2 (fd_out [1], 1); close (fd1);
dup2 (fd_err [1], 2); close (fd2);
execl ("python", "python", 0);
perror ("execution failed");
exit (1);
}
}

write (fd0, "print None\n", 11);
write (1, b, read (fd1, b, sizeof (b) /* 5 or more */));

#define write0(q...) write (fd0, "print " #q "\n", \
sizeof (#q) + 5 + 1 - 1)
write0 (True); write (1, b, read (fd1, b, sizeof (b)));
write0 (5, str (5.0), ...);
write (1, b, read (fd1, b, sizeof (b)));
return write0 (True, False, None,
True and False,
True and None,
False and None,
True and False and None) && 0;
}

#endif

 

[MrEntropy]

Greetings,
I'm having a little trouble getting an idea running. I am writing a C
program which is really a frontend to a Python program. Now, my C
program starts up, does some initialisation like initialisation of it's
variables and Py_Initialize() and then it fork()s. After forking the
child launches the python program with Py_Main() and with the parent I
want to be able to read the variables of the Python program. I have
tried many ways but cannot find a solution to do this. The ONLY testcase
in fact that I could get working is this:

#include <Python.h>
#include <stdio.h>

int main(){
char* buffer;
PyObject* mod;
PyObject* dict;
PyObject* obj;

Py_Initialize();

PyRun_SimpleString("foo = 'bar'");
mod = PyImport_AddModule("__main__");
dict = PyModule_GetDict(mod);
obj = PyMapping_GetItemString(dict, "foo");
buffer = PyString_AsString(obj);
printf(buffer);
Py_Finalize();
return 0;
}

And that only proves to me that I can get the variables out of python,
nothing to do with forking. When i try doing very much the same thing to
a forked child I get sedfaults. In other words the child gets
Py_Main()'d and executes the child, while in the parent i do much the
same as the above with the exception of PyImport_AddModule() being
changed to PyImport_ImportModule() because main is already preset i gather.

So my questions are these:

Does the fact that I initialise before the fork have anything to do with
my failure? And if so, is it because some pointers are invalidated?

Is the problem to do with the inability to perform operations on a
python environment initialised through C while it's executing? Because
i'm technically executing operations on it from the parent while the
child in the same environment is running the program.

In the program that the child is running, i need to access some
variables it has in __main__. main also has an instantiated object 'd'
which i also need some variables from, i.e. __main__.d.foo. How can I
access these?

Overall, I would REALLY appreciate an example of how to do what I want
to achieve, examples help me a lot so if you wouldn't mind typing one up
or pointing me to some relevant documentation i'd appreciate it.

Thankyou.