Problem flushing stderr in embedded python

[Farshid Lashkari]

Hi,

My application has python embedded into it. I noticed that when I run any
python code the output is buffered and doesn't get flushed until my
application exits. To fix this I simply flush sys.stdout and sys.stderr
every once in while by using the following code:

//Get handle to python stdout file and flush it
PyObject *pyStdout = PySys_GetObject("stdout");
if(pyStdout && PyFile_Check(pyStdout)) {
PyObject *result = PyObject_CallMethod(pyStdout,"flush",NULL);
Py_XDECREF(result);
}

//Get handle to python stderr file and flush it
PyObject *pyStderr = PySys_GetObject("stderr");
if(pyStderr && PyFile_Check(pyStderr)) {
PyObject *result = PyObject_CallMethod(pyStderr,"flush",NULL);
Py_XDECREF(result);
}

This works for stdout but not for stderr. In order to flush stderr I have to
do:

PyRun_SimpleString("sys.stderr.flush()");

Does anybody know why my first method doesn't work with stderr? I checked
and PyObject_CallMethod is being called for both objects and no errors are
occuring. The solution I have is acceptable, but I would still like to know
why my first method didn't work.

BTW, I'm running python 2.3 on Windows XP

Thanks,

Farshid

 

[caroundw5h]

Hi,

My application has python embedded into it. I noticed that when I run any
python code the output is buffered and doesn't get flushed until my
application exits. To fix this I simply flush sys.stdout and sys.stderr
every once in while by using the following code:

//Get handle to python stdout file and flush it
PyObject *pyStdout = PySys_GetObject("stdout");
if(pyStdout && PyFile_Check(pyStdout)) {
PyObject *result = PyObject_CallMethod(pyStdout,"flush",NULL);
Py_XDECREF(result);
}

//Get handle to python stderr file and flush it
PyObject *pyStderr = PySys_GetObject("stderr");
if(pyStderr && PyFile_Check(pyStderr)) {
PyObject *result = PyObject_CallMethod(pyStderr,"flush",NULL);
Py_XDECREF(result);
}

This works for stdout but not for stderr. In order to flush stderr I have to
do:

PyRun_SimpleString("sys.stderr.flush()");

Does anybody know why my first method doesn't work with stderr? I checked
and PyObject_CallMethod is being called for both objects and no errors are
occuring. The solution I have is acceptable, but I would still like to know
why my first method didn't work.

BTW, I'm running python 2.3 on Windows XP

Thanks,

Farshid

I'm sorry to notice that you haven't received any responses back to
your questions even though you have found a workaround. I've noticed
this trend quite a lot in the python community and having learned
python first and now C(in order to understand python better) I'm
frankly disspapointed. I believe python and C are a wonderful marriage
but I suppose because many ppl come to python as their intro language
they info on extending and embedding is limited. I myself haven't
started using the python api but will soon, I hope I for one will be
someone who will be able to provide an answer next time.

Good luck.

BTW: from a C perspective. I don't understand your problem though.
stdout and stderr are the normal output device for your system -
usally your screen. I don't see why you should have to call one via
python sys module.
Perhaps because one doesn't normally flush(stderr) explicity but
flush(stdout).

interested to know as well.

 

[Farshid Lashkari]

I don't understand it from a C perspective either. I poked around the source
code for python and found where it is flushing the file. It is simply
calling fflush on the FILE handle. Also, I'm somewhat bewildered as to why
running the string "sys.stderr.flush()" is different than directly calling
the flush method of the stderr object. Oh well, I guess some things are
better left unknown.

 

[Donn Cave]

Quoth "Farshid Lashkari" <[email protected]>:
| I don't understand it from a C perspective either. I poked around the source
| code for python and found where it is flushing the file. It is simply
| calling fflush on the FILE handle. Also, I'm somewhat bewildered as to why
| running the string "sys.stderr.flush()" is different than directly calling
| the flush method of the stderr object. Oh well, I guess some things are
| better left unknown.

It could be different if the stderr object - i.e., PySys_GetObject("stderr") -
is actually not sys.stderr. I don't understand why, but am not familiar
with PySys_*, so I offer that as a hypothesis you might be able to test.
By the way, since as you observed it's only calling fflush(), I think you
could leave Python out of it altogether and do this with just
fflush(stdout);
fflush(stderr);

Donn Cave, (e-mail address removed)

 

[Farshid Lashkari]

Here's the code for PySys_GetObject. It's getting the object from the sys
dictionary, so it should be the correct object.

PyObject *
PySys_GetObject(char *name)
{
PyThreadState *tstate = PyThreadState_Get();
PyObject *sd = tstate->interp->sysdict;
if (sd == NULL)
return NULL;
return PyDict_GetItemString(sd, name);
}

I've tried calling fflush on stderr, but that doesn't work either.

I've encountered another weird problem also. If my python code causes an
error and I don't flush stderr, then I get a runtime error when my app
exits. It also prints out:

Exception exceptions.NameError: "global name 'y' is not defined" in 'garbage
collection' ignored
Fatal Python error: unexpected exception during garbage collection.

Does this give anybody a clue as to what's happening?

 

[Donn Cave]

Quoth "Farshid Lashkari" <[email protected]>:
....
| I've tried calling fflush on stderr, but that doesn't work either.

OK, this would seem to mean that your sys.stderr has been replaced
with something else. I guess I don't know for sure what you can
count on in Windows, but I would have to expect that fflush on stderr
really does work - that is, it flushes stderr's buffer.

| I've encountered another weird problem also. If my python code causes an
| error and I don't flush stderr, then I get a runtime error when my app
| exits. It also prints out:
|
| Exception exceptions.NameError: "global name 'y' is not defined" in 'garbage
| collection' ignored
| Fatal Python error: unexpected exception during garbage collection.
|
| Does this give anybody a clue as to what's happening?

Doesn't do anything for me, but you may find out what that is when you
locate the code for the faux stderr you're (not) flushing. I would be
more worried about this 'y' thing in principle, though, because it sounds
like a problem in the C (or whatever) to Python interface that could have
other unpleasant consequences.

Donn Cave, (e-mail address removed)

 

[Farshid Lashkari]

Well I've somewhat solved my problem. First off, the 'y' problem was just an
error I purposely created in order to trigger an exception. My application
uses a bunch of calls to PyObject_Call to run python code. After each of
these calls I added:

if(PyErr_Occurred()) PyErr_Print();

This will force the errors to be printed to stderr, and now calling the
flush method on the stderr object works. I still don't understand why
running the string "sys.stderr.flush()" is different. Maybe it forces any
current errors to be printed and then flushes them too. My app doesn't
replace the python stderr object either.

Anyways, thanks to everybody for your help and suggestions.

-Farshid