file.close() freezes my script

[schwerdy]

Hi Python-Fans

I'm confused! :-(

I created a script that handles SC and CSV files. To convert CSV (or
in my case: pipe ('|') seperated values) to SC, I want to hunt my CSV
lines through /usr/bin/psc. Therefore I did:

tofilter, fromfilter, error = os.popen3("psc -S -L -d '|'")
map(tofilter.write,lines)
tofilter.close()

For some CSV-files my script freezes at tofilter.close(). Removing
some lines from the "lines" sequence (that represents the CSV-file)
always helps. But imho "psc" is not the cause. Piping my CSV-file
through psc in bash always works. Isn't it crazy? I tried python2.2
and python2.3 from the debian distribution.

Anyone who has an explanation, workaround, hint or anything?

Schwerdy

 

[Thomas Guetttler]

Am Fri, 05 Sep 2003 01:41:03 -0700 schrieb schwerdy:

Hi Python-Fans

I'm confused! :-(

I created a script that handles SC and CSV files. To convert CSV (or
in my case: pipe ('|') seperated values) to SC, I want to hunt my CSV
lines through /usr/bin/psc. Therefore I did:

tofilter, fromfilter, error = os.popen3("psc -S -L -d '|'")
map(tofilter.write,lines)
tofilter.close()

You don't need the '|'.

Next thing: If psc write to stderr *and* stdout you
might get a deadlock because the buffer gets full.
It is better to use the popen version which writes
stderr and stdout to one stream. Or you redirect
stderr to a file:
" psc .... 2> /tmp/foo"

thomas

 

[Bob Halley]

I created a script that handles SC and CSV files. To convert CSV (or
in my case: pipe ('|') seperated values) to SC, I want to hunt my CSV
lines through /usr/bin/psc. Therefore I did:

tofilter, fromfilter, error = os.popen3("psc -S -L -d '|'")
map(tofilter.write,lines)
tofilter.close()

For some CSV-files my script freezes at tofilter.close(). Removing
some lines from the "lines" sequence (that represents the CSV-file)
always helps. But imho "psc" is not the cause. Piping my CSV-file
through psc in bash always works. Isn't it crazy? I tried python2.2
and python2.3 from the debian distribution.

Anyone who has an explanation, workaround, hint or anything?

This sounds like a classic pipe deadlock. My guess is that the child
process is blocked trying to write to its stdout, which you haven't
read. The parent is blocking trying to finish all that writing.

The issue is described in the Python Library Reference, section 6.8.2,
"Flow Control Issues".

If you wrote psc's output to a file, used select(), or used different
threads (e.g. a writer thread and a reader thread), you could avoid
the deadlock.

/Bob

 

[schwerdy]

Thank you for your answer

This sounds like a classic pipe deadlock.

I didn't have the time so far to do a workaround, but I'll report if I
fixing succeeds.

Thanks again!

Schwerdy

 

[schwerdy]

Thank you for your answer

This sounds like a classic pipe deadlock.

I didn't have the time so far to do a workaround, but I'll report if
fixing it succeeds.

Thanks again!

Schwerdy