creating a block file for file-like object

[Iain]

Hi,

I have a function that only accepts filenames, rather than file-like
objects (because its a wrapper around a C++ function, I think).

I want to feed some potentially large "files" into this function, but
they are coming in as streams (eg from a url) and so are only
represented in my code as file-like objects.

Can someone give me some pointers as to how I might create some sort
of blocking device file or named pipe or something that will give this
stream a filename without having to actually write the data to disk
and then read it back in before deleting it?

I've sort of tried the FIFO thing, but I think I'm getting caught by
its blocking behaviour on open so as soon as I try to open the named
pipe (whether for reading or writing) my script just hangs.

Any help would be appreciated.

Cheers
Iain

 

[Lawrence D'Oliveiro]

In message

Can someone give me some pointers as to how I might create some sort
of blocking device file or named pipe ...

mkfifo /path/to/named/pipe

 

[Iain]

In message


    mkfifo /path/to/named/pipe

Thanks.

I did get that far myself with os.mkfifo - my problem is actually
using it. To me it seemed like I wanted to do something like

streamobj = urllib.urlopen("http://whereever.com/file")
fifoobj = open("/path/to/named/pipe","w")
fifoobj.write(streamobj.read())
TroublesomeFunction("/path/to/named/pipe")

But as soon as you get to the second line the code hangs (apparently
because of the blocking behaviour of the named pipe).

Any pointers here would be much appreciated.

 

[Iain]

Thanks.

I did get that far myself with os.mkfifo - my problem is actually
using it. To me it seemed like I wanted to do something like

streamobj = urllib.urlopen("http://whereever.com/file")
fifoobj = open("/path/to/named/pipe","w")
fifoobj.write(streamobj.read())
TroublesomeFunction("/path/to/named/pipe")

But as soon as you get to the second line the code hangs (apparently
because of the blocking behaviour of the named pipe).

Any pointers here would be much appreciated.

Well I did work out *a* solution this way:

pipename = os.tmpnam()
os.mkfifo(pipename)
pid = os.fork()
if pid==0:
fifoobj = open(pipename,"w")
fifoobj.write(streamobj.read())
fifoobj.close()
os.unlink(pipename)
else:
TroublesomeFunction(pipename)

I'd have to say it doesn't strike me as the BEST solution, but it
works. In particular the use of os.tmpnam() gives a warning that its
use is a potential security vulnerability, but this is inevitable if
taking the named pipe approach (any other suggestions are most
welcome, of course). And it doesn't fail very gracefully in that if
TroublesomeFunction stops before attempting to open/read the pipe,
then the child process stays hung waiting for the other end of the
pipe to open. In an interactive python shell you also get some weird
behaviour in this situation, but I'm not entirely sure what the exact
cause of that is.

 

[Lawrence D'Oliveiro]

In message

Well I did work out *a* solution this way:

pipename = os.tmpnam()
os.mkfifo(pipename)
pid = os.fork()
if pid==0:
fifoobj = open(pipename,"w")
fifoobj.write(streamobj.read())
fifoobj.close()
os.unlink(pipename)
else:
TroublesomeFunction(pipename)

OK, so TroublesomeFunction is reading from the pipe while the child is
writing? Naturally you'd get a block if you tried to do both in the same
process.

And it doesn't fail very gracefully in that if
TroublesomeFunction stops before attempting to open/read the pipe,
then the child process stays hung waiting for the other end of the
pipe to open.

Perhaps the parent should open the pipe for reading, before calling
TroublesomeFunction. If the parent then dies, the child will get a "broken
pipe" signal, which by default should kill it.

 

[Iain]

Perhaps the parent should open the pipe for reading, before calling
TroublesomeFunction. If the parent then dies, the child will get a "broken
pipe" signal, which by default should kill it.

Yeah, that seems to work well, I think. Thanks for the help! I also
realised the child process was continuing and returning when I didn't
really want it to. So for anyone else who stumbles across this, it
ended up being

pipename = os.tmpnam()
os.mkfifo(pipename)
pid = os.fork()
if pid==0:
fifoobj = open(pipename,"w")
fifoobj.write(streamobj.read())
fifoobj.close()
os.unlink(pipename)
os._exit(os.EX_OK)
else:
fifoopen = open(pipename,"r")
TroublesomeFunction(pipename)