creating a block file for file-like object

Discussion in 'Python' started by Iain, Nov 7, 2008.

  1. Iain

    Iain Guest

    Hi,

    I have a function that only accepts filenames, rather than file-like
    objects (because its a wrapper around a C++ function, I think).

    I want to feed some potentially large "files" into this function, but
    they are coming in as streams (eg from a url) and so are only
    represented in my code as file-like objects.

    Can someone give me some pointers as to how I might create some sort
    of blocking device file or named pipe or something that will give this
    stream a filename without having to actually write the data to disk
    and then read it back in before deleting it?

    I've sort of tried the FIFO thing, but I think I'm getting caught by
    its blocking behaviour on open so as soon as I try to open the named
    pipe (whether for reading or writing) my script just hangs.

    Any help would be appreciated.

    Cheers
    Iain
     
    Iain, Nov 7, 2008
    #1
    1. Advertising

  2. In message
    <>, Iain
    wrote:

    > Can someone give me some pointers as to how I might create some sort
    > of blocking device file or named pipe ...


    mkfifo /path/to/named/pipe
     
    Lawrence D'Oliveiro, Nov 7, 2008
    #2
    1. Advertising

  3. Iain

    Iain Guest

    On Nov 7, 4:42 pm, Lawrence D'Oliveiro <l...@geek-
    central.gen.new_zealand> wrote:
    > In message
    > <>, Iain
    > wrote:
    >
    > > Can someone give me some pointers as to how I might create some sort
    > > of blocking device file or named pipe ...

    >
    >     mkfifo /path/to/named/pipe


    Thanks.

    I did get that far myself with os.mkfifo - my problem is actually
    using it. To me it seemed like I wanted to do something like

    streamobj = urllib.urlopen("http://whereever.com/file")
    fifoobj = open("/path/to/named/pipe","w")
    fifoobj.write(streamobj.read())
    TroublesomeFunction("/path/to/named/pipe")

    But as soon as you get to the second line the code hangs (apparently
    because of the blocking behaviour of the named pipe).

    Any pointers here would be much appreciated.
     
    Iain, Nov 8, 2008
    #3
  4. Iain

    Iain Guest

    On Nov 8, 10:00 am, Iain <> wrote:
    > On Nov 7, 4:42 pm, Lawrence D'Oliveiro <l...@geek-
    >
    > central.gen.new_zealand> wrote:
    > > In message
    > > <>, Iain
    > > wrote:

    >
    > > > Can someone give me some pointers as to how I might create some sort
    > > > of blocking device file or named pipe ...

    >
    > >     mkfifo /path/to/named/pipe

    >
    > Thanks.
    >
    > I did get that far myself with os.mkfifo - my problem is actually
    > using it. To me it seemed like I wanted to do something like
    >
    > streamobj = urllib.urlopen("http://whereever.com/file")
    > fifoobj = open("/path/to/named/pipe","w")
    > fifoobj.write(streamobj.read())
    > TroublesomeFunction("/path/to/named/pipe")
    >
    > But as soon as you get to the second line the code hangs (apparently
    > because of the blocking behaviour of the named pipe).
    >
    > Any pointers here would be much appreciated.


    Well I did work out *a* solution this way:

    pipename = os.tmpnam()
    os.mkfifo(pipename)
    pid = os.fork()
    if pid==0:
    fifoobj = open(pipename,"w")
    fifoobj.write(streamobj.read())
    fifoobj.close()
    os.unlink(pipename)
    else:
    TroublesomeFunction(pipename)

    I'd have to say it doesn't strike me as the BEST solution, but it
    works. In particular the use of os.tmpnam() gives a warning that its
    use is a potential security vulnerability, but this is inevitable if
    taking the named pipe approach (any other suggestions are most
    welcome, of course). And it doesn't fail very gracefully in that if
    TroublesomeFunction stops before attempting to open/read the pipe,
    then the child process stays hung waiting for the other end of the
    pipe to open. In an interactive python shell you also get some weird
    behaviour in this situation, but I'm not entirely sure what the exact
    cause of that is.
     
    Iain, Nov 10, 2008
    #4
  5. In message
    <>, Iain
    wrote:

    > Well I did work out *a* solution this way:
    >
    > pipename = os.tmpnam()
    > os.mkfifo(pipename)
    > pid = os.fork()
    > if pid==0:
    > fifoobj = open(pipename,"w")
    > fifoobj.write(streamobj.read())
    > fifoobj.close()
    > os.unlink(pipename)
    > else:
    > TroublesomeFunction(pipename)


    OK, so TroublesomeFunction is reading from the pipe while the child is
    writing? Naturally you'd get a block if you tried to do both in the same
    process.

    > And it doesn't fail very gracefully in that if
    > TroublesomeFunction stops before attempting to open/read the pipe,
    > then the child process stays hung waiting for the other end of the
    > pipe to open.


    Perhaps the parent should open the pipe for reading, before calling
    TroublesomeFunction. If the parent then dies, the child will get a "broken
    pipe" signal, which by default should kill it.
     
    Lawrence D'Oliveiro, Nov 10, 2008
    #5
  6. Iain

    Iain Guest


    > Perhaps the parent should open the pipe for reading, before calling
    > TroublesomeFunction. If the parent then dies, the child will get a "broken
    > pipe" signal, which by default should kill it.


    Yeah, that seems to work well, I think. Thanks for the help! I also
    realised the child process was continuing and returning when I didn't
    really want it to. So for anyone else who stumbles across this, it
    ended up being

    pipename = os.tmpnam()
    os.mkfifo(pipename)
    pid = os.fork()
    if pid==0:
    fifoobj = open(pipename,"w")
    fifoobj.write(streamobj.read())
    fifoobj.close()
    os.unlink(pipename)
    os._exit(os.EX_OK)
    else:
    fifoopen = open(pipename,"r")
    TroublesomeFunction(pipename)
     
    Iain, Nov 11, 2008
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Showjumper
    Replies:
    1
    Views:
    706
    Showjumper
    Mar 19, 2005
  2. morrell
    Replies:
    1
    Views:
    965
    roy axenov
    Oct 10, 2006
  3. Patrick Kowalzick
    Replies:
    5
    Views:
    477
    Patrick Kowalzick
    Mar 14, 2006
  4. kgk
    Replies:
    1
    Views:
    299
    Marc 'BlackJack' Rintsch
    Jul 11, 2007
  5. Wolfgang Nádasi-Donner
    Replies:
    0
    Views:
    125
    Wolfgang Nádasi-Donner
    May 31, 2007
Loading...

Share This Page