Doubts related to subprocess.Popen()

Discussion in 'Python' started by srinivasan srinivas, Jan 20, 2009.

  1. Hi,
    Does subprocess.Popen() count a new open file for each suprocess? I mean does it occupy an entry in file descriptor table of parent process?
    If so, wat is each file descriptor connected to?

    Thanks,
    Srini


    Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
     
    srinivasan srinivas, Jan 20, 2009
    #1
    1. Advertising

  2. srinivasan srinivas wrote:

    > Hi,
    > Does subprocess.Popen() count a new open file for each suprocess? I mean
    > does it occupy an entry in file descriptor table of parent process? If so,
    > wat is each file descriptor connected to?


    Usually, each new process has three file-descriptors associated with it -
    stdin,stdout and stderr.

    So when you span a process, the overall count of FDs should increase by
    three.

    Additionally, there are more FDs created if you chose to pipe communication
    between the child-process and the parent.

    This is a unix-thing btw, nothing to do with subprocess per se.

    Diez
     
    Diez B. Roggisch, Jan 20, 2009
    #2
    1. Advertising

  3. srinivasan srinivas

    Mark Wooding Guest

    srinivasan srinivas <> writes:

    > Does subprocess.Popen() count a new open file for each suprocess? I
    > mean does it occupy an entry in file descriptor table of parent
    > process? If so, wat is each file descriptor connected to?


    On Unix, subprocess.Popen will use up a file descriptor in the parent
    for each use of subprocess.PIPE. The descriptor in question is one end
    of a pipe; the child process holds the other end.

    I guess the situation is similar on Windows, but I don't know for sure.

    -- [mdw]
     
    Mark Wooding, Jan 20, 2009
    #3
  4. Do parent process will have different file descriptor in it for each subprocesses or paprent uses a single file descriptor for all?
    I really want to know creation of each subprocess will occupy an entry in parents'file descriptor table. B'cos if i create more than 200 subprocesses, i am getting 'Too many open files' error.

    Thanks,
    Srini
    ----- Original Message ----
    From: Mark Wooding <>
    To: python-list@python..org
    Sent: Tuesday, 20 January, 2009 6:16:17 PM
    Subject: Re: Doubts related to subprocess.Popen()

    srinivasan srinivas <> writes:

    > Does subprocess.Popen() count a new open file for each suprocess? I
    > mean does it occupy an entry in file descriptor table of parent
    > process?  If so, wat is each file descriptor connected to?


    On Unix, subprocess.Popen will use up a file descriptor in the parent
    for each use of subprocess.PIPE.  The descriptor in question is one end
    of a pipe; the child process holds the other end.

    I guess the situation is similar on Windows, but I don't know for sure.

    -- [mdw]
    --
    http://mail.python.org/mailman/listinfo/python-list



    Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
     
    srinivasan srinivas, Jan 20, 2009
    #4
  5. On Jan 20, 2009, at 9:19 AM, srinivasan srinivas wrote:

    > Do parent process will have different file descriptor in it for each
    > subprocesses or paprent uses a single file descriptor for all?
    > I really want to know creation of each subprocess will occupy an
    > entry in parents'file descriptor table. B'cos if i create more than
    > 200 subprocesses, i am getting 'Too many open files' error.


    If you're on Unix, a command like lsof can be useful at a time like
    this.

    lsof = "ls open files"

    If you're on Linux, there's the /proc tree to investigate.





    >
    > ----- Original Message ----
    > From: Mark Wooding <>
    > To:
    > Sent: Tuesday, 20 January, 2009 6:16:17 PM
    > Subject: Re: Doubts related to subprocess.Popen()
    >
    > srinivasan srinivas <> writes:
    >
    >> Does subprocess.Popen() count a new open file for each suprocess? I
    >> mean does it occupy an entry in file descriptor table of parent
    >> process? If so, wat is each file descriptor connected to?

    >
    > On Unix, subprocess.Popen will use up a file descriptor in the parent
    > for each use of subprocess.PIPE. The descriptor in question is one
    > end
    > of a pipe; the child process holds the other end.
    >
    > I guess the situation is similar on Windows, but I don't know for
    > sure.
    >
    > -- [mdw]
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
    >
    >
    > Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
    > --
    > http://mail.python.org/mailman/listinfo/python-list
     
    Philip Semanchuk, Jan 20, 2009
    #5
  6. srinivasan srinivas

    Jeff McNeil Guest

    On Jan 20, 9:19 am, srinivasan srinivas <>
    wrote:
    > Do parent process will have different file descriptor in it for each subprocesses or paprent uses a single file descriptor for all?
    > I really want to know creation of each subprocess will occupy an entry in parents'file descriptor table. B'cos if i create more than 200 subprocesses, i am getting 'Too many open files' error.
    >
    > Thanks,
    > Srini
    >
    > ----- Original Message ----
    > From: Mark Wooding <>
    > To:
    > Sent: Tuesday, 20 January, 2009 6:16:17 PM
    > Subject: Re: Doubts related to subprocess.Popen()
    >
    > srinivasan srinivas <> writes:
    >
    > > Does subprocess.Popen() count a new open file for each suprocess? I
    > > mean does it occupy an entry in file descriptor table of parent
    > > process? If so, wat is each file descriptor connected to?

    >
    > On Unix, subprocess.Popen will use up a file descriptor in the parent
    > for each use of subprocess.PIPE. The descriptor in question is one end
    > of a pipe; the child process holds the other end.
    >
    > I guess the situation is similar on Windows, but I don't know for sure.
    >
    > -- [mdw]
    > --http://mail.python.org/mailman/listinfo/python-list
    >
    > Add more friends to your messenger and enjoy! Go tohttp://messenger.yahoo.com/invite/


    Have you upped your open files limit? My test script:

    import subprocess

    procs = []
    for i in xrange(400):
    procs.append(subprocess.Popen("/bin/cat",
    stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    stderr=subprocess.PIPE))

    By default, 'ulimit -n' returns 1024, which is the number of open
    files allowed. Running the test script without changing it results
    in:

    Traceback (most recent call last):
    File "test.py", line 9, in <module>
    stderr=subprocess.PIPE))
    File "/usr/lib/python2.5/subprocess.py", line 593, in __init__
    errread, errwrite)
    File "/usr/lib/python2.5/subprocess.py", line 1002, in
    _execute_child
    errpipe_read, errpipe_write = os.pipe()
    OSError: [Errno 24] Too many open files

    Now, up that limit to 8192 via ulimit -n, and run the script again:

    [root@marvin jeff]# ulimit -n 8192
    [root@marvin jeff]# python test.py
    [root@marvin jeff]#

    HTH,

    Jeff
     
    Jeff McNeil, Jan 20, 2009
    #6
  7. srinivasan srinivas

    Mark Wooding Guest

    "Diez B. Roggisch" <> writes:

    > Usually, each new process has three file-descriptors associated with
    > it - stdin,stdout and stderr.
    >
    > So when you span a process, the overall count of FDs should increase
    > by three.


    Yes, but that's irrelevant. There are two file limits which are
    relevant:

    * the per-process limit on file descriptors -- basically, the largest
    number which can be a file descriptor -- and

    * the overall number of files which can be open at a time.

    Note that, in Unix, the same file can be referred to by many different
    descriptors in many different processes, and fork(2), exec(2) and dup(2)
    don't change the number of files open. However, dup(2) allocates a new
    descriptor in the calling process, so it may hit the per-process limit.

    > Additionally, there are more FDs created if you chose to pipe
    > communication between the child-process and the parent.


    And these are the ones I mentioned.

    OK, in more detail: each pipe(2) call allocates two files (one for each
    end) and two file descriptors (one for each file). If you call Popen
    with PIPE specified for each of stdin, stdout and stderr, then that's a
    total of six descriptors and six files. But the parent will close half
    of them after calling fork(2) (freeing three descriptors), and the child
    will close all six after dup2(2)-ing them over the descriptors 0, 1, and
    2. The net result is:

    * six new files in the global file table, and
    * three new descriptors in the parent.

    (The child ends up with no new descriptors at the end of all this.)

    -- [mdw]
     
    Mark Wooding, Jan 20, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michele Simionato

    subprocess.Popen

    Michele Simionato, Dec 12, 2004, in forum: Python
    Replies:
    5
    Views:
    17,700
  2. Ivo Woltring
    Replies:
    4
    Views:
    3,576
    Jeff Shannon
    Jan 27, 2005
  3. Ameya

    doubts related to disk files in C

    Ameya, Nov 29, 2004, in forum: C Programming
    Replies:
    3
    Views:
    410
    Randy Howard
    Nov 30, 2004
  4. somenath

    doubts related to structure

    somenath, Feb 16, 2007, in forum: C Programming
    Replies:
    4
    Views:
    362
    santosh
    Feb 16, 2007
  5. File.popen/IO.popen

    , May 20, 2006, in forum: Ruby
    Replies:
    1
    Views:
    244
    Robert Klemme
    May 20, 2006
Loading...

Share This Page