Doubts related to subprocess.Popen()

[srinivasan srinivas]

Hi,
Does subprocess.Popen() count a new open file for each suprocess? I mean does it occupy an entry in file descriptor table of parent process?
If so, wat is each file descriptor connected to?

Thanks,
Srini


Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

 

[Diez B. Roggisch]

Hi,
Does subprocess.Popen() count a new open file for each suprocess? I mean
does it occupy an entry in file descriptor table of parent process? If so,
wat is each file descriptor connected to?

Usually, each new process has three file-descriptors associated with it -
stdin,stdout and stderr.

So when you span a process, the overall count of FDs should increase by
three.

Additionally, there are more FDs created if you chose to pipe communication
between the child-process and the parent.

This is a unix-thing btw, nothing to do with subprocess per se.

Diez

 

[Mark Wooding]

Does subprocess.Popen() count a new open file for each suprocess? I
mean does it occupy an entry in file descriptor table of parent
process? If so, wat is each file descriptor connected to?

On Unix, subprocess.Popen will use up a file descriptor in the parent
for each use of subprocess.PIPE. The descriptor in question is one end
of a pipe; the child process holds the other end.

I guess the situation is similar on Windows, but I don't know for sure.

-- [mdw]

 

[srinivasan srinivas]

Do parent process will have different file descriptor in it for each subprocesses or paprent uses a single file descriptor for all?
I really want to know creation of each subprocess will occupy an entry in parents'file descriptor table. B'cos if i create more than 200 subprocesses, i am getting 'Too many open files' error.

Thanks,
Srini
----- Original Message ----
From: Mark Wooding <[email protected]>
To: [email protected]
Sent: Tuesday, 20 January, 2009 6:16:17 PM
Subject: Re: Doubts related to subprocess.Popen()

Does subprocess.Popen() count a new open file for each suprocess? I
mean does it occupy an entry in file descriptor table of parent
process?  If so, wat is each file descriptor connected to?

On Unix, subprocess.Popen will use up a file descriptor in the parent
for each use of subprocess.PIPE.  The descriptor in question is one end
of a pipe; the child process holds the other end.

I guess the situation is similar on Windows, but I don't know for sure.

-- [mdw]
--
http://mail.python.org/mailman/listinfo/python-list



Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

 

[Philip Semanchuk]

Do parent process will have different file descriptor in it for each
subprocesses or paprent uses a single file descriptor for all?
I really want to know creation of each subprocess will occupy an
entry in parents'file descriptor table. B'cos if i create more than
200 subprocesses, i am getting 'Too many open files' error.

If you're on Unix, a command like lsof can be useful at a time like
this.

lsof = "ls open files"

If you're on Linux, there's the /proc tree to investigate.

----- Original Message ----
From: Mark Wooding <[email protected]>
To: (e-mail address removed)
Sent: Tuesday, 20 January, 2009 6:16:17 PM
Subject: Re: Doubts related to subprocess.Popen()

Does subprocess.Popen() count a new open file for each suprocess? I
mean does it occupy an entry in file descriptor table of parent
process? If so, wat is each file descriptor connected to?

On Unix, subprocess.Popen will use up a file descriptor in the parent
for each use of subprocess.PIPE. The descriptor in question is one
end
of a pipe; the child process holds the other end.

I guess the situation is similar on Windows, but I don't know for
sure.

-- [mdw]
--
http://mail.python.org/mailman/listinfo/python-list



Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

 

[Jeff McNeil]

Do parent process will have different file descriptor in it for each subprocesses or paprent uses a single file descriptor for all?
I really want to know creation of each subprocess will occupy an entry in parents'file descriptor table. B'cos if i create more than 200 subprocesses, i am getting 'Too many open files' error.

Thanks,
Srini

----- Original Message ----
From: Mark Wooding <[email protected]>
To: (e-mail address removed)
Sent: Tuesday, 20 January, 2009 6:16:17 PM
Subject: Re: Doubts related to subprocess.Popen()

Does subprocess.Popen() count a new open file for each suprocess? I
mean does it occupy an entry in file descriptor table of parent
process? If so, wat is each file descriptor connected to?

On Unix, subprocess.Popen will use up a file descriptor in the parent
for each use of subprocess.PIPE. The descriptor in question is one end
of a pipe; the child process holds the other end.

I guess the situation is similar on Windows, but I don't know for sure.

-- [mdw]
--http://mail.python.org/mailman/listinfo/python-list

Add more friends to your messenger and enjoy! Go tohttp://messenger.yahoo.com/invite/

Have you upped your open files limit? My test script:

import subprocess

procs = []
for i in xrange(400):
procs.append(subprocess.Popen("/bin/cat",
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE))

By default, 'ulimit -n' returns 1024, which is the number of open
files allowed. Running the test script without changing it results
in:

Traceback (most recent call last):
File "test.py", line 9, in <module>
stderr=subprocess.PIPE))
File "/usr/lib/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib/python2.5/subprocess.py", line 1002, in
_execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

Now, up that limit to 8192 via ulimit -n, and run the script again:

[root@marvin jeff]# ulimit -n 8192
[root@marvin jeff]# python test.py
[root@marvin jeff]#

HTH,

Jeff

 

[Mark Wooding]

Usually, each new process has three file-descriptors associated with
it - stdin,stdout and stderr.

So when you span a process, the overall count of FDs should increase
by three.

Yes, but that's irrelevant. There are two file limits which are
relevant:

* the per-process limit on file descriptors -- basically, the largest
number which can be a file descriptor -- and

* the overall number of files which can be open at a time.

Note that, in Unix, the same file can be referred to by many different
descriptors in many different processes, and fork(2), exec(2) and dup(2)
don't change the number of files open. However, dup(2) allocates a new
descriptor in the calling process, so it may hit the per-process limit.

Additionally, there are more FDs created if you chose to pipe
communication between the child-process and the parent.

And these are the ones I mentioned.

OK, in more detail: each pipe(2) call allocates two files (one for each
end) and two file descriptors (one for each file). If you call Popen
with PIPE specified for each of stdin, stdout and stderr, then that's a
total of six descriptors and six files. But the parent will close half
of them after calling fork(2) (freeing three descriptors), and the child
will close all six after dup2(2)-ing them over the descriptors 0, 1, and
2. The net result is:

* six new files in the global file table, and
* three new descriptors in the parent.

(The child ends up with no new descriptors at the end of all this.)

-- [mdw]