How to schedule system calls with Python

[Jeremy]

I need to write a Python script that will call some command line
programs (using os.system). I will have many such calls, but I want
to control when the calls are made. I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors. I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.

How can I use Python to schedule these commands?

Thanks,
Jeremy

 

[TerryP]

I need to write a Python script that will call some command line
programs (using os.system).  I will have many such calls, but I want
to control when the calls are made.  I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors.  I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.

How can I use Python to schedule these commands?

Thanks,
Jeremy

External programs are not system calls; external programs are invoked
through system calls; for example system() is a function call which
when implemented under UNIX systems invokes some form of fork() and
exec(), and likely spawn() under Windows NT.

If you want simple sequenceal execution of external programs, use a
suitable blocking function to execute them (like system) combined with
a simple loop over the sequence of commands to run.


for prog in ['cmd1', 'cmd2', 'cmd3']:
os.system(prog)

blah.



For anything more detailed (or complex) in response, try being more
detailed yourself .

 

[Jeremy]

I need to write a Python script that will call some command line
programs (using os.system).  I will have many such calls, but I want
to control when the calls are made.  I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors.  I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.

How can I use Python to schedule these commands?

Thanks,
Jeremy

External programs are not system calls; external programs are invoked
through system calls; for example system() is a function call which
when implemented under UNIX systems invokes some form of fork() and
exec(), and likely spawn() under Windows NT.

If you want simple sequenceal execution of external programs, use a
suitable blocking function to execute them (like system) combined with
a simple loop over the sequence of commands to run.

for prog in ['cmd1', 'cmd2', 'cmd3']:
    os.system(prog)

blah.

For anything more detailed (or complex) in response, try being more
detailed yourself .

This is the solution I wanted. I thought that os.system(prog) would
return immediately regardless of how long prog takes to run. I should
have tried this simple solution first. Thanks for being patient.

Jeremy

 

[TerryP]

This is the solution I wanted.  I thought that os.system(prog) would
return immediately regardless of how long prog takes to run.  I should
have tried this simple solution first.  Thanks for being patient.

Jeremy

launching external programs, irregardless of language, generally falls
into 3 major categories:

0.) blocks until program is done; like system
1.) replaces your program with process, never returns; like exec
2.) quickly return after asynchronously launching the program

Most languages will implement the first method because of the standard
system() function in C, which is fairly popular in it's own right.
Most multi-tasking operating systems will implement some form of exec
function, which Python exposes through the os module. The last method
is the least portable, because obviously if the OS lacks multi-tasking
you're screwed. The best examples of 2. are the UNIX popen() function
and Microsoft's spawn() family, when used with the P_DETACH flag.
Python being made with much loving kindless, exposes each interface.
More system specific functions are exposed through the os module in a
fairly portable way in so much as is actually possible under Unix and
Windows.


The standard subprocess module provides a portable Popen class and a
few convenience functions for all occasions - including good
documentation on how to use it to establish the common goals most
people will ever have, when wishing to invoke an external program. I
personally suggest learning how to use the subprocess module, unless
you may as well be programming in the C or Lisp family


Vive la différence!

 

[MRAB]

I need to write a Python script that will call some command line
programs (using os.system). I will have many such calls, but I want
to control when the calls are made. I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors. I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.

How can I use Python to schedule these commands?

Thanks,
Jeremy

External programs are not system calls; external programs are invoked
through system calls; for example system() is a function call which
when implemented under UNIX systems invokes some form of fork() and
exec(), and likely spawn() under Windows NT.

If you want simple sequenceal execution of external programs, use a
suitable blocking function to execute them (like system) combined with
a simple loop over the sequence of commands to run.


for prog in ['cmd1', 'cmd2', 'cmd3']:
os.system(prog)

blah.



For anything more detailed (or complex) in response, try being more
detailed yourself .

You could use multithreading: put the commands into a queue; start the
same number of worker threads as there are processors; each worker
thread repeatedly gets a command from the queue and then runs it using
os.system(); if a worker thread finds that the queue is empty when it
tries to get a command, then it terminates.

 

[Ishwor Gurung]

Jeremy,
Hi

I need to write a Python script that will call some command line
programs (using os.system).  I will have many such calls, but I want
to control when the calls are made.  I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors.  I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.
Right.

How can I use Python to schedule these commands?

If I were as lucky as you, I would have used multiprocessing module[1]
(my platform does not have sem_open() syscall). Others suggestions are
as good as it can be but yeah you could get a lot of work done using
multiprocessing module(all the relevant bits are explained in the
module doc).

[1] http://docs.python.org/library/multiprocessing.html

 

[Ishwor Gurung]

How can I use Python to schedule these commands?

If I were as lucky as you, I would have used multiprocessing module[1]
(my platform does not have sem_open() syscall). Others suggestions are

s/have/implement/g

 

[Ishwor Gurung]

How can I use Python to schedule these commands?

If I were as lucky as you, I would have used multiprocessing module[1]
(my platform does not have sem_open() syscall). Others suggestions are
as good as it can be but yeah you could get a lot of work done using
multiprocessing module(all the relevant bits are explained in the
module doc).

My bad, I assumed Win32. Make sure _your_ platform also supports
sem_open syscall before you used the core feature of multiprocessing
module. >.<

 

[Jeremy]

External programs are not system calls; external programs are invoked
through system calls; for example system() is a function call which
when implemented under UNIX systems invokes some form of fork() and
exec(), and likely spawn() under Windows NT.

If you want simple sequenceal execution of external programs, use a
suitable blocking function to execute them (like system) combined with
a simple loop over the sequence of commands to run.

for prog in ['cmd1', 'cmd2', 'cmd3']:
    os.system(prog)

blah.

For anything more detailed (or complex) in response, try being more
detailed yourself .

You could use multithreading: put the commands into a queue; start the
same number of worker threads as there are processors; each worker
thread repeatedly gets a command from the queue and then runs it using
os.system(); if a worker thread finds that the queue is empty when it
tries to get a command, then it terminates.

Yes, this is it. If I have a list of strings which are system
commands, this seems like a more intelligent way of approaching it.
My previous response will work, but won't take advantage of multiple
cpus/cores in a machine without some manual manipulation. I like this
idea.

Thanks!
Jeremy

 

[Jeremy]

Jeremy,
Hi

I need to write a Python script that will call some command line
programs (using os.system).  I will have many such calls, but I want
to control when the calls are made.  I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors.  I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.
Right.

How can I use Python to schedule these commands?

If I were as lucky as you, I would have used multiprocessing module[1]
(my platform does not have sem_open() syscall). Others suggestions are
as good as it can be but yeah you could get a lot of work done using
multiprocessing module(all the relevant bits are explained in the
module doc).

[1]http://docs.python.org/library/multiprocessing.html

Again another great suggestion. I was not aware of the
multiprocessing module, and I'm not (yet) sure if I understand why I
should use instead of multithreading as explained by a previous post.

Jeremy

 

[Minesh Patel]

Again another great suggestion.  I was not aware of the

multiprocessing module, and I'm not (yet) sure if I understand why I
should use instead of multithreading as explained by a previous post.

http://docs.python.org/library/multiprocessing.html

First paragraph...

 

[catalinfest]

And if you have this commands :alias ,ll ?
This commands not working , see "sh: ll: command not found"
and for "watch" and "top" need to use something to out from loops .

I need to write a Python script that will call some command line
programs (using os.system).  I will have many such calls, but I want
to control when the calls are made.  I won't know in advance how long
each program will run and I don't want to have 10 programs running
when I only have one or two processors.  I want to run one at a time
(or two if I have two processors), wait until it's finished, and then
call the next one.

How can I use Python to schedule these commands?

Thanks,
Jeremy

External programs are not system calls; external programs are invoked
through system calls; for example system() is a function call which
when implemented under UNIX systems invokes some form of fork() and
exec(), and likely spawn() under Windows NT.

If you want simple sequenceal execution of external programs, use a
suitable blocking function to execute them (like system) combined with
a simple loop over the sequence of commands to run.

for prog in ['cmd1', 'cmd2', 'cmd3']:
    os.system(prog)

blah.

For anything more detailed (or complex) in response, try being more
detailed yourself .

 

[Jorgen Grahn]

.

launching external programs, irregardless of language, generally falls
into 3 major categories:

0.) blocks until program is done; like system
1.) replaces your program with process, never returns; like exec
2.) quickly return after asynchronously launching the program

Most languages will implement the first method because of the standard
system() function in C, which is fairly popular in it's own right.
Most multi-tasking operating systems will implement some form of exec
function, which Python exposes through the os module. The last method
is the least portable, because obviously if the OS lacks multi-tasking
you're screwed. The best examples of 2. are the UNIX popen() function
and Microsoft's spawn() family, when used with the P_DETACH flag.

Not sure that popen() fits nicely into that category -- you have to
eat the child's output or feed it with input, or it will eventually
stall.

Python being made with much loving kindless, exposes each interface.

Nicely put!

/Jorgen

 

[Jorgen Grahn]

Yes, this is it. If I have a list of strings which are system
commands, this seems like a more intelligent way of approaching it.
My previous response will work, but won't take advantage of multiple
cpus/cores in a machine without some manual manipulation. I like this
idea.

Note that you do not need *need* multithreading for this. To me it
seems silly to have N threads sitting just waiting for one process
each to die -- those threads contribute nothing to the multiprocessing
you want.

In Unix, you can have one process fork() and exec() as many programs
as you like, have them run on whatever CPUs you have, and wait for
them to die and reap them using wait() and related calls. (Not sure
what the equivalent is in non-Unix OSes or portable Python.)

/Jorgen

 

[Jorgen Grahn]

Note that you do not need *need* multithreading for this. To me it
seems silly to have N threads sitting just waiting for one process
each to die -- those threads contribute nothing to the multiprocessing
you want.

....

Another way to approach this, if you do want to use threads, is to use a
counting semaphore. Set it to the maximum number of threads you want to
run at any one time. Then loop starting up worker threads in the main
thread. acquire() the semaphore before starting the next worker thread;
when the semaphore reaches 0, your main thread will block. Each worker
thread should then release() the semaphore when it exits; this will
allow the main thread to move on to creating the next worker thread.

This doesn't address the assignment of threads to CPU cores, but I have
used this technique many times, and it is simple and fairly easy to
implement. [---]

But do you *want* to handle the CPUs manually, anyway? Your program
typically has no idea what other processes are running on the machine
or how important they are.

(Of course, in this case the threads do next to nothing, so controlling
them on that detailed level neither helps nor hurts performance.)

/Jorgen