Re: submit jobs on multi-core

Discussion in 'Python' started by Oscar Benjamin, Sep 11, 2012.

  1. On 2012-09-11, Dhananjay <> wrote:
    > --===============0316394162==
    > Content-Type: multipart/alternative; boundary=20cf30776bd309ffd004c96557e2
    >
    > --20cf30776bd309ffd004c96557e2
    > Content-Type: text/plain; charset=ISO-8859-1
    >
    > Dear all,
    >
    > I have a python script in which I have a list of files to input one by one
    > and for each file I get a number as an output.
    > I used for loop to submit the file to script.
    > My script uses one file at a time and returns the output.
    >
    > My computers has 8 cores.
    > Is there any way that I could submit 8 jobs at a time and get all the
    > output faster ?
    > In other words, how can I modify my script so that I could submit 8 jobs
    > together on 8 different processors ?
    >
    > I am bit new to this stuff, please suggest me some directions.
    >
    > Thank you.


    The simplest way I've found to do this is to use something like GNU parallel.
    I don't know if there's a Windows equivalent but it works well for me on linux
    and you can use it for any program (not just python scripts).

    >From the wikipedia page:

    http://en.wikipedia.org/wiki/GNU_parallel

    """
    The most common usage is to replace the shell loop, for example

    (for x in `cat list` ; do
    do_something $x
    done) | process_output

    to the form of

    cat list | parallel do_something | process_output
    """

    Note that there are two basic types of parallel execution depending on whether
    or not your parallel processes need to communicate with one another. I'm
    assuming that you really just want to run independent jobs simultaneously.
    Otherwise the other suggestions may be more relevant.

    Oscar
    Oscar Benjamin, Sep 11, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. hardikh2000

    JOBs............JOBs............JOBs

    hardikh2000, Aug 16, 2005, in forum: Python
    Replies:
    0
    Views:
    411
    hardikh2000
    Aug 16, 2005
  2. nicky boy
    Replies:
    0
    Views:
    689
    nicky boy
    Feb 7, 2009
  3. nicky boy
    Replies:
    0
    Views:
    730
    nicky boy
    Feb 7, 2009
  4. Laszlo Nagy

    Re: submit jobs on multi-core

    Laszlo Nagy, Sep 11, 2012, in forum: Python
    Replies:
    1
    Views:
    188
    zig-zag
    Sep 11, 2012
  5. Jason Friedman

    Re: submit jobs on multi-core

    Jason Friedman, Sep 13, 2012, in forum: Python
    Replies:
    1
    Views:
    221
    Matej Cepl
    Sep 13, 2012
Loading...

Share This Page