Re: Threads not Improving Performance in Program

Discussion in 'Python' started by Jean-Paul Calderone, Mar 19, 2009.

  1. On Thu, 19 Mar 2009 09:50:51 -0700, Ryan Rosario <> wrote:
    >I have a parser that needs to process 7 million files. After running
    >for 2 days, it had only processed 1.5 million. I want this script to
    >parse several files at once by using multiple threads: one for each
    >file currently being analyzed.


    Threads don't magically make a program faster. In Python in particular,
    threads won't do much, if anything, to speed up a CPU bound task. Assuming
    you have more than one CPU, Python still limits you to one thread executing
    Python bytecode at a time. You could try running multiple processes instead,
    if there really is more hardware that's sitting idle with your single
    threaded version.

    Jean-Paul
     
    Jean-Paul Calderone, Mar 19, 2009
    #1
    1. Advertising

  2. Jean-Paul Calderone

    Ryan Rosario Guest

    On Mar 19, 10:35 am, Jean-Paul Calderone <> wrote:
    > On Thu, 19 Mar 2009 09:50:51 -0700, Ryan Rosario <> wrote:
    > >I have a parser that needs to process 7 million files. After running
    > >for 2 days, it had only processed 1.5 million. I want this script to
    > >parse several files at once by using multiple threads: one for each
    > >file currently being analyzed.

    >
    > Threads don't magically make a program faster.  In Python in particular,
    > threads won't do much, if anything, to speed up a CPU bound task.  Assuming
    > you have more than one CPU, Python still limits you to one thread executing
    > Python bytecode at a time.  You could try running multiple processes instead,
    > if there really is more hardware that's sitting idle with your single
    > threaded version.
    >
    > Jean-Paul


    Thank you for your response. I did not realize that. That seems like a
    huge limitation for such a great language.
    I will look into forking off processes instead of using threads.

    R.
     
    Ryan Rosario, Mar 19, 2009
    #2
    1. Advertising

  3. Jean-Paul Calderone

    Tim Rowe Guest

    > Thank you for your response. I did not realize that. That seems like a
    > huge limitation for such a great language.
    > I will look into forking off processes instead of using threads.


    If that's what you need to do, yes it is. If it isn't, no it's not. No
    language is optimum for all possible applications, each one has
    different compromises. You've just discovered one of Python's.

    --
    Tim Rowe
     
    Tim Rowe, Mar 19, 2009
    #3
  4. On Mar 20, 4:21 am, Tim Rowe <> wrote:
    > > Thank you for your response. I did not realize that. That seems like a
    > > huge limitation for such a great language.
    > > I will look into forking off processes instead of using threads.

    >
    > If that's what you need to do, yes it is. If it isn't, no it's not. No
    > language is optimum for all possible applications, each one has
    > different compromises. You've just discovered one of Python's.
    >
    > --
    > Tim Rowe


    try this: http://docs.python.org/library/multiprocessing.html#module-multiprocessing

    - Vijayendra
     
    Vijayendra Bapte, Mar 20, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robin

    Improving Performance in ASP.Net

    Robin, Mar 4, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    415
    Scott Allen
    Mar 4, 2005
  2. ruds
    Replies:
    13
    Views:
    589
    squirrel
    Apr 8, 2007
  3. Dale
    Replies:
    2
    Views:
    782
    Mike Meyer
    Jan 11, 2008
  4. yawnmoth
    Replies:
    6
    Views:
    1,543
    yawnmoth
    Aug 5, 2008
  5. Ryan Rosario
    Replies:
    1
    Views:
    243
    odeits
    Mar 19, 2009
Loading...

Share This Page