high performance/threaded applications in Python - your experiences?

Discussion in 'Python' started by Jay Loden, Jun 23, 2007.

  1. Jay Loden

    Jay Loden Guest

    All,

    In studying Python, I have predictably run across quite a bit of talk about the GIL and threading in Python. As my day job, I work with a (mostly Java) application that is heavily threaded. As such our application takes good advantage of multiple processors and we can often scale through simply adding processing power to a server.

    I was hoping for some experiences that some of you on the list may have had in dealing with Python in a high performance and/or threaded environment. In essence, I'm wondering how big of a deal the GIL can be in a real-world scenario where you need to take advantage of multiple processor machines, thread pools, etc. How much does it get in the way (or not), and how difficult have you found it to architect applications for high performance? I have read a number of articles and opinions on whether or not the GIL is a good thing, and how it affects threaded performance on multiple processor machines, but what I haven't seen is experiences of people who have actually done it and reported back "it was a nightmare" or "it's no big deal" ;)

    Your thoughts and opinions are welcome, especially those with relevant experiences. Thanks!

    -Jay
     
    Jay Loden, Jun 23, 2007
    #1
    1. Advertising

  2. Jay Loden

    Ivan Voras Guest

    Jay Loden wrote:

    > I was hoping for some experiences that some of you on the list may have had in dealing with Python in a high performance and/or threaded environment. In essence, I'm wondering how big of a deal the GIL can be in a real-world scenario where you need to take advantage of multiple processor machines, thread pools, etc. How much does it get in the way (or not), and how difficult have you found it to architect applications for high performance? I have read a number of articles and opinions on whether or not the GIL is a good thing, and how it affects threaded performance on multiple processor machines, but what I haven't seen is experiences of people who have actually done it and reported back "it was a nightmare" or "it's no big deal" ;)


    The theory: If your threads mostly do IO, you can get decent CPU usage
    even with Python. If the threads are CPU-bound (e.g. you do a lot of
    computational work), you'll effectively only make use of one processor.

    In practice, I've noticed that Python applications don't scale very much
    across CPUs even if they're doing mostly IO. I blame cache trashing or
    similar effect caused by too many global synchronization events. I
    didn't measure but the speedup may even be negative with large-ish
    number of CPUs (>=4).

    OTOH, if you can get by with using forking instead of threads (given
    enough effort) you can achieve very good scaling.

    --
    (\__/)
    (o_O)
    (> < )

    This is Bunny.
    Copy Bunny into your signature to help him on his way to world domination!


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.4 (MingW32)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

    iD8DBQFGfTGhldnAQVacBcgRAqZaAKCG866O3Lg5DYH/fl9/Ig4EwclmSwCfXw6W
    99Is2/l13kbYq6P+IJBne+w=
    =x2s3
    -----END PGP SIGNATURE-----
     
    Ivan Voras, Jun 23, 2007
    #2
    1. Advertising

  3. Ivan Voras wrote:
    > Jay Loden wrote:
    >
    >> I was hoping for some experiences that some of you on the list may have had in dealing with Python in a high performance and/or threaded environment. In essence, I'm wondering how big of a deal the GIL can be in a real-world scenario where you need to take advantage of multiple processor machines, thread pools, etc. How much does it get in the way (or not), and how difficult have you found it to architect applications for high performance? I have read a number of articles and opinions on whether or not the GIL is a good thing, and how it affects threaded performance on multiple processor machines, but what I haven't seen is experiences of people who have actually done it and reported back "it was a nightmare" or "it's no big deal" ;)

    >
    > The theory: If your threads mostly do IO, you can get decent CPU usage
    > even with Python. If the threads are CPU-bound (e.g. you do a lot of
    > computational work), you'll effectively only make use of one processor.
    >
    > In practice, I've noticed that Python applications don't scale very much
    > across CPUs even if they're doing mostly IO. I blame cache trashing or
    > similar effect caused by too many global synchronization events. I
    > didn't measure but the speedup may even be negative with large-ish
    > number of CPUs (>=4).
    >
    > OTOH, if you can get by with using forking instead of threads (given
    > enough effort) you can achieve very good scaling.


    Also, see the 'processing' package in the Python cheeseshop. It allows
    you to use processes rather than threads with most of the same
    abstractions. I hear it recently acquired the ability to pass file
    handles between processes on the same machine :)

    - Josiah
     
    Josiah Carlson, Jun 24, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Walzer

    IDLE--what are your experiences with it?

    Kevin Walzer, Oct 22, 2004, in forum: Python
    Replies:
    6
    Views:
    363
    Ivo Woltring
    Oct 24, 2004
  2. Jon Harrop
    Replies:
    25
    Views:
    866
    kwikius
    Jun 27, 2007
  3. Jon Harrop
    Replies:
    1
    Views:
    328
    Richard Heathfield
    Jun 21, 2007
  4. OnTargetEmbedded

    VDC is Interested in Your Engineering Experiences

    OnTargetEmbedded, Mar 26, 2010, in forum: C Programming
    Replies:
    0
    Views:
    285
    OnTargetEmbedded
    Mar 26, 2010
  5. OnTargetEmbedded
    Replies:
    0
    Views:
    302
    OnTargetEmbedded
    Mar 26, 2010
Loading...

Share This Page