Development for dual core machine

Discussion in 'Python' started by Andy, Aug 19, 2007.

  1. Andy

    Andy Guest

    Hi guys,

    I'm sorry, I'm not sure this is the correct group to be asking this
    kind of question...

    I'm going to develop a software package that includes a web server
    (and PHP code) , a database, and some Python code of course. I am
    being strongly suggested to make it to work on a dual- or multi-core
    computer, but I'm confused on how to take advantage of the multiple
    CPUs.

    >From what I read, I think that simply by making the package run in

    several separate processes (web server, database server, Python
    interpreter, etc.), and/or using multiple threads (which I will
    anyway) the package should be able to use multiple CPUs.

    Is my understanding correct? So actually, all I have to do is just
    write my multi-threaded Python code as usual? And how is it decided
    which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
    BIOS?

    Any advice greatly appreciated.
    Andy
     
    Andy, Aug 19, 2007
    #1
    1. Advertising

  2. Andy wrote:

    > I'm going to develop a software package that includes a web server
    > (and PHP code) , a database, and some Python code of course. I am
    > being strongly suggested to make it to work on a dual- or
    > multi-core computer,


    No problem. CPython will work on any dual core CPU.

    > but I'm confused on how to take advantage of the multiple CPUs.


    First: Use a web server that can make use of multiple cores.

    Second: Use a data base that can make use of multiple cores.

    Third, for using multiple cores from CPython: this is an FAQ, please
    look at the mailing list archives. Whether you should optimize your
    python application to use all cores strongly depends on what your
    Python application actually does.

    > From what I read, I think that simply by making the package run in
    > several separate processes (web server, database server, Python
    > interpreter, etc.), and/or using multiple threads (which I will
    > anyway) the package should be able to use multiple CPUs.


    CPython has the GIL (global interpreter lock). Please search for it
    in the archives, it's been discussed exhaustingly.

    > And how is it decided which process and which threads go to CPU 1,
    > CPU 2, etc.? Perhaps the BIOS?


    No, the kernel (i. e., Linux). The BIOS is completely out of this.

    Regards,


    Björn

    --
    BOFH excuse #388:

    Bad user karma.
     
    Bjoern Schliessmann, Aug 19, 2007
    #2
    1. Advertising

  3. Andy

    samwyse Guest

    Andy wrote:
    > Hi guys,
    >
    > I'm sorry, I'm not sure this is the correct group to be asking this
    > kind of question...
    >
    > I'm going to develop a software package that includes a web server
    > (and PHP code) , a database, and some Python code of course. I am
    > being strongly suggested to make it to work on a dual- or multi-core
    > computer, but I'm confused on how to take advantage of the multiple
    > CPUs.
    >
    >>From what I read, I think that simply by making the package run in

    > several separate processes (web server, database server, Python
    > interpreter, etc.), and/or using multiple threads (which I will
    > anyway) the package should be able to use multiple CPUs.
    >
    > Is my understanding correct? So actually, all I have to do is just
    > write my multi-threaded Python code as usual? And how is it decided
    > which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
    > BIOS?
    >
    > Any advice greatly appreciated.
    > Andy
    >


    The Python interpreter is not multi-cpu aware, so using Python threads
    won't work on multiple CPUs. If your tasks are CPU-bound, then fork
    multiple processes. Most web servers (Apache) can handle this
    automatically for you.
     
    samwyse, Aug 19, 2007
    #3
  4. Andy

    Bryan Olson Guest

    Andy wrote:
    > I'm going to develop a software package that includes a web server
    > (and PHP code) , a database, and some Python code of course. I am
    > being strongly suggested to make it to work on a dual- or multi-core
    > computer, but I'm confused on how to take advantage of the multiple
    > CPUs.
    >
    > From what I read, I think that simply by making the package run in
    > several separate processes (web server, database server, Python
    > interpreter, etc.), and/or using multiple threads (which I will
    > anyway) the package should be able to use multiple CPUs.


    Right. There is a theoretical possibility that Python's GIL
    could limit thread parallelism, but in this kind of application
    it should not matter in the least. Either you're I/O bound, or
    there are plenty of tasks the operating system could schedule.

    If you have the option to run the Python interpreter within a
    web-server process, that's usually a performance winner.

    > Is my understanding correct? So actually, all I have to do is just
    > write my multi-threaded Python code as usual? And how is it decided
    > which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
    > BIOS?


    The O.S. actually. A lot of really smart people have put a whole
    lot of work into making the O.S. do that well.

    If you usually write your Python apps multi-threaded, as I do,
    that's fine. Multi-core efficiency has only a little to do with
    it. Web service are, for the most part, intrinsically
    parallelizable: run multiple web servers, and multiple Python
    interpreters serving multiple connections.

    The only hard part is shared data. Scalability is all about
    the database.


    --
    --Bryan
     
    Bryan Olson, Aug 19, 2007
    #4
  5. Andy

    Paul Rubin Guest

    Andy <> writes:
    > >From what I read, I think that simply by making the package run in

    > several separate processes (web server, database server, Python
    > interpreter, etc.), and/or using multiple threads (which I will
    > anyway) the package should be able to use multiple CPUs.


    Python threading doesn't support multiple CPU's because of the GIL.

    One approach to using your multiple cores is to embed a Python
    interpreter into a web server that runs in multiple processes,
    e.g. mod_python under Apache.

    Another is write your application as a separate process (e.g. as an
    FastCGI), then run multiple instances of it, connected to a concurrent
    web server. For what I'm currently doing, we're using lighthttpd as
    the http server and flup to connect to a set of Python FCGI's.

    For that matter, if your machine is just dual core, maybe it's ok to
    just run a single Python process, figuring that will run on one core
    and your database/httpd will run on the other one. However, you
    should figure the dual core situation won't last. Dual socket
    motherboards are fairly cheap these days, so we have a number of
    4-core machines (two AMD dual core cpu's); Intel is already shipping
    quad core cpu's, so that puts 8 cores in a dual socket board (you can
    already buy Macintoshes configured that way). Higher end server
    motherboards have 4 sockets, so you have to expect that 16-core
    servers will be common pretty soon.

    So if you're working on a cpu-intensive application it's worth your
    while figuring out how to parallelize it.

    Note that the most careful concurrency stuff probably is in the
    database. Serious ones already know how to use multiple CPU's.
     
    Paul Rubin, Aug 19, 2007
    #5
  6. samwyse wrote:
    > The Python interpreter is not multi-cpu aware, so using Python
    > threads won't work on multiple CPUs.


    Are you sure about this?

    Regards,


    Björn

    --
    BOFH excuse #12:

    dry joints on cable plug
     
    Bjoern Schliessmann, Aug 19, 2007
    #6
  7. Paul Rubin wrote:

    > Python threading doesn't support multiple CPU's because of the
    > GIL.


    :s/support/take full advantage of/

    Regards,


    Björn

    --
    BOFH excuse #46:

    waste water tank overflowed onto computer
     
    Bjoern Schliessmann, Aug 19, 2007
    #7
  8. Andy

    Andy Guest

    Thanks guys for the suggestions.

    Andy
     
    Andy, Aug 20, 2007
    #8
  9. Andy wrote:

    > Thanks guys for the suggestions.
    >
    > Andy


    It might be that you have to set the CPU affinity for your python process so
    that it works at all. I had that problem on a dual core machine with
    hyperthread enabled. Using taskset
    (http://www.linuxcommand.org/man_pages/taskset1.html) helped solving the
    problem:

    taskset -c <CPU#> python ...

    Best regards,
    wr


    --
    pinkrose.dhis.org,
     
    Willi Richert, Aug 20, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andy
    Replies:
    5
    Views:
    2,842
  2. malv

    Dual Core outlook

    malv, Feb 7, 2006, in forum: Python
    Replies:
    6
    Views:
    378
  3. robert
    Replies:
    2
    Views:
    480
  4. Simon Wittber

    Help me use my Dual Core CPU!

    Simon Wittber, Sep 12, 2006, in forum: Python
    Replies:
    23
    Views:
    776
    mystilleef
    Sep 18, 2006
  5. Tim Golden

    RE: Help me use my Dual Core CPU!

    Tim Golden, Sep 12, 2006, in forum: Python
    Replies:
    13
    Views:
    539
    Michael
    Oct 1, 2006
Loading...

Share This Page