Questions about GIL and web services from a n00b

Discussion in 'Python' started by Chris H, Apr 15, 2011.

  1. Chris H

    Chris H Guest

    So I'm in a startup where we are considering using python as our primary
    development language for all the wonderful reasons you would expect.
    However, I've had a couple of things come up from mentors and other
    developers that is causing me to rethink whether python is the right
    choice. I hope this is the right list for this type of discussion
    (please forgive me if not and point me in the right direction).

    We are looking to build an e-commerce integration product so the
    majority of our work will be linking external SOAP and REST based web
    service API's to our own REST based API and backend database. I have
    had the following comments/questions come to me:

    1. Are you sure you want to use python because threading is not good due
    to the Global Lock (GIL)? Is this really an issue for multi-threaded
    web services as seems to be indicated by the articles from a Google
    search? If not, how do you avoid this issue in a multi-threaded process
    to take advantage of all the CPU cores available?

    2. Are there good web services frameworks available for building a REST
    based service? I admit I have looked at web2py, Django, pyramid/pylons,
    and a few others. SOAP seems to be pretty well supported but I'm not
    finding the same for quick development of REST based services for
    exchanging JSON or XML formatted data. This is probably just my n00b
    status, but what tools are best for building a simple REST data exchange
    API?

    Thanks,
    Chris
    Chris H, Apr 15, 2011
    #1
    1. Advertising

  2. Chris H

    sturlamolden Guest

    On Apr 15, 6:33 pm, Chris H <> wrote:

    > 1. Are you sure you want to use python because threading is not good due
    > to the Global Lock (GIL)?  Is this really an issue for multi-threaded
    > web services as seems to be indicated by the articles from a Google
    > search?  If not, how do you avoid this issue in a multi-threaded process
    > to take advantage of all the CPU cores available?



    First, if you are stupid enough to to compute-bound work in Python,
    without using a library, you have worse problems than the GIL. How
    incompetent would you need to be to write multi-threaded matrix
    multiplication or FFTs in pure Python, and blame the GIL for
    lack for performance?

    Second, if you think "advantage of all the CPU cores available"
    will make difference for an I/O bound webservice, you're living
    in cloud cookoo land. How on earth will multiple CPU cores
    give you or your clients a faster network connection? The network
    connection is likely to saturate long before you're burning the
    CPU.

    Sturla
    sturlamolden, Apr 17, 2011
    #2
    1. Advertising

  3. Chris H

    sturlamolden Guest

    On Apr 15, 6:33 pm, Chris H <> wrote:

    > 1. Are you sure you want to use python because threading is not good due
    > to the Global Lock (GIL)?  Is this really an issue for multi-threaded
    > web services as seems to be indicated by the articles from a Google
    > search?  If not, how do you avoid this issue in a multi-threaded process
    > to take advantage of all the CPU cores available?



    By the way:

    The main issue with the GIL is all the FUD written by
    people who don't properly understand the issue. It's
    not easy to discern the real information from the
    unqualified FUD. I whish people who think in Java
    would just shut up about things they don't understand.
    The problem is they think they understand more than
    they do.

    Also, people that write multi-threaded programs which
    fails to scale from false-sharing issues, really should
    not pollute the web with FUD about Python's GIL. Java's
    "free threading" model is not better when all you use it
    for is to create dirty cache lines that must be reloaded
    everywhere.


    Sturla
    sturlamolden, Apr 17, 2011
    #3
  4. > 1. Are you sure you want to use python because threading is not
    good due
    > to the Global Lock (GIL)?  Is this really an issue for multi-threaded
    > web services as seems to be indicated by the articles from a Google
    > search?  If not, how do you avoid this issue in a multi-threaded process
    > to take advantage of all the CPU cores available?


    To take advantage of the cores on your server you'll want to consider
    a multi-process design instead of multi-threading. You can achieve
    this with something like http://projects.unbit.it/uwsgi/, which will
    allow you to manage the processes. I've used this behind apache
    successfully.

    >
    > 2. Are there good web services frameworks available for building a REST
    > based service?  I admit I have looked at web2py, Django, pyramid/pylons,
    > and a few others.  SOAP seems to be pretty well supported but I'm not
    > finding the same for quick development of REST based services for
    > exchanging JSON or XML formatted data.  This is probably just my n00b
    > status, but what tools are best for building a simple REST data exchange
    > API?


    I've personally used Pyramid to implement REST web services with
    multiple data transport formats (json, xml) for the same endpoints
    with minimal fuss. It's basically as simple as returning an object
    from your view and defining a renderer that knows how to translate
    this object to the desired format. Look at
    http://docs.pylonsproject.org/projects/pyramid/1.0/narr/renderers.html#views-which-use-a-renderer
    and http://docs.pylonsproject.org/projects/pyramid/1.0/narr/viewconfig.html#view-config-chapter
    for more information.

    Lamont
    Lamont Nelson, Apr 19, 2011
    #4
  5. Chris H

    Chris H Guest

    On 4/19/11 3:48 AM, Lamont Nelson wrote:
    > > 1. Are you sure you want to use python because threading is not

    > good due
    >> to the Global Lock (GIL)? Is this really an issue for multi-threaded
    >> web services as seems to be indicated by the articles from a Google
    >> search? If not, how do you avoid this issue in a multi-threaded process
    >> to take advantage of all the CPU cores available?

    > To take advantage of the cores on your server you'll want to consider
    > a multi-process design instead of multi-threading. You can achieve
    > this with something like http://projects.unbit.it/uwsgi/, which will
    > allow you to manage the processes. I've used this behind apache
    > successfully.
    >
    >> 2. Are there good web services frameworks available for building a REST
    >> based service? I admit I have looked at web2py, Django, pyramid/pylons,
    >> and a few others. SOAP seems to be pretty well supported but I'm not
    >> finding the same for quick development of REST based services for
    >> exchanging JSON or XML formatted data. This is probably just my n00b
    >> status, but what tools are best for building a simple REST data exchange
    >> API?

    > I've personally used Pyramid to implement REST web services with
    > multiple data transport formats (json, xml) for the same endpoints
    > with minimal fuss. It's basically as simple as returning an object
    > from your view and defining a renderer that knows how to translate
    > this object to the desired format. Look at
    > http://docs.pylonsproject.org/projects/pyramid/1.0/narr/renderers.html#views-which-use-a-renderer
    > and http://docs.pylonsproject.org/projects/pyramid/1.0/narr/viewconfig.html#view-config-chapter
    > for more information.
    >
    > Lamont


    Thanks to everyone who has replied. I have a better understanding of
    the limitations around the GIL as well as the relatively architectures
    for avoiding the issue. Now to find the right framework for a simple
    web service. So far I've heard pyramid the most, but others we are
    looking into include rest.ish.io, web2py, and flask. Anyone with
    experience across these as to what is best for someone starting from
    scratch now?

    Chris
    Chris H, Apr 20, 2011
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris H
    Replies:
    8
    Views:
    203
    sturlamolden
    Apr 17, 2011
  2. disappearedng

    n00b questions for javascript!

    disappearedng, Nov 12, 2008, in forum: Javascript
    Replies:
    61
    Views:
    425
    Dr J R Stockton
    Nov 24, 2008
  3. Chris Angelico

    Re: About GIL Questions!

    Chris Angelico, Jun 20, 2013, in forum: Python
    Replies:
    0
    Views:
    113
    Chris Angelico
    Jun 20, 2013
  4. Thanatos xiao

    About GIL Questions!

    Thanatos xiao, Jun 20, 2013, in forum: Python
    Replies:
    0
    Views:
    94
    Thanatos xiao
    Jun 20, 2013
  5. Joshua Landau

    Re: About GIL Questions!

    Joshua Landau, Jun 20, 2013, in forum: Python
    Replies:
    0
    Views:
    94
    Joshua Landau
    Jun 20, 2013
Loading...

Share This Page