python simply not scaleable enough for google?

Discussion in 'Python' started by Robert P. J. Day, Nov 11, 2009.

  1. Robert P. J. Day, Nov 11, 2009
    #1
    1. Advertising

  2. Robert P. J. Day

    samwyse Guest

    samwyse, Nov 11, 2009
    #2
    1. Advertising

  3. Robert P. J. Day

    Paul Rubin Guest

    "Robert P. J. Day" <> writes:
    > http://groups.google.com/group/unladen-swallow/browse_thread/thread/4edbc406f544643e?pli=1
    > thoughts?


    I'd bet it's not just about multicore scaling and general efficiency,
    but also the suitability of the language itself for large, complex
    projects. It's just not possible to be everything for everybody.
    Python is beginner-friendly, has a very fast learning curve for
    experienced programmers in other languages, and is highly productive
    for throwing small and medium sized scripts together, that are
    debugged through iterated testing. One might say it's optimized for
    those purposes. I use it all the time because a lot of my programming
    fits the pattern. The car analogy is the no-frills electric commuter
    car, just hop in and aim it where you want to go; if you crash it,
    brush yourself off and restart. But there are times (large production
    applications) when you really want the Airbus A380 with the 100's of
    automatic monitoring systems and checkout procedures to follow before
    you take off, even if the skill level needed to use it is much higher
    than the commuter car.
     
    Paul Rubin, Nov 14, 2009
    #3
  4. On 2009-11-13, at 19:53, Paul Rubin wrote:

    > "Robert P. J. Day" <> writes:
    >> http://groups.google.com/group/unladen-swallow/browse_thread/thread/4edbc406f544643e?pli=1
    >> thoughts?

    >
    > I'd bet it's not just about multicore scaling and general efficiency,
    > but also the suitability of the language itself for large, complex
    > projects. It's just not possible to be everything for everybody.
    > Python is beginner-friendly, has a very fast learning curve for
    > experienced programmers in other languages, and is highly productive
    > for throwing small and medium sized scripts together, that are
    > debugged through iterated testing. One might say it's optimized for
    > those purposes. I use it all the time because a lot of my programming
    > fits the pattern. The car analogy is the no-frills electric commuter
    > car, just hop in and aim it where you want to go; if you crash it,
    > brush yourself off and restart. But there are times (large production
    > applications) when you really want the Airbus A380 with the 100's of
    > automatic monitoring systems and checkout procedures to follow before
    > you take off, even if the skill level needed to use it is much higher
    > than the commuter car.


    OK. The quoted link deals with Unladen Swallow, which is an attempt to deal with the
    very real performance limitations of current Python systems. The remarks above deal with
    productivity scalability, which is a totally different matter. So...

    People can and do write large programs in Python, not just `throwing...medium sized
    scripts together'. Unlike, say, Javascript, it has the necessary machinery to build very
    large programs that are highly maintainable. One can reasonably compare it with Java, C#,
    and Smalltalk; the facilities are comparable, and all of those (as well as Python) are
    used for building enterprise systems.

    I believe that the A380's control software is largely written in Ada, which is a
    perfectly fine programming language that I would prefer not to write code in. For
    approximately 10 years, US DOD pretty much required the use of Ada in military (and
    aerospace) software (though a a couple of years ago I discovered that there is still
    one remaining source of Jovial compilers that still sells to DOD). According to a
    presentation by Lt. Colonel J. A. Hamilton, `Programming Language Policy in the DOD:
    After The Ada Mandate', given in 1999, `We are unlikely to see a return of a programming
    language mandate' (www.drew-hamilton.com/stc99/stcAda_99.pdf). As I understand it,
    the removal of the Ada mandate came from the realization (20 years after many computer
    scientists *told* DOD this) that software engineering processes contribute more to
    reliability than do programming language structures (c.f. Fred Brooks, `No Silver
    Bullet').

    So: to sum up, there are lots of large systems where Python might be totally appropriate,
    especially if complemented with processes that feature careful specification and strong
    automated testing. There are some large systems where Python would definitely NOT be
    the language of choice, or even usable at all, because different engineering processes
    were in place.

    From a productivity viewpoint, there is no data to say that Python is more, less, or equally
    scalable than <Language X> in that it produces correctly-tested, highly-maintainable programs
    at a lower, higher, or equal cost. I would appreciate it if people who wanted to comment on
    Python's scalability or lack thereof would give another programming language that they would
    compare it with.

    -- v
     
    Vincent Manis, Nov 14, 2009
    #4
  5. Robert P. J. Day

    Aahz Guest

    In article <>,
    Robert P. J. Day <> wrote:
    >
    >http://groups.google.com/group/unladen-swallow/browse_thread/thread/4edbc406f544643e?pli=1
    >
    > thoughts?


    Haven't seen this elsewhere in the thread:

    http://dalkescientific.com/writings/diary/archive/2009/11/15/100000_tasklets.html
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it." --Brian W. Kernighan
     
    Aahz, Nov 19, 2009
    #5
  6. Robert P. J. Day

    Robin Becker Guest

    Aahz wrote:
    > In article <>,
    > Robert P. J. Day <> wrote:
    >> http://groups.google.com/group/unladen-swallow/browse_thread/thread/4edbc406f544643e?pli=1
    >>
    >> thoughts?

    >
    > Haven't seen this elsewhere in the thread:
    >
    > http://dalkescientific.com/writings/diary/archive/2009/11/15/100000_tasklets.html



    I looked at this and it looks very good in that stackless appears twice as fast
    as go(lang) (I used to be in the department of computing at Imperial so I
    suppose I have to side with McCabe).

    Anyhow, my reading of why Pike was so proud of his set up and tear down of the
    tasks example was that these were real threads.

    Presumably that means they could potentially run in parallel on the 100000 cpu
    machines of the future.

    I'm not so clear on whether the threadless tasklets will run on separate cpus.
    --
    Robin Becker
     
    Robin Becker, Nov 20, 2009
    #6
  7. Robert P. J. Day

    sturlamolden Guest

    On 20 Nov, 11:12, Robin Becker <> wrote:

    > Presumably that means they could potentially run in parallel on the 100000 cpu
    > machines of the future.
    >
    > I'm not so clear on whether the threadless tasklets will run on separate cpus.


    You can make a user-space scheduler and run a 100000 tasklets on a
    threadpool. But there is a GIL in stackless as well.

    Nobody wants 100000 OS threads, not with Python, not with Go, not with
    C.

    Also note that Windows has native support for "taskelets", regardless
    of language. They are called "fibers" (as opposed to "threads") and
    are created using the CreateFiber system call. I would not be
    surprised if Unix'es has this as well. We do not need Stackless for
    light-weight threads. We can just take Python's threading modules' C
    code and replace CreateThread with CreateFiber.
     
    sturlamolden, Nov 20, 2009
    #7
  8. Robert P. J. Day

    Aahz Guest

    In article <>,
    sturlamolden <> wrote:
    >
    >Also note that Windows has native support for "taskelets", regardless
    >of language. They are called "fibers" (as opposed to "threads") and are
    >created using the CreateFiber system call. I would not be surprised if
    >Unix'es has this as well. We do not need Stackless for light-weight
    >threads. We can just take Python's threading modules' C code and
    >replace CreateThread with CreateFiber.


    Are you advocating a high-fiber diet?
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it." --Brian W. Kernighan
     
    Aahz, Nov 20, 2009
    #8
  9. Robert P. J. Day

    sturlamolden Guest

    On 20 Nov, 22:45, (Aahz) wrote:

    > Are you advocating a high-fiber diet?


    Only if you are a ruminant.

    No really...

    Windows has user-space threads natively. But you must reserve some
    stack space for them (from virtual memory), which mainly makes them
    useful on 64 bit systems.
     
    sturlamolden, Nov 21, 2009
    #9
  10. Robert P. J. Day

    Nobody Guest

    On Fri, 20 Nov 2009 09:51:49 -0800, sturlamolden wrote:

    > You can make a user-space scheduler and run a 100000 tasklets on a
    > threadpool. But there is a GIL in stackless as well.
    >
    > Nobody wants 100000 OS threads, not with Python, not with Go, not with
    > C.
    >
    > Also note that Windows has native support for "taskelets", regardless
    > of language. They are called "fibers" (as opposed to "threads") and
    > are created using the CreateFiber system call. I would not be
    > surprised if Unix'es has this as well. We do not need Stackless for
    > light-weight threads. We can just take Python's threading modules' C
    > code and replace CreateThread with CreateFiber.


    POSIX.1-2001 and POSIX.1-2004 have makecontext(), setcontext(),
    getcontext() and swapcontext(), but obsoleted by POSIX.1-2008.

    They are available on Linux; I don't know about other Unices.
     
    Nobody, Nov 21, 2009
    #10
  11. Robert P. J. Day

    Robin Becker Guest

    sturlamolden wrote:
    > On 20 Nov, 11:12, Robin Becker <> wrote:
    >
    >> Presumably that means they could potentially run in parallel on the 100000 cpu
    >> machines of the future.
    >>
    >> I'm not so clear on whether the threadless tasklets will run on separate cpus.

    >
    > You can make a user-space scheduler and run a 100000 tasklets on a
    > threadpool. But there is a GIL in stackless as well.
    >
    > Nobody wants 100000 OS threads, not with Python, not with Go, not with
    > C.
    >
    > Also note that Windows has native support for "taskelets", regardless
    > of language. They are called "fibers" (as opposed to "threads") and
    > are created using the CreateFiber system call. I would not be
    > surprised if Unix'es has this as well. We do not need Stackless for
    > light-weight threads. We can just take Python's threading modules' C
    > code and replace CreateThread with CreateFiber.
    >

    ........

    not really sure about all the parallelism that will actually be achievable, but
    apparently the goroutines are multiplexed onto native threads by the run time.
    Apparently each real thread is run until it blocks and then another goroutine is
    allowed to make use of the thread. Apparently the gccgo runtime has 1 goroutine
    per thread and is different to the fast compilers.
    --
    Robin Becker
     
    Robin Becker, Nov 23, 2009
    #11
  12. Robert P. J. Day

    Robin Becker Guest

    sturlamolden wrote:
    > On 20 Nov, 11:12, Robin Becker <> wrote:
    >
    >> Presumably that means they could potentially run in parallel on the 100000 cpu
    >> machines of the future.
    >>
    >> I'm not so clear on whether the threadless tasklets will run on separate cpus.

    >
    > You can make a user-space scheduler and run a 100000 tasklets on a
    > threadpool. But there is a GIL in stackless as well.
    >
    > Nobody wants 100000 OS threads, not with Python, not with Go, not with
    > C.
    >
    > Also note that Windows has native support for "taskelets", regardless
    > of language. They are called "fibers" (as opposed to "threads") and
    > are created using the CreateFiber system call. I would not be
    > surprised if Unix'es has this as well. We do not need Stackless for
    > light-weight threads. We can just take Python's threading modules' C
    > code and replace CreateThread with CreateFiber.
    >

    ........

    not really sure about all the parallelism that will actually be achievable, but
    apparently the goroutines are multiplexed onto native threads by the run time.
    Apparently each real thread is run until it blocks and then another goroutine is
    allowed to make use of the thread. Apparently the gccgo runtime has 1 goroutine
    per thread and is different to the fast compilers.
    --
    Robin Becker
     
    Robin Becker, Nov 23, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Sascha Kerschhofer

    SVG scaleable?

    Sascha Kerschhofer, Jul 15, 2003, in forum: XML
    Replies:
    6
    Views:
    2,065
  2. Jens Piegsa

    Scaleable Swing components

    Jens Piegsa, Dec 16, 2006, in forum: Java
    Replies:
    0
    Views:
    325
    Jens Piegsa
    Dec 16, 2006
  3. jsqrd

    Scaleable naviagtion pane

    jsqrd, Mar 2, 2007, in forum: ASP .Net
    Replies:
    2
    Views:
    296
    jsqrd
    Mar 2, 2007
  4. metaperl
    Replies:
    4
    Views:
    336
    Chris Uppal
    Mar 3, 2007
  5. Terry Reedy
    Replies:
    85
    Views:
    1,503
    Aaron Watters
    Nov 21, 2009
Loading...

Share This Page