YASS (Yet Another Success Story)

Discussion in 'Python' started by k3xji, Jun 20, 2009.

  1. k3xji

    k3xji Guest

    Hi all,

    Started a project year ago with hard goals in mind : Developing a game
    server which can handle thousands of clients simultaneously.System
    must be stable, scalable, efficient, and if the client need to migrate
    the server to another OS, this will be a matter of time, if possible
    without changing any code. Also, the system should be controlled via
    database operations, meaning some specific DB changes should be
    determined by the system on-the-fly and necassary processing should be
    done at runtime(Client timeout values, adding new game lobbies..etc..)
    And all of this should be a one-man project dues to the fact that the
    project is a volunteer work, thus is separate from our ordinary jobs,
    there is no budget:). This was 8 months ago, and I still remember we
    are talking with my colleguae to decide which programming language is
    better for our requirements. We wrote the tradeoffs of different
    languages on board and compare our objectives against them by giving
    numbers. I remember, the last three standing programming languages
    are: C++, Java, and Python. We decided that code maintability and
    stability is cheaper than native code efficiency and also easy porting
    is important. C++ is only successfull on efficiency but maintaining
    the code, even if we use 3rd party libraries is a nightmare,
    especially in a massive multithreaded project like a game server.
    Also, we decided a game server's bottleneck is on network I/O other
    than the actual processing which means there should be not so much
    performance degradation between C++ and an interpreted language. So
    the final is between Java and Python. I know Java has been adding very
    good optimizations for interpreter efficiency over the years(JIT
    compiling..etc.) And I remember somewhere reading that Java is 10
    times faster than Python. But Python, on the other hand is a perfect
    language and very simple which means the project may be completed
    sooner and efficiency again will be not a so much problem as the
    bottleneck should be on network I/O if the server is coded correctly.
    Although we have made this decision, we still had some doubts but we
    move on with Python.
    I don't know Python at the time and only coded few simple projects
    with it.
    I started with reading python socket/threading tutorials, then I
    downloaded and read actual Python code to get deeper and deeper. This
    was 10 months ago. Now we have been able to implement the game server
    with Python using MySQL as a backend DB and currently 5000 players
    (increasing) are playing multiplayer games with it with a system load
    of 0.4-0.8(see Unix getloadavg output) on a 1.8 Ghz Dual Xeon
    processor. Let me give some benchmarks, please note that one thing I
    learn through this journey is that testing/benchmarking a server
    software is really difficult and maybe in some conditions impossible
    thing to do correctly as inactive client connections may cause
    indeterministic behavior and also WAN latencies are very different
    than LAN latencies..etc. Anyway, we had to do a benchmark before going
    with this server on live. We installed a UnrealIRCd server which is
    also using select() as an I/O paradigm just as we are and supporting
    many of the features we have. This server is a IRC server written in C
    and being written since nearly 10 years. We connect thousands of
    clients and do simple messaging between them. I don't have an official
    test/benchmark result or a nice graph for this, however the system
    running UnrealIRCD is locked up when processing 4 Mb/sec of data,
    where our 6 month old Python server is locking up the system resources
    after 3.5 Mb/sec. We have done other tests including stressing server
    accept(), adding inactive connections but I am really not %100
    confident about the test environments and do not want to give you
    false impression. But we decided that this 0.5 Mb factor is still not
    enough for us and we move on to inspect what is really causing the
    bottlenecks and how to optimize them. Here, to be honest, I cannot see
    very uch help from Python. Profiling/Optimization is the only place I
    am upset with Python. Here is the requirements for the profiler: I
    want to be able to enable/disable profiler tool on the fly without
    requiring any kind of restart and the profiler should be multithreaded
    (thus thread-safe). I decided that cProfiler and other profiler tools
    are in need of some updates to fulfill my reqýrements, which makes me
    write my own profiler. It took me 3 days to come up with a profiler
    that satisfies my requirements, this profiler will profile only my
    code, meaning the server code to avoid performance degradation and
    timings will need not be nested-aware and timing precision is not very
    important (other than secs). I have implemented the profiler with the
    help of decorators and started profiling my code on-live. Here comes
    the another part of Python that is some kind of shady: optimization.
    After profiling the code, it turns out most of the time is spent on
    the following:
    1) Selecting/Deselecting interest sets for select()
    2) Unnecessary function calls. (for integrating some OOP concepts, I
    think we have overlooked the performance factor and there really are
    some functions which can be inlined but declared as a function)
    3) Redundant try-except's in all over place(Again our fault to make
    the system stable, we have put some debug purposed-assert like try-
    excepts in the main server flow.)

    After, looking on these subjects for 2 weeks, we came up with
    solutions for all of them, and we see that it is working approximately
    5 times faster. This test is done in live, we run two server processes
    which are all running on a predefined CPU(means CPU affinity of the
    process is set before hand to avoid mis-interpreting system diagnose
    tools like top(1)). When same client number is connected to both
    servers, the second one is giving 5 times better values in top and ps
    tools. Also, the processing time (meaning the server loop except the
    select() call) is also 10 times faster than before. Just one note
    about optimizing Python code: do not optimize Python code based on
    your assumptions, just go and test if it really runs faster. I don't
    want to go to details of this hint, but believe me making Python code
    optimized may be very very tricky.

    It is then I decided to write up here this as a success story, as I am
    very newcomer to Python but come up with a nearly commercial product
    in a very short period of time and I don't think this is about my
    personal characteristics and intelligence or so:), as I am old enough
    to know/meet that there are much much more brilliant people than I am
    and they also have similar experiences with Python.

    So, one last note: every software project goes same tasks as above
    often much much more officially and carefully, I would suggest
    managers to see that just do not listen to the ordinary brain-washes.
    Python is a great choice for easy developing, easy debugging, easy
    maintaining and most importantly very very time-friendly. Of course
    there will be tasks .n which Python is suitable, but hey, if it Python
    is in the list, take it seriously.

    Thanks,

    Special thanks to people in this group who had answered my silly
    Python questions along the way.

    Sümer Cip
     
    k3xji, Jun 20, 2009
    #1
    1. Advertising

  2. En Sat, 20 Jun 2009 07:58:02 -0300, k3xji <> escribió:

    > Started a project year ago with hard goals in mind : Developing a game
    > server which can handle thousands of clients simultaneously. [...]
    > I don't know Python at the time and only coded few simple projects
    > with it.


    And you still could write the server - that's very good (and shows your
    own great skills and Python ease of use...)

    > After profiling the code, it turns out most of the time is spent on
    > the following:
    > [...] 3) Redundant try-except's in all over place(Again our fault to make
    > the system stable, we have put some debug purposed-assert like try-
    > excepts in the main server flow.)


    I don't think this should make a difference. Setting up a try/except block
    usually has a very small cost (if no exception is actually raised). Care
    to tell us more details?

    > Just one note
    > about optimizing Python code: do not optimize Python code based on
    > your assumptions, just go and test if it really runs faster. I don't
    > want to go to details of this hint, but believe me making Python code
    > optimized may be very very tricky.


    Yes, specially if you came from a statically typed language; what looks
    "innocent" may have a significant cost (e.g. resolving obj.name), and what
    looks complicated may be fairly fast (e.g. a list comprehension).

    > It is then I decided to write up here this as a success story, as I am
    > very newcomer to Python but come up with a nearly commercial product
    > in a very short period of time and I don't think this is about my
    > personal characteristics and intelligence or so:), as I am old enough
    > to know/meet that there are much much more brilliant people than I am
    > and they also have similar experiences with Python.


    Thanks for sharing your experience!

    > So, one last note: every software project goes same tasks as above
    > often much much more officially and carefully, I would suggest
    > managers to see that just do not listen to the ordinary brain-washes.
    > Python is a great choice for easy developing, easy debugging, easy
    > maintaining and most importantly very very time-friendly. Of course
    > there will be tasks .n which Python is suitable, but hey, if it Python
    > is in the list, take it seriously.


    Nice summary!

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jun 27, 2009
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Max M. Stalnaker

    python success story

    Max M. Stalnaker, Jan 30, 2006, in forum: Python
    Replies:
    3
    Views:
    292
    Steve Holden
    Jan 30, 2006
  2. tiewknvc9
    Replies:
    3
    Views:
    863
    Andrew Thompson
    Nov 15, 2006
  3. Berehem
    Replies:
    4
    Views:
    567
    Lawrence Kirby
    Apr 28, 2005
  4. Andrew Jonathan Fine

    Author of a Python Success Story Needs a Job!

    Andrew Jonathan Fine, Dec 28, 2009, in forum: Python
    Replies:
    27
    Views:
    701
    Simon Brunning
    Jan 15, 2010
  5. Gregory Seidman

    A success story

    Gregory Seidman, Mar 10, 2006, in forum: Ruby
    Replies:
    0
    Views:
    86
    Gregory Seidman
    Mar 10, 2006
Loading...

Share This Page