ANN: Shed Skin 0.2, an experimental (restricted) Python-to-C++compiler

Discussion in 'Python' started by Mark Dufour, Jul 19, 2009.

  1. Mark Dufour

    Mark Dufour Guest

    Hi all,

    I have just released version 0.2 of Shed Skin, an experimental
    (restricted) Python-to-C++ compiler (
    It comes with 7 new example programs (for a total of 40 example
    programs, at over 12,000 lines) and several important improvements/bug
    fixes. See for the
    full changelog.

    The new example programs consist of Disco, an elegant go player (see,
    a larger Voronoi implementation at 800 lines, a TSP algorithm
    simulating ant colonies, a nicer neural network algorithm and three
    compressors (Lempel-Ziv, huffman block, and arithmetic).

    Other than bug fixes for these programs, this release adds some
    important optimizations. First and foremost, inlining was greatly
    improved, resulting in potential speedups across the board. Second,
    loops such as 'for a, b in enumerate/zip(sequence[, sequence])' should
    now be dramatically faster (also inside list comprehensions), by
    avoiding allocation of intermediate tuples. Finally, basic list
    slicing should now be much faster.

    Please try it out!
    Mark Dufour.
    Mark Dufour, Jul 19, 2009
    1. Advertisements

  2. Mark Dufour

    William Dode Guest

    I just tested it with a litle game, to find the places of horse on
    a board 5x5. The result is :

    c 5s
    gcj 7s
    java 7s
    shedskin 8s
    python + psyco 18s
    cython avec malloc *int 18s
    cython 55s avec [] python
    python 303s (5m3s)
    William Dode, Jul 20, 2009
    1. Advertisements

  3. Note that both Psyco and Cython make a lot less assumptions about Python
    code than Shed Skin does. Psyco has the advantage of just needing to jump
    in when it finds out that it can help, so it's the most broadly compatible
    of the three. But Cython also supports quite a large corpus of dynamic
    Python code by now. Shed Skin has a lot of restrictions, many of which are
    by design. It's not intended to compile dynamic code, and I think that's a
    good thing, because that's what makes it fast for the code that it
    supports. Getting the same speed in Cython requires a bit more explicit
    typing, simply because Cython does not assume these restrictions.

    I think that all three have their raison d'être, and it currently looks
    like all three are there to stay and to keep growing better. And I'm also
    happy to read that some optimisations jump from one to the other. ;)

    Stefan Behnel, Jul 20, 2009
  4. Mark Dufour

    Bearophile Guest

    William Dode':
    Nice timings, can you please show me the Python, Java and C code
    versions? I may do more tests.
    The purpose of all those "example programs" in ShedSkin is to find
    bugs and to find details to speedup.

    Bearophile, Jul 20, 2009
  5. Mark Dufour

    skip Guest

    William> c 5s
    William> gcj 7s
    William> java 7s
    William> shedskin 8s
    William> python + psyco 18s
    William> cython avec malloc *int 18s
    William> cython 55s avec [] python
    William> python 303s (5m3s)

    I read just enough French to know that "avec" means "with", but I don't
    understand the difference between "avec malloc *int" and "avec []". Can you
    explain please?

    skip, Jul 20, 2009
  6. Mark Dufour

    Bearophile Guest

    Skip Montanaro:
    Maybe it's the time difference between using a Python list from Cython
    and using a C "array" allocated with a malloc from Cython.

    Bearophile, Jul 20, 2009
  7. Mark Dufour

    srepmub Guest

    also, which shedskin options did you use? did you use -bw, to disable
    bounds and wrap-around checking? this can make quite a difference for
    code that does a lot of indexing. if the code uses random numbers,
    then -r can also make a big difference, to use C rand(), instead of
    Python compatible random numbers.

    and which C++ compiler flags did you use? the default -O2, or
    something like -O3 -fomit-frame-pointer -msse2..?

    srepmub, Jul 20, 2009
  8. Mark Dufour

    William Dode Guest

    Of course, the codes are here :

    Like you'll see, i tried to use exactly the same code for each langage.
    I used the default, shedksin; make
    shedskin 0.2

    With -bw and -O3 -fomit-frame-pointer -msse2 i have 5.5s (instead of 8)

    Let me know if you find better.
    William Dode, Jul 21, 2009
  9. Mark Dufour

    William Dode Guest

    yes, it's this
    William Dode, Jul 21, 2009
  10. Mark Dufour

    srepmub Guest

    thanks. now I'm wondering how fast does the C version become with
    these flags..? :)

    srepmub, Jul 21, 2009
  11. Mark Dufour

    William Dode Guest

    I don't see any difference...
    William Dode, Jul 21, 2009
  12. Mark Dufour

    Bearophile Guest

    Nick Craig-Wood:
    You can show us the Python (SSPython) code, and we can try to find the
    problem. Sometimes there's no simple ways to solve such problems.

    Generally for not very large progrograms if SS doesn't compile in
    about a minute or so then it's gone in infinite loop (there's a
    compilation flag that avoids some infinite loops, try it).

    Bearophile, Jul 21, 2009
  13. Mark Dufour

    Bearophile Guest

    William Dode':
    It's a cute solver.

    Few more versions of mine:

    #1, a Psyco version of mine:

    #2, unrolled Psyco version:

    #3, a quick D (D1) version:

    Timings (no printing), seconds, best of 3:
    #1: 4.79
    #2: 3.67
    #3: 1.10
    Your Psyco version: 13.37
    Your C version, compiled with GCC 4.3.2, -s -O3 -fomit-frame-pointer:

    I have timed the whole running time of the programs, with an external
    timer, so my timings include the start of the PythonVM and the final
    cleanup of the GC.

    Please, feel free to time my code again, so we can compare with your
    other timings (Java, etc) better.

    I have used Psyco 1.6 final and Python 2.6.2 on a Core2 CPU at 2 GHz.

    To compile the D1 code you can use the free DMD compiler for D1, I
    have used version 1.042, ).

    In such benchmarks it's often better to start from the fastest low
    level code (written in an imperative language as C/D/C++, or a version
    in a functional language like Clean or OCaML) and then use it to
    create higher level versions (in Python, etc). This offers you a
    baseline timing, and usually the small differences of the higher level
    code compared to the original C/Clean code don't invalidate the test.

    I have changed only small things in the program, as you can see the
    Psyco program #2 is faster than your C code :) If you learn how to
    use it well, Psyco1.6 can often lead to very good performance. But lot
    of people don't know how to use Psyco well (I too am ignorant: I don't
    know how to profile Python programs yet).

    Bearophile, Jul 21, 2009
  14. Mark Dufour

    Terry Reedy Guest

    Split it into pieces and compile each separately.

    Terry Reedy, Jul 21, 2009
  15. Mark Dufour

    greg Guest

    Posting benchmark times for Pyrex or Cython is pretty
    meaningless without showing the exact code that was
    used, since times can vary enormously depending on
    how much you C-ify things.
    greg, Jul 22, 2009
  16. Mark Dufour

    Bearophile Guest

    Bearophile, Jul 22, 2009
  17. Mark Dufour

    William Dode Guest

    I updated the script (python, c and java) with your unrolled version
    + somes litle thinks.

    I also tried with python3.1, unladen Q2, ironpython1.1.1

    Unfortunately it doesn't work more with shedskin, i'll see on the
    shedskin group...

    c 1.85s
    gcj 2.15s
    java 2.8s
    python2.5 + psyco 3.1s
    unladen-2009Q2 145s (2m45)
    python2.5 254s (4m14s)
    python3.1 300s (5m)
    ironpython1.1.1 680s (11m20)
    William Dode, Jul 22, 2009
  18. Mark Dufour

    srepmub Guest

    please send any program that doesn't work with shedskin (it still is
    experimental after all) to me, or create an issue at, and I will have a look at the problem.

    srepmub, Jul 22, 2009
  19. Mark Dufour

    William Dode Guest

    William Dode, Jul 22, 2009
  20. Cool; it would be interesting to see the numbers for Jython and Boo as
    well if it's not too much effort.

    George Sakkis, Jul 22, 2009
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.