platform-specific overrides of functions and class methods (expandingon imputils demo code)

Discussion in 'Python' started by lkcl, Aug 16, 2009.

  1. lkcl

    lkcl Guest

    i've just had to put something together for pyjamas-desktop which may
    prove to be useful to other people, so i'm pointing people in its
    general direction, for archive purposes.

    the purpose behind the platform override system is to allow
    implementations of a common API, in python, to share the majority of
    their codebase (modules, classes, functions etc.) _but_, for _very_
    specific platform implementation purposes, allow for "overrides".

    example:

    test.py
    def test():
    print "hello"

    platform/testOverridingPlatformName.py:
    def test():
    print "this is not kansas any more"

    code is here:
    http://pyjamas.svn.sourceforge.net/viewvc/pyjamas/trunk/pyjd/

    modules to pay attention to: importers.py (a modified version of Demos/
    imputils/importers.py) and modcompile.py. setup/init function is very
    simple - see __init__py.in last few lines.

    on reading modcompile.py you may be forgiven for going "wtf????" but
    basically PlatformParser loads the python source; turns it into an
    AST; then also the "platform-specific" version is loaded and turned
    into an AST; then, the two ASTs are handed to the "merge" function
    which does a top-level "replace" of functions and a top-level
    "replace" of class methods.

    _theen_ the resultant "merged" AST is handed to a module which was
    derived from compiler/pycodegen.py - none of the code in there can
    cope with being handed an already-compiled AST so it was necessary to
    make a class that did. looking around on the internet i find quite a
    few people asking about how to do this, so ... take a look at
    modcompile.Module, and how it's used. very simple.

    i've disabled saving and detection of .pyc files for now, because the
    job of loading code from .pyc should really be done by PlatformParser
    (which understands the concept of... duh, a platform). the _last_
    thing that you want to happen is to run one platform's code, generate
    some .pyc files, then change a config file to run a different back-end
    platform, for example (which is possible with pyjamas-desktop) and end
    up running the _wrong_ platform-specific code.

    for those people wondering, "why in god's green earth would anyone
    want to _do_ such a thing???" it's quite simple: the alternative is a
    complete dog's dinner, in pyjamas:

    def do_some_DOM_manipulation(document):
    # heeeere we go...
    if platform == 'hulahop' # this is for XULrunner
    do_some_stuff()
    elif platform == 'pywebkitgtk':
    do_something_different()
    elif platform == 'mshtml' # for pywin32 and comtypes....
    do_something_dreadful()
    else:
    do_the_default_here()

    now imagine that across an API with ... four hundred functions.

    i just... couldn't bring myself to do that. not when the pyjs (python-
    to-javascript) compiler _already_ solved this problem (thanks to james
    tauber) by deploying the AST merge concept.

    all i did was merge that with the imputils demo code so that, rather
    than at compile-time the pyjs compiler goes and generates five
    platform-specific versions of the same application, pyjamas-desktop at
    _run_ time can do exactly the same thing.

    but - it's a generic enough idea to be of value elsewhere - for
    example, all the loooovely code in e.g. the setup distutils? all
    those looovely "if os.platform == 'win32'", and "if sys.this = 'posix'
    " could be replaced with platform-specific overrides that got merged
    at run-time, thus _dramatically_ simplifying the look and the
    useability of the code.

    enjoy.

    l.
     
    lkcl, Aug 16, 2009
    #1
    1. Advertising

  2. lkcl

    lkcl Guest

    Re: platform-specific overrides of functions and class methods(expanding on imputils demo code)

    thought that people might like to know: i found also that imputil, the
    standard python module, was lacking the necessary complexity in being
    a substitute for the standard __import__ function.

    the additions required were very simple:

    # note the addition of level=-1 which is ignored

    def _import_hook(self, fqname, globals=None, locals=None,
    fromlist=None,
    level=-1):

    .....

    # Grrr, some people "import os.path" or do "from os.path
    import ..."
    if len(parts) == 2 and hasattr(top_module, parts[1]):
    if fromlist:
    return getattr(top_module, parts[1])
    else:
    return top_module

    # assume that the module has already been imported,
    # walk from top_module to find it.
    mod = top_module

    for k in parts[1:]:
    if not hasattr(mod, k):
    print "no mod", mod, k, parts
    mod = None
    break
    mod = getattr(mod, k)

    if mod:
    return mod

    # If the importer does not exist, then we have to bail. A
    missing
    # importer means that something else imported the module, and
    we have
    # no knowledge of how to get sub-modules out of the thing.
    raise ImportError, 'No module named ' + fqname

    it's the module-walking that's been added: going from the "top"
    module, looking for an attribute of the split-parts. this code makes
    the assumption that if top module already exists, and it has a sub-
    module, and that has a sub-sub-module, and that has a sub-sub-sub-
    module etc. then it is perfectly reasonable to return that pre-
    existing, already-imported [sub-]+ module as the return result.

    the reason why this had to be added is because comtypes.gen.{some
    module} auto-generator relies on it, and calls __import__ *direct*.

    if somebody would like to add this to the python bugtracker, as a
    contribution, that would be great. alternatively, you might like to
    have a word with the python developers to get them to remove the
    censorship on my contributions.

    l.
     
    lkcl, Aug 17, 2009
    #2
    1. Advertising

  3. lkcl

    Aahz Guest

    Re: platform-specific overrides of functions and class methods(expanding on imputils demo code)

    In article <>,
    lkcl <> wrote:
    >
    >if somebody would like to add this to the python bugtracker, as a
    >contribution, that would be great. alternatively, you might like to
    >have a word with the python developers to get them to remove the
    >censorship on my contributions.


    Excuse me? What censorship?
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Given that C++ has pointers and typecasts, it's really hard to have a
    serious conversation about type safety with a C++ programmer and keep a
    straight face. It's kind of like having a guy who juggles chainsaws
    wearing body armor arguing with a guy who juggles rubber chickens wearing
    a T-shirt about who's in more danger." --Roy Smith
     
    Aahz, Aug 21, 2009
    #3
  4. lkcl

    lkcl Guest

    Re: platform-specific overrides of functions and class methods(expanding on imputils demo code)

    On Aug 21, 12:58 am, (Aahz) wrote:
    > In article <>,
    >
    > lkcl <> wrote:
    >
    > >if somebody would like to add this to the python bugtracker, as a
    > >contribution, that would be great. alternatively, you might like to
    > >have a word with the python developers to get them to remove the
    > >censorship on my contributions.

    >
    > Excuse me? What censorship?


    i was ordered to cease contributing to python. the cause was the
    provision of a port of python to run under mingw32, which passed all
    but eight of the regression tests.

    l.
     
    lkcl, Sep 6, 2009
    #4
  5. lkcl

    Terry Reedy Guest

    Re: platform-specific overrides of functions and class methods(expanding on imputils demo code)

    lkcl wrote:
    > On Aug 21, 12:58 am, (Aahz) wrote:
    >> In article <>,
    >>
    >> lkcl <> wrote:
    >>
    >>> if somebody would like to add this to the python bugtracker, as a
    >>> contribution, that would be great. alternatively, you might like to
    >>> have a word with the python developers to get them to remove the
    >>> censorship on my contributions.

    >> Excuse me? What censorship?

    >
    > i was ordered to cease contributing to python.


    As I remember from the public discussion, you were asked to cease using
    the bugtracker as a progress blog ...

    > the cause was the
    > provision of a port of python to run under mingw32, which passed all
    > but eight of the regression tests.


    and wait until you had a complete patch to submit. There is also a real
    question of whether such ports should be folded into the main CPython
    code base, as some have been, or maintained as separate projects, as
    other have been.

    tjr
     
    Terry Reedy, Sep 6, 2009
    #5
  6. lkcl

    Aahz Guest

    Re: platform-specific overrides of functions and class methods(expanding on imputils demo code)

    In article <>,
    Terry Reedy <> wrote:
    >lkcl wrote:
    >> On Aug 21, 12:58 am, (Aahz) wrote:
    >>> In article <>,
    >>> lkcl <> wrote:
    >>>>
    >>>> if somebody would like to add this to the python bugtracker, as a
    >>>> contribution, that would be great. alternatively, you might like to
    >>>> have a word with the python developers to get them to remove the
    >>>> censorship on my contributions.
    >>>
    >>> Excuse me? What censorship?

    >>
    >> i was ordered to cease contributing to python.

    >
    >As I remember from the public discussion, you were asked to cease using
    >the bugtracker as a progress blog ...
    >
    >> the cause was the
    >> provision of a port of python to run under mingw32, which passed all
    >> but eight of the regression tests.

    >
    >and wait until you had a complete patch to submit.


    That matches my memory as well. Luke, your contributions would
    definitely be welcome, but contributions must follow some process.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "It's 106 miles to Chicago. We have a full tank of gas, a half-pack of
    cigarettes, it's dark, and we're wearing sunglasses." "Hit it."
     
    Aahz, Sep 13, 2009
    #6
  7. lkcl

    lkcl Guest

    Re: platform-specific overrides of functions and class methods(expanding on imputils demo code)

    On Sep 6, 5:49 pm, Terry Reedy <> wrote:
    > lkclwrote:
    > > On Aug 21, 12:58 am, (Aahz) wrote:
    > >> In article <>,

    >
    > >>lkcl <> wrote:

    >
    > >>> if somebody would like to add this to the python bugtracker, as a
    > >>> contribution, that would be great. alternatively, you might like to
    > >>> have a word with the python developers to get them to remove the
    > >>> censorship on my contributions.
    > >> Excuse me? What censorship?

    >
    > > i was ordered to cease contributing to python.

    >
    > As I remember from the public discussion, you were asked to cease using
    > the bugtracker as a progress blog ...


    i used the bugtracker as a substitute for svn access, so that other
    people who may be interested could follow, pick up and contribute.

    to describe valuable contributions as "a prrrogress blllllooggggg"
    makes it sound like the work is at the same level as much of the
    mindless drivel found on e.g. wordpress.

    if that's how it was being viewed, then it's no wonder there were
    complaints.

    at the time that i was ordered to cease contributing, the work done
    passed all but 8-12 of the standard python regression tests. out of
    several thousand.


    > > the cause was the
    > > provision of a port of python to run under mingw32, which passed all
    > > but eight of the regression tests.

    >
    > and wait until you had a complete patch to submit.


    1) i've stopped. it's declared complete. there. it can now be
    added. does that help?

    2) you know as well as i do that there is never going to be a
    "complete" patch. no-one in the world is a "perfect" coder. work is
    _always_ ongoing. there is _always_ going to be "a better patch".

    3) "wait until there is a complete patch" flies in the face of the
    basic tenets of both commercial _and_ free software development. you
    _never_ "wait until the work is completed". release early, release
    often, remember?

    > There is also a real
    > question of whether such ports should be folded into the main CPython
    > code base, as some have been, or maintained as separate projects, as
    > other have been.


    the mingw32 port comprises two (necessary) features: 1) cross-
    compiling support, which the mingw32 port uses to create python.exe
    (from a posix system) 2) actual support for mingw32-compiled python,
    including proper support on ILP32 systems, checking support for
    features properly using autoconf tests not "#ifdef WIN32" etc. etc.

    cross-compiling of cpython is tricky as hell, starting from pgen and
    going from there, to create python.exe. to be able to run that - in
    order to compile the modules and run the regression tests is _even
    more_ tricky. but - it's been done.

    roumen and the other contributors to the cross-compile support have
    consistently had their contributions considered to be worthless,
    basically, for nearly five years, now. _despite_ clear evidence
    indicating a need, and clear requests for cross-compiling to be
    included in the standard python distribution.

    for example, from the gentoo team, who have a bitch of a job cross-
    compiling python for embedded-mips systems.


    then there's the issue of continuing to rely on the proprietary
    microsoft compiler toolchain. you _know_ that's going to cause
    problems, ultimately, with an important free software project such as
    python.

    so to throw mud in the face of people willing to think ahead and put
    in the work to get you [PSF / python-devs] out of a potentially
    difficult [political/strategic] situation is just ... i don't
    understand it. i even solved the issue of compiling python under
    mingw32 using assemblies, so that it could be linked against MSVCRT80,
    91 etc.

    these kinds of multiple interdependent technical issues aren't the
    sorts of things you "put into a single patch and then stop" - you keep
    on at it, picking at one part of the pattern, moving to the next
    issue, pick some more, solve that, and keep moving.

    so - these are the kinds of things i received [not exact words]:
    "please stop submitting continual patches, we think it's about the
    same level of value as some of the trash we see on wordpress.com".
    "please stop doing work because we think it's worthless". "we don't
    really like and don't really want to maintain the win32 port of python
    anyway, so why are you bothering to work on it?"

    do you notice a theme, here? there isn't anything which is
    particularly encouraging. or places any value on the work being
    done. or indicates a willingness to extend free software.

    this theme was so completely contrary to what i was expecting
    [openness, trust and respect] that i just... went "does not compute"
    and continued coding. i don't mean that in a flippant way - i really
    _mean_, my mind went "this message [from e.g. martin] doesn't match my
    expectations of a free software project developer/contributor. what's
    the next technical challenge i have to solve?" note the discontinuity
    in the internal dialog between those two sentences. there was no
    _judging_ of the message received; no "pffh. that idiot. let's spend
    some time and energy _actively_ dismissing the person and the
    message". it was _literally_: "does not compute. equals zero. next
    issue on list: do coding". it's a pathological approach, i know - but
    it's one that solves technical issues with extreme [some say alarming]
    speed.

    anyway. it was an interesting experience. and you can make use of
    someone with such capabilities, and _not_ get stressed, and work out
    how to make use of them -or you can throw their work in their face,
    and move the goalposts around, close all their bugreports (making up
    reasons where possible) - all to get them to "go away". "stop
    increasing my stress levels".

    overall, from the experience, i did get the overall impression that
    the core developers are slightly overwhelmed, and don't have time to
    focus on anthing other than what's already on "the roadmap". i.e.
    "unscheduled" major free software contributions - such as ports to new
    systems - are viewed as a burden rather than a responsibility that
    should, without question and without hesitation, be incorporated and
    encouraged, in order to increase the reach of python and in order to
    build a stronger, larger python community.

    one of the arguments utilised to discourage the work on the mingw32
    port was that "it would have to be supported - by us. officially."
    such statements are self-fulfilling, and increase the "burden of
    responsibility" and the stress.

    a very sensible person told me that stress is where the mind has an
    internal picture (expectations / viewpoint), makes a comparison with
    the outside world, and the discrepancy is so large that the mind
    CANNOT COPE.... and seeks to place BLAME on the EXTERNAL world.

    if people are willing, there are _always_ ways in which contributions
    can be accepted, such that free software can progress. it's just that
    in the [extremely rapid] development cycle of the mingw32 cross-
    compiled port of python, i didn't meet the "expectations" of the
    python developers. there's nothing i can do about peoples' _internal_
    expectations, and i'm really sorry, but i don't have the social-
    interaction-capacity to _understand_ peoples' expectations [i don't
    get enough practice]. so, i "make do" by assuming that free software
    people are always going to be receptive, enthusiastic, open and
    encouraging, and that, when they encounter an experience that
    indicates that they might fall short of the "ideal" free software
    project, they'll move sharp-ish to correct the issue encountered.

    my limitations are that if they're _not_ any or all of those things,
    or if those corrections _aren't_ made, i simply do the mental
    equivalent of "syntax error. decrease priority of message. pop(next
    task on stack)". i don't _forget_ the incoming message - it just
    gets... a lower priority, and thus appears to have been ignored. "too
    much energy will be consumed processing this discrepancy between _my_
    ideal expectations of how free software contributors should
    communicate, and work on free software, and this message. that's a
    distraction from spending energy on getting free software work done.
    decision equals easy: work on free software".

    that's what happened to you, martin. it looked like i was ignoring
    you. it also didn't help that i also was sufficiently busy focussing
    approx 11 hours non-stop a day for several weeks that some of the
    things you asked me to stop doing, so as to reduce your stress levels,
    i _literally_ didn't see them.

    martin and guido: you asked me to apologise, some months ago. to do
    so requires that i express clearly what it was that i was apologising
    _for_, as what i'd already apologised for simply wasn't enough and
    wasn't clear enough, (and my previous apologies had several areas
    where people assumed the worst, and assumed that i was trying to
    disguise ego trips as apologies). that's taken nearly a year to go
    by, for me to subconsciously work on what to say, and work through it,
    above.

    so - with the context above: you can understand and appreciate this:
    that i'm sorry that i have different expectations of how free software
    projects should operate, and for not knowing what the right thing to
    do is when my expectations aren't met. as in - i _really_ don't
    know. i seem to have this strange belief that if you solve the
    technical challenges, people will work out how to move the goalposts
    _towards_ incorporating the solutions, rather than away -
    _irrespective_ of the "social" issues, placing "social interaction"
    issues at absolute rock-bottom priority against achieving the
    technical goal. i'm sorry that this ethos and the beliefs and
    expectations that i have cause people distress.

    l.
     
    lkcl, Sep 15, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Remy Cool
    Replies:
    1
    Views:
    433
    Remy Cool
    Aug 27, 2003
  2. F. GEIGER
    Replies:
    9
    Views:
    1,226
    F. GEIGER
    May 3, 2004
  3. Andy Leszczynski

    wxPython demo /Process does not open new demo

    Andy Leszczynski, Feb 18, 2005, in forum: Python
    Replies:
    1
    Views:
    647
    Andy Leszczynski
    Feb 18, 2005
  4. Kenneth McDonald
    Replies:
    5
    Views:
    325
    Kenneth McDonald
    Sep 26, 2008
  5. MaggotChild
    Replies:
    14
    Views:
    202
    MaggotChild
    Sep 23, 2009
Loading...

Share This Page