Python Packages : A loomingproblem? packages might no longer work? (well not on your platform or pyt

Discussion in 'Python' started by David Lyon, Apr 23, 2009.

  1. David Lyon

    David Lyon Guest

    Hi all,

    I'm working on a python package manager gui. Mainly because I struggle on
    windows
    getting python packages installed. In my mind, there seemed to be some
    minor
    problems, but then I did the numbers. Do the numbers I writa about here
    reflect
    reality?

    Should we discuss stuff like this? Debate welcome...


    = Introduction =

    One of the big challenges for Python going forward is providing a testing
    infrastructure for Python Packages.

    There are now over 6,000 packages listed on PyPi - and this number can only
    get bigger.

    Then, there are the three major operating systems:

    * Windows
    * Mac
    * Linix/Unix

    To complicate the problem, there are now many versions of each operating
    system.

    Multiply those two combinations by all of the versions of Python that
    already exist (not to mention the ones coming) and we start to that we are
    heading into complexity. If there are 4 major windows revisions and 4 major
    Linuxes, and two major Mac platforms, we end up with perphaps (6,000 x (4 +
    4 + 2)) 60,000 delivery possibilities.

    That number then needs to be multiplied by the number of python versions,
    which possibly include 2.1, 2.2, 2.3, 2.4, 2.5, 2.6 and coming up.. the 3
    series...

    So that could be (60,000 x 7 (python versions)) 420,000 variations of known
    python packages.

    To date... the testing has been done... we have to assume... manually with
    some automation.

    But we can't expect package maintainers to be forever testing their own
    code on platforms that they simply don't have access to.

    A more reasonable and cost effective option is to have this testing done on
    a server farm virtual environment building infrastructure.

    In simple terms, we need to build all the packages that exist for Python on
    a daily basis on all of the environments and report any issues back to the
    registered maintainers.

    This job is too big to be done manually. We need to use either a
    Super-Computer or a Server Farm. Fortunately, Server Farms are close at
    hand.

    = Server Farm Virtual Environments =

    Google and Amazon web services are two organisations amongst others that
    offer commercial virtual server farms that could be employed to do the
    above build process of all the python packages.

    Python scripts would be developed utilising the python "test" frameworks to
    supervise the build on each and every platform.

    With this basic structure, a daily building/testing infrastructure working
    across the different versions of python and operating systems, could easily
    become a reality.

    At present AWS offer virtual environments for both Windows and Linux. These
    can be seen on these links:

    *
    http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=209
    *
    http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=208

    A service to do building on Mac Virtual Machines needs to be located.

    = Test Scripts =

    A test script will be developed that will cycle through all the packages on
    pypi, download the package and build it on all available platforms.

    The results of the build can then be made available via some sort of web
    delivery system. Describing on which platforms the builds were successful
    and not.

    In the past, it has been difficult for developers to test on all platforms.

    These facilities are bound to improve overal code quality across the python
    universe.

    = Scope of Testing =

    It's important to define what and can be and what cannot be tested.

    The scope of the framework will be:

    * to check that each package can be installed on all the relevant
    platforms
    using the setup.py script

    * to run the built in tests within the package

    * to check that the package can be de-installed on the relevant platforms
    David Lyon, Apr 23, 2009
    #1
    1. Advertising

  2. Re: Python Packages : A loomingproblem? packages might no longer work? (well not on your platform orpython version anyway)

    On Wed, 22 Apr 2009 19:06:18 -0400, David Lyon wrote:


    > = Introduction =
    >
    > One of the big challenges for Python going forward is providing a
    > testing infrastructure for Python Packages.


    Why is it the responsibility of the Python language to provide such a
    testing infrastructure?


    > There are now over 6,000 packages listed on PyPi - and this number can
    > only get bigger.
    >
    > Then, there are the three major operating systems:
    >
    > * Windows
    > * Mac
    > * Linix/Unix


    Linux.

    > To complicate the problem, there are now many versions of each operating
    > system.
    >
    > Multiply those two combinations by all of the versions of Python that
    > already exist (not to mention the ones coming) and we start to that we
    > are heading into complexity. If there are 4 major windows revisions and
    > 4 major Linuxes, and two major Mac platforms, we end up with perphaps
    > (6,000 x (4 + 4 + 2)) 60,000 delivery possibilities.



    But most of those packages will probably work without modification on any
    platform supported by Python. We can take that as a given (subject to
    correction), because the Python language itself has been tested on those
    platforms.


    > That number then needs to be multiplied by the number of python
    > versions, which possibly include 2.1, 2.2, 2.3, 2.4, 2.5, 2.6 and coming
    > up.. the 3 series...


    Why? Why should every package on PyPI need to support all those Python
    versions? That should be the decision of the package maintainer. If they
    want to support every version of Python back to 1.0, they can, and if
    they want to only support version 2.5 that's fine too.


    > So that could be (60,000 x 7 (python versions)) 420,000 variations of
    > known python packages.


    Could be but won't be.


    > To date... the testing has been done... we have to assume... manually
    > with some automation.


    Why do we "have to" assume this? For all we know, three quarters of the
    packages on PyPI have never been tested *at all*.


    > But we can't expect package maintainers to be forever testing their own
    > code on platforms that they simply don't have access to.


    We don't.


    > A more reasonable and cost effective option is to have this testing done
    > on a server farm virtual environment building infrastructure.


    You haven't demonstrated that we need to have all this testing done in
    the first place.


    > In simple terms, we need to build all the packages that exist for Python
    > on a daily basis on all of the environments and report any issues back
    > to the registered maintainers.


    We "need" to do this? Why? What's the dire problem you are trying to
    solve? That somebody downloads a random package off PyPI and it doesn't
    work? Oh dear, how will we cope!


    > This job is too big to be done manually. We need to use either a
    > Super-Computer


    What "Super-Computers" do you know of that run Python?


    > or a Server Farm. Fortunately, Server Farms are close at
    > hand.
    >
    > = Server Farm Virtual Environments =
    >
    > Google and Amazon web services are two organisations amongst others that
    > offer commercial virtual server farms that could be employed to do the
    > above build process of all the python packages.


    Who is paying for this?



    > Python scripts would be developed utilising the python "test" frameworks
    > to supervise the build on each and every platform.


    Who do you suggest should develop these scripts?


    > With this basic structure, a daily building/testing infrastructure
    > working across the different versions of python and operating systems,
    > could easily become a reality.


    Oh yeah, "easily".




    --
    Steven.
    Steven D'Aprano, Apr 23, 2009
    #2
    1. Advertising

  3. David Lyon

    David Lyon Guest

    Hi Steven,

    You make some good points...

    Let me try to answer them..

    > Why is it the responsibility of the Python language to provide such a
    > testing infrastructure?


    First define "Python language".

    Ok... we know it as the core interpretor. But to the developer it is also
    all the packages that everybody has contributed.

    The "Python language" is the interpretor and all the packages the
    "community" has put together.

    > Why? Why should every package on PyPI need to support all those Python
    > versions? That should be the decision of the package maintainer. If they
    > want to support every version of Python back to 1.0, they can, and if
    > they want to only support version 2.5 that's fine too.


    Why shouldn't packages support more than one python version?

    Looking at it conversely....

    Why should the package developer dictacte which python version the package
    will run on ?

    > For all we know, three quarters of the
    > packages on PyPI have never been tested *at all*.


    Right. Why not run some tests....

    > What's the dire problem you are trying to
    > solve?


    Backward and forward compatability of python package resources.

    > What "Super-Computers" do you know of that run Python?


    Google. Amazon web services..

    > Who is paying for this?


    >From as little as $30 per month. Funding isn't so much the issue.


    >> With this basic structure, a daily building/testing infrastructure
    >> working across the different versions of python and operating systems,
    >> could easily become a reality.

    >
    > Oh yeah, "easily".


    pypi_packagelist = getallpypipackages()

    for package in pypi_packagelist:
    testpackageonallplatforms(package)

    Best Regards

    David
    David Lyon, Apr 23, 2009
    #3
  4. Re: Python Packages : A loomingproblem? packages might no longer work? (well not on your platform orpython version anyway)

    On Wed, 22 Apr 2009 23:27:23 -0400, David Lyon wrote:


    >> Why? Why should every package on PyPI need to support all those Python
    >> versions? That should be the decision of the package maintainer. If
    >> they want to support every version of Python back to 1.0, they can, and
    >> if they want to only support version 2.5 that's fine too.

    >
    > Why shouldn't packages support more than one python version?


    They can. That depends on the developer of the package.


    > Looking at it conversely....
    >
    > Why should the package developer dictacte which python version the
    > package will run on ?


    Because they're the developer. Who else should decide what Python
    versions to support? What are you going to do, hold a gun to their head
    and force them to support Python 2.1 when they've written it using 2.6
    features?



    >> For all we know, three quarters of the packages on PyPI have never been
    >> tested *at all*.

    >
    > Right. Why not run some tests....


    Be my guest.


    >> What's the dire problem you are trying to solve?

    >
    > Backward and forward compatability of python package resources.


    That's not a problem, that's a feature.

    What's the *problem* that you are trying to solve? What bad thing will
    happen if we don't build your proposed system?


    >> What "Super-Computers" do you know of that run Python?

    >
    > Google. Amazon web services..


    They're not super computers. They're distributed networks of "regular"
    computers.


    >> Who is paying for this?

    >
    >>From as little as $30 per month. Funding isn't so much the issue.


    Whether it is $30 a month or $30,000 a month, who do you expect to pay it?


    >>> With this basic structure, a daily building/testing infrastructure
    >>> working across the different versions of python and operating systems,
    >>> could easily become a reality.

    >>
    >> Oh yeah, "easily".

    >
    > pypi_packagelist = getallpypipackages()
    >
    > for package in pypi_packagelist:
    > testpackageonallplatforms(package)



    You don't have either getallpypipackages() or
    testpackageonallplatforms(), so this is just Py in the Sky (pun intended)
    fantasizing. But okay, for the sake of the argument, let's pretend you
    have your cluster, and your test suite. You run the test, and get a list
    of 18,000 errors (that's an average of three errors per package). Now
    what?




    --
    Steven
    Steven D'Aprano, Apr 23, 2009
    #4
  5. Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    On Wed, Apr 22, 2009 at 9:33 PM, David Lyon <> wrote:
    > Hi Steve,
    >>> Why should the package developer dictacte which python version the
    >>> package will run on ?

    >>
    >> Because they're the developer. Who else should decide what Python
    >> versions to support?

    > The developer shouldn't be making such decisions at all....
    > What hardware or operating systems we run his/her programs on is
    > up to the real world to decide.
    > If you think about it logically... why are we even asking our
    > developers to even "build" their packages for specific python
    > versions in the first place?
    > They should just:


    Programmers need to know what they are programming for. Each version
    of the language changes things, and the old interpreters often cannot
    understand things introduced in later versions. There is no way to
    magically get code written for one version to work for a different
    version. Code written on any python less than 3 will work on any
    higher python version up to (but not including) 3, and the same is
    true for the 3 series, but going back down needs somebody to do the
    conversion, unless the programmer was meticulous in making it work on
    all versions.

    This problem of course, isn't just looming, it's been around for a
    while. Every time a new python version is introduced, there is chaos
    as some library writers move over, and some stay with the old, and
    others bend over backwards to support everybody. Unfortunately, it
    can't be helped, and the best people to handle the situation are the
    library authors themselves. The only way to prevent this would be to
    not release any new versions ever. In the case of python 3, there are
    probably some who would prefer to have no new versions ever, but no
    progress can be made without asking the question to move along with
    the moving target or stay comfortable.

    Now, a system to give developers access to more test configurations
    would help make it easier to support more systems. This is the
    approach of snakebite. But some giant network that automatically tests
    all of pypi on all configurations is infeasible, somehow automatically
    forcing libraries to work is not even possible.
    Patrick Mullen, Apr 23, 2009
    #5
  6. Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    >>> Why? Why should every package on PyPI need to support all those Python
    >>> versions? That should be the decision of the package maintainer. If
    >>> they want to support every version of Python back to 1.0, they can, and
    >>> if they want to only support version 2.5 that's fine too.

    >>
    >> Why shouldn't packages support more than one python version?

    >
    > They can. That depends on the developer of the package.


    And if the developer decides to support multiple versions currently
    there is no infrastructural help that would be available to him. On
    the other hand, there is infrastructural help available to every
    developer for distributing their apps: it's PyPI itself. You could
    ask, why do we need PyPI? Let the developers distribute their code in
    any way they like, python.org doesn't need to support this.

    The OP is just thinking out loud that it would be great if developers
    could count on some help for testing various platforms and versions.
    And I agree, it would indeed be great.

    >> Looking at it conversely....
    >>
    >> Why should the package developer dictacte which python version the
    >> package will run on ?

    >
    > Because they're the developer. Who else should decide what Python
    > versions to support? What are you going to do, hold a gun to their head
    > and force them to support Python 2.1 when they've written it using 2.6
    > features?


    Nobody wants to point a gun to anyone's head. But if a developer says
    to himself (without anybody holding a gun, remember) that it would be
    great if there were ways to make sure that his code works on 2.5, 2.6
    and 2.7 (and not 2.4 or before for some reason) because he has good
    reasons to believe that users of all these 3 python versions want to
    use his code, then it would be really great if this developer had some
    help from python.org. He already has help for distribution (PyPI),
    community advise (comp.lang.python), job postings, etc, a good
    multiplatform/multiversion testing framework would be a nice addition.

    It wouldn't be mandatory for developers, just as it is not mandatory
    to upload packages to PyPI, but it would be nice if something like
    this is available to those developers who want to use it.

    >>> For all we know, three quarters of the packages on PyPI have never been
    >>> tested *at all*.

    >>
    >> Right. Why not run some tests....

    >
    > Be my guest.
    >
    >
    >>> What's the dire problem you are trying to solve?

    >>
    >> Backward and forward compatability of python package resources.

    >
    > That's not a problem, that's a feature.
    >
    > What's the *problem* that you are trying to solve? What bad thing will
    > happen if we don't build your proposed system?


    I fail to understand what is so mysterious about this. The problem is
    the following: developer X wants to support python versions a, b, c, d
    on platforms U, V, W. Developer X has no access to platforms V and W
    and also has no access to python versions b, c and d. Developer X can
    not run tests on these platforms and python versions although he
    really wants to. This situation is what is commonly referred to as a
    problem.

    >>> What "Super-Computers" do you know of that run Python?

    >>
    >> Google. Amazon web services..

    >
    > They're not super computers. They're distributed networks of "regular"
    > computers.
    >
    >
    >>> Who is paying for this?

    >>
    >>>From as little as $30 per month. Funding isn't so much the issue.

    >
    > Whether it is $30 a month or $30,000 a month, who do you expect to pay it?
    >
    >
    >>>> With this basic structure, a daily building/testing infrastructure
    >>>> working across the different versions of python and operating systems,
    >>>> could easily become a reality.
    >>>
    >>> Oh yeah, "easily".

    >>
    >> pypi_packagelist = getallpypipackages()
    >>
    >> for package in pypi_packagelist:
    >> testpackageonallplatforms(package)

    >
    >
    > You don't have either getallpypipackages() or
    > testpackageonallplatforms(), so this is just Py in the Sky (pun intended)
    > fantasizing. But okay, for the sake of the argument, let's pretend you
    > have your cluster, and your test suite. You run the test, and get a list
    > of 18,000 errors (that's an average of three errors per package). Now
    > what?


    This testing infrastructure should be available to developers who
    choose to opt in. Once they run their code and notice that there are
    errors, they will go back and fix these errors that otherwise would be
    undetected until a real person runs into them. Or they could clearly
    state what platform and python versions the code runs on for sure and
    on what platforms and python versions the code fails. This would be a
    great advantage to developers.

    Actually, the OP's suggestion is a really good one, the people behind
    snakebite.org realized this already some time ago. To me the idea is
    very clear and obviously would make the life of developers a whole lot
    easier and consequently the life of users as well.

    Cheers,
    Daniel



    --
    Psss, psss, put it down! - http://www.cafepress.com/putitdown
    Daniel Fetchinson, Apr 23, 2009
    #6
  7. Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    On Wed, 22 Apr 2009 23:47:08 -0700, Daniel Fetchinson wrote:

    >>>> Why? Why should every package on PyPI need to support all those
    >>>> Python versions? That should be the decision of the package
    >>>> maintainer. If they want to support every version of Python back to
    >>>> 1.0, they can, and if they want to only support version 2.5 that's
    >>>> fine too.
    >>>
    >>> Why shouldn't packages support more than one python version?

    >>
    >> They can. That depends on the developer of the package.

    >
    > And if the developer decides to support multiple versions currently
    > there is no infrastructural help that would be available to him. On the
    > other hand, there is infrastructural help available to every developer
    > for distributing their apps: it's PyPI itself. You could ask, why do we
    > need PyPI? Let the developers distribute their code in any way they
    > like, python.org doesn't need to support this.


    They don't *need* to support distribution. That they do is a nice-to-
    have, not a must-have -- there are other solutions, like SourceForge. We
    can be grateful for PyPI without believing that it's a must-have.


    > The OP is just thinking out loud that it would be great if developers
    > could count on some help for testing various platforms and versions. And
    > I agree, it would indeed be great.


    To me, the OP doesn't seem to be asking "Wouldn't it be good if
    developers could get access to whatever systems they wanted?" and more
    like "Somebody needs to develop this amazing technology that will just
    magically make every package on PyPI work on all these different
    platforms, otherwise the Sky Will Fall". Just look at the subject line:
    the issue is described as "a looming problem", it's suggested that
    packages "might no longer work".

    He's also worried about the combinational explosion of packages,
    platforms and versions, which to my mind is only a problem if you think
    that every package must work on every platform with every version of
    Python.

    I suspect you're primed to read the OP's suggestion in the most positive
    light, because you're already aware of snakebite, and so you're seeing
    his posts in terms of what snakebite is offering. I had never even heard
    of snakebite before now, so I was seeing it in a very different light.


    [...]
    >>>> What's the dire problem you are trying to solve?
    >>>
    >>> Backward and forward compatability of python package resources.

    >>
    >> That's not a problem, that's a feature.
    >>
    >> What's the *problem* that you are trying to solve? What bad thing will
    >> happen if we don't build your proposed system?

    >
    > I fail to understand what is so mysterious about this. The problem is
    > the following: developer X wants to support python versions a, b, c, d
    > on platforms U, V, W. Developer X has no access to platforms V and W and
    > also has no access to python versions b, c and d. Developer X can not
    > run tests on these platforms and python versions although he really
    > wants to. This situation is what is commonly referred to as a problem.


    That's *a* problem. It has many solutions, snakebite being just one. Is
    it the "looming" problem the OP is concerned about?


    > This testing infrastructure should be available to developers who choose
    > to opt in. Once they run their code and notice that there are errors,
    > they will go back and fix these errors that otherwise would be
    > undetected until a real person runs into them. Or they could clearly
    > state what platform and python versions the code runs on for sure and on
    > what platforms and python versions the code fails. This would be a great
    > advantage to developers.


    It certainly would.

    But the opt-in arrangement you are talking about doesn't sound much like
    the OP's suggestion that some cross-platform build-bot goes out and pulls
    in every single package on PyPI and tests them all on every combination
    of platforms and versions.


    > Actually, the OP's suggestion is a really good one, the people behind
    > snakebite.org realized this already some time ago. To me the idea is
    > very clear and obviously would make the life of developers a whole lot
    > easier and consequently the life of users as well.


    Snakebite certainly sounds beneficial for development of Python. I'm not
    sure that it would scale to every developer with a package on PyPI, but
    maybe someday.



    --
    Steven
    Steven D'Aprano, Apr 23, 2009
    #7
  8. Re: Python Packages : A looming problem?

    In message <>, David
    Lyon wrote:

    > There are now over 6,000 packages listed on PyPi - and this number can
    > only get bigger.
    >
    > Then, there are the three major operating systems:
    >
    > * Windows
    > * Mac
    > * Linix/Unix


    Major Linux distros provide 20,000 or more packages each, without breaking a
    sweat. What's your big deal?

    > To complicate the problem, there are now many versions of each operating
    > system.


    The Linux solution is to leave distro packaging to the distro maintainers.
    Release the source, and they will make up the necessary packages for their
    specific one-click installers. You don't have to worry about it.
    Lawrence D'Oliveiro, Apr 23, 2009
    #8
  9. David Lyon

    David Stanek Guest

    Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    On Thu, Apr 23, 2009 at 2:47 AM, Daniel Fetchinson
    <> wrote:
    >
    > The OP is just thinking out loud that it would be great if developers
    > could count on some help for testing various platforms and versions.
    > And I agree, it would indeed be great.
    >


    I think you interpreted the OP differently. As I said before the idea
    is not a bad one, but as a package developer I get to decide which
    platforms and versions my code supports.

    --
    David
    blog: http://www.traceback.org
    twitter: http://twitter.com/dstanek
    David Stanek, Apr 23, 2009
    #9
  10. David Lyon

    David Stanek Guest

    Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    On Thu, Apr 23, 2009 at 12:33 AM, David Lyon <> wrote:
    > Hi Steve,
    >>> Why should the package developer dictacte which python version the
    >>> package will run on ?

    >>
    >> Because they're the developer. Who else should decide what Python
    >> versions to support?

    > The developer shouldn't be making such decisions at all....
    > What hardware or operating systems we run his/her programs on is
    > up to the real world to decide.
    > If you think about it logically... why are we even asking our
    > developers to even "build" their packages for specific python
    > versions in the first place?
    > They should just:
    >


    You should abandon this argument. Your original idea of a huge build
    infrastructure had some merit. This is just off the wall.

    If I use win32com how do you expect me to support Linux? What about
    the many packages on PYPI containing C? What if I decide to write only
    to Python 3? Who will support the other platforms if not the
    developer?

    --
    David
    blog: http://www.traceback.org
    twitter: http://twitter.com/dstanek
    David Stanek, Apr 23, 2009
    #10
  11. David Lyon

    David Lyon Guest

    On Thu, 23 Apr 2009 07:04:35 -0400, David Stanek <>
    wrote:

    > If I use win32com how do you expect me to support Linux?


    Of course not...

    > What about the many packages on PYPI containing C?


    Exactly.

    > What if I decide to write only to Python 3?


    Fair enough. But don't forget it is open source.

    Let me ask these two questions...

    - What about the use case where somebody likes the code and wants
    to use it on Python 2.5?

    - Should not that user be able to share back with other
    Python 2.5 users?

    > Who will support the other platforms if not the developer?


    It's Open Source don't forget....

    Fact is these days.. developers come and go....

    If anything.... my suggestion promotes preserving the resources
    of the original developer rather than letting them expire just
    because their operating system does....

    (I'm talking windows here)

    David
    David Lyon, Apr 23, 2009
    #11
  12. Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    David Lyon wrote:

    >> What if I decide to write only to Python 3?

    >
    > Fair enough. But don't forget it is open source.


    So what?

    > Let me ask these two questions...
    >
    > - What about the use case where somebody likes the code and wants
    > to use it on Python 2.5?


    A patch, a fork, whatever.

    > - Should not that user be able to share back with other
    > Python 2.5 users?


    Probably. But not necessarily on the same project or repository.
    Especially if the port is not supported by the original/trunk developer.


    > If anything.... my suggestion promotes preserving the resources
    > of the original developer rather than letting them expire just
    > because their operating system does....


    I think your idea would be very costing in terms of man-years, and the
    net result would be... entropy, chaos, and a lot of CO2 :)
    Marco Mariani, Apr 23, 2009
    #12
  13. David Lyon

    Paul Boddie Guest

    Re: Python Packages : A looming problem?

    On 23 Apr, 10:59, Lawrence D'Oliveiro <l...@geek-
    central.gen.new_zealand> wrote:
    >
    > The Linux solution is to leave distro packaging to the distro maintainers.
    > Release the source, and they will make up the necessary packages for their
    > specific one-click installers. You don't have to worry about it.


    And the crucial message, that may not have sunk in yet amongst the
    many people messing around with "enhancing" distutils, is that in
    order for the distro maintainers to have something decent to work
    with, distutils and related tools should produce stuff which isn't too
    clever for its own good: "dist" as in "distribute", not as "install,
    but not uninstall, and then try and own everything from the
    developer's toolchain to the user's installation environment".

    Paul
    Paul Boddie, Apr 23, 2009
    #13
  14. Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    >>>>> Why? Why should every package on PyPI need to support all those
    >>>>> Python versions? That should be the decision of the package
    >>>>> maintainer. If they want to support every version of Python back to
    >>>>> 1.0, they can, and if they want to only support version 2.5 that's
    >>>>> fine too.
    >>>>
    >>>> Why shouldn't packages support more than one python version?
    >>>
    >>> They can. That depends on the developer of the package.

    >>
    >> And if the developer decides to support multiple versions currently
    >> there is no infrastructural help that would be available to him. On the
    >> other hand, there is infrastructural help available to every developer
    >> for distributing their apps: it's PyPI itself. You could ask, why do we
    >> need PyPI? Let the developers distribute their code in any way they
    >> like, python.org doesn't need to support this.

    >
    > They don't *need* to support distribution. That they do is a nice-to-
    > have, not a must-have -- there are other solutions, like SourceForge. We
    > can be grateful for PyPI without believing that it's a must-have.


    Nobody believes PyPI is a must-have and for sure I didn't say it is a
    must-have. PyPI is nice to have not only for developers but also for
    users. A multiplatform/multiversion test farm would also be nice for
    developers and users alike.

    >> The OP is just thinking out loud that it would be great if developers
    >> could count on some help for testing various platforms and versions. And
    >> I agree, it would indeed be great.

    >
    > To me, the OP doesn't seem to be asking "Wouldn't it be good if
    > developers could get access to whatever systems they wanted?" and more
    > like "Somebody needs to develop this amazing technology that will just
    > magically make every package on PyPI work on all these different
    > platforms, otherwise the Sky Will Fall". Just look at the subject line:
    > the issue is described as "a looming problem", it's suggested that
    > packages "might no longer work".


    You and I probably have a different approach to posts to c.l.p. I try
    to interpret things in the best possible light and get the most out of
    a suggestion. You are right, the OP was not formulating his proposal
    clearly and didn't use a sophisticated enough language but what is
    clear is that at the core of the proposal there is something valuable.
    Having a test farm would be very useful and would benefit the python
    community.

    > He's also worried about the combinational explosion of packages,
    > platforms and versions, which to my mind is only a problem if you think
    > that every package must work on every platform with every version of
    > Python.


    I don't think every package should work on every platform and with
    every version of python. But I do think that many developers want to
    support more platforms and versions than what they have access to.
    Having a test farm would be beneficial to these developers and their
    users.

    > I suspect you're primed to read the OP's suggestion in the most positive
    > light, because you're already aware of snakebite, and so you're seeing
    > his posts in terms of what snakebite is offering. I had never even heard
    > of snakebite before now, so I was seeing it in a very different light.


    I swear I didn't read this paragraph when I wrote above that I usually
    try to interpret posts in the best possible light! This is actually
    very funny, you used the exact same phrase :) But even if you have
    never heard of snakebite it is simply beyond my understanding how a
    developer can argue against a testing farm like this. You seriously
    think it would not be a good idea?


    >>>>> What's the dire problem you are trying to solve?
    >>>>
    >>>> Backward and forward compatability of python package resources.
    >>>
    >>> That's not a problem, that's a feature.
    >>>
    >>> What's the *problem* that you are trying to solve? What bad thing will
    >>> happen if we don't build your proposed system?

    >>
    >> I fail to understand what is so mysterious about this. The problem is
    >> the following: developer X wants to support python versions a, b, c, d
    >> on platforms U, V, W. Developer X has no access to platforms V and W and
    >> also has no access to python versions b, c and d. Developer X can not
    >> run tests on these platforms and python versions although he really
    >> wants to. This situation is what is commonly referred to as a problem.

    >
    > That's *a* problem.


    Yes, a problem, that is why I wrote, a problem.

    > It has many solutions, snakebite being just one.


    Yes, something integrated into PyPI would be another one.

    > Is it the "looming" problem the OP is concerned about?


    What are we worried about, the use of the English language or the
    technical merit of a proposal?

    >> This testing infrastructure should be available to developers who choose
    >> to opt in. Once they run their code and notice that there are errors,
    >> they will go back and fix these errors that otherwise would be
    >> undetected until a real person runs into them. Or they could clearly
    >> state what platform and python versions the code runs on for sure and on
    >> what platforms and python versions the code fails. This would be a great
    >> advantage to developers.

    >
    > It certainly would.


    Great, so we agree!

    > But the opt-in arrangement you are talking about doesn't sound much like
    > the OP's suggestion that some cross-platform build-bot goes out and pulls
    > in every single package on PyPI and tests them all on every combination
    > of platforms and versions.


    Yes, an automated process for all packages and all platforms and all
    versions would be foolish.


    >> Actually, the OP's suggestion is a really good one, the people behind
    >> snakebite.org realized this already some time ago. To me the idea is
    >> very clear and obviously would make the life of developers a whole lot
    >> easier and consequently the life of users as well.

    >
    > Snakebite certainly sounds beneficial for development of Python. I'm not
    > sure that it would scale to every developer with a package on PyPI, but
    > maybe someday.


    That is why the proposal is an important one. Let's say snakebite is
    not suitable for PyPI. What can be done on PyPI that helps developers
    with multiplatform/multiversion testing?

    Cheers,
    Daniel


    --
    Psss, psss, put it down! - http://www.cafepress.com/putitdown
    Daniel Fetchinson, Apr 23, 2009
    #14
  15. Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    >> The OP is just thinking out loud that it would be great if developers
    >> could count on some help for testing various platforms and versions.
    >> And I agree, it would indeed be great.
    >>

    >
    > I think you interpreted the OP differently. As I said before the idea
    > is not a bad one, but as a package developer I get to decide which
    > platforms and versions my code supports.


    Yes, I also think the original idea of the OP is a good one. And I
    agree with you that the developers need to be in charge what
    platform/version their code supports. Trouble is that currently if
    they decide to support multiple versions and platform they don't have
    any help from python.org or PyPI. It would be great if after a
    developer decided to support some platforms and some versions there
    was some infrastructural help in place.

    Cheers,
    Daniel


    --
    Psss, psss, put it down! - http://www.cafepress.com/putitdown
    Daniel Fetchinson, Apr 23, 2009
    #15
  16. David Lyon

    Paul Boddie Guest

    Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    On 23 Apr, 17:46, Daniel Fetchinson <> wrote:
    >
    > You and I probably have a different approach to posts to c.l.p. I try
    > to interpret things in the best possible light and get the most out of
    > a suggestion.


    There may be merit in the suggestion, but there also has to be
    skepticism, even if it is regarded as "stop energy" in certain
    circles.

    [...]

    > That is why the proposal is an important one. Let's say snakebite is
    > not suitable for PyPI. What can be done on PyPI that helps developers
    > with multiplatform/multiversion testing?


    I think that PyPI plays an important role in publicising packages and
    getting them out there in their "raw" form, but it's not the centre of
    the universe. Maybe the answer doesn't involve PyPI at all or, more
    likely, involves PyPI as a peripheral thing from which people acquire
    those packages that have to work together on a specific platform. Such
    a solution wouldn't have that much to do with PyPI, other than letting
    people download large collections of packages very quickly,
    potentially from mirror sites, which is something that various GNU/
    Linux distribution services permit (and something which seems to have
    been a struggle with PyPI).

    Paul
    Paul Boddie, Apr 23, 2009
    #16
  17. David Lyon

    norseman Guest

    Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    David Lyon wrote:
    >
    > On Thu, 23 Apr 2009 07:04:35 -0400, David Stanek <>
    > wrote:
    >
    >> If I use win32com how do you expect me to support Linux?

    >
    > Of course not...
    >
    >> What about the many packages on PYPI containing C?

    >
    > Exactly.
    >
    >> What if I decide to write only to Python 3?

    >
    > Fair enough. But don't forget it is open source.
    >
    > Let me ask these two questions...
    >
    > - What about the use case where somebody likes the code and wants
    > to use it on Python 2.5?
    >
    > - Should not that user be able to share back with other
    > Python 2.5 users?
    >
    >> Who will support the other platforms if not the developer?

    >
    > It's Open Source don't forget....
    >
    > Fact is these days.. developers come and go....
    >
    > If anything.... my suggestion promotes preserving the resources
    > of the original developer rather than letting them expire just
    > because their operating system does....
    >
    > (I'm talking windows here)


    Go David! I like your attitude!!

    >
    > David
    >
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >

    ==============================================

    This topic has been going round and round and yet nobody has actually
    touched the "magic button".

    Like whoever started this topic (and the rest of you) I have logged in
    somewhere and looked for Help, Snippets and more. I find something I
    like and download and it fails miserably. Thus the perceived "need" for
    some big brother certification thing.

    It is far more simple to police ourselves.
    The posting needs a DATE. The Help or Comment needs to explicitly
    specify the VERSION(s) it addresses. The code needs to state OS and
    program and version used to write it. And from there - user beware.

    DATE is supplied for date of posting, but who says that was when the
    contents were created? The content needs to specify its creation DATE.

    VERSION(s) which HELP and COMMENTS were created under/towards/"at time
    of" need explicit noting. Even one liners in postings.

    The author of the code needs to state OS (and version of) and the
    compiler/interpreter (and version of) and version the code was written
    to. (was it for a specific only or was the effort to be more general or
    was backward compatibility attempted, is the snippet even supposed to
    run?) As for programming language version compatibility - you the author
    need to download them and do that yourself if you need to know.

    There is one other thing that really pisses me off when it's missing.
    The idiot puts in the disclaimer but never tells anyone what the hell
    that damn thing is supposed to do. AAARRRRRGGGggggghhh !!!

    BB's, User Lists, all repositories can make these required before
    acceptance.


    On the dream list: It would be nice to have access to an OS (and or
    machine architecture) not in possession for testing compatibility in
    cases where it's an issue.
    But then just asking usually gets it done. :)

    Today is: 20090423
    Steve
    norseman, Apr 23, 2009
    #17
  18. David Lyon

    David Stanek Guest

    Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    On Thu, Apr 23, 2009 at 1:12 PM, norseman <> wrote:
    >
    > BB's, User Lists, all repositories can make these required before
    > acceptance.
    >
    >


    This is open source. I volunteer my time on the projects that I
    maintain. If you don't like the quality or lack of documentations,
    tests, etc. Contribute. Or just don't use the software.

    What maybe another option is to have a karma rating on PYPI. Maybe
    based off of home much you are included in other packages or some
    other factors.

    --
    David
    blog: http://www.traceback.org
    twitter: http://twitter.com/dstanek
    David Stanek, Apr 23, 2009
    #18
  19. David Lyon

    norseman Guest

    Re: Python Packages : A looming problem? packages might no longerwork? (well not on your platform or python version anyway)

    David Stanek wrote:
    > On Thu, Apr 23, 2009 at 1:12 PM, norseman <> wrote:
    >> BB's, User Lists, all repositories can make these required before
    >> acceptance.
    >>
    >>

    >
    > This is open source. I volunteer my time on the projects that I
    > maintain. If you don't like the quality or lack of documentations,
    > tests, etc. Contribute. Or just don't use the software.
    >
    > What maybe another option is to have a karma rating on PYPI. Maybe
    > based off of home much you are included in other packages or some
    > other factors.
    > ^

    ..how much..
    (nice to know I'm not the only one that does that :)

    I find that a useful idea. A good concept for "retro grading".

    ============================================
    Whenever I run a team I make it a practice to never make the others do
    something I won't. I do ask for volunteers to do things I cannot.

    Lets start with - what's done is done. (What is out there is there.)
    If you think I'm somehow able to take a look at a plain piece of code
    you posted yesterday and know that it was written 26 years ago on a CP/M
    2.0 dual HD DS 8" floppy drive system using Microsoft Assembly for the
    Z80 chip, intended as a generic print driver for the IBM Seletric
    converted to operate with computers of the day - you have an unrealistic
    esteem of my abilities.

    If you won't make the request to those supplying things you maintain - I
    have an unrealistic esteem of your efforts in protecting Open Source and
    its use by others.

    Fair enough?

    Everyone OK?


    If ever you are in Sacramento, CA; E-me a note. There is a pub down the
    way from where I work. I'll buy you a pint. While there, perhaps we
    could write that crawler that collects for the "retro grading".


    Steve
    norseman, Apr 23, 2009
    #19
  20. David Lyon

    David Lyon Guest

    On Thu, 23 Apr 2009 14:02:19 -0700, norseman <> wrote:
    > If you think I'm somehow able to take a look at a plain piece of code
    > you posted yesterday and know that it was written 26 years ago on a CP/M
    > 2.0 dual HD DS 8" floppy drive system using Microsoft Assembly for the
    > Z80 chip, intended as a generic print driver for the IBM Seletric
    > converted to operate with computers of the day - you have an unrealistic
    > esteem of my abilities.


    Funny.... :)

    A build test under the last 5 significant versions of python interpretors
    would probably pick up the fact that it is dead code....

    hello:
    mov bh,25
    mov ah,bh

    you had cp/m version 2? - wow

    We're all ok... as far as we know

    David
    David Lyon, Apr 24, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ryan Moore

    might as well ask.....

    Ryan Moore, Jul 24, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    397
    Elliot M. Rodriguez
    Jul 24, 2003
  2. Skip Montanaro
    Replies:
    0
    Views:
    83
    Skip Montanaro
    Jul 8, 2013
  3. Ian Kelly
    Replies:
    0
    Views:
    91
    Ian Kelly
    Jul 8, 2013
  4. Skip Montanaro
    Replies:
    0
    Views:
    86
    Skip Montanaro
    Jul 8, 2013
  5. Joshua Landau
    Replies:
    0
    Views:
    89
    Joshua Landau
    Jul 8, 2013
Loading...

Share This Page