standard libraries don't behave like standard 'libraries'

S

Sriram Srinivasan

I guess why every programming language has some kind of a 'standard
library' built in within it.
In my view it must not be called as a 'library' at all. what it does
is like a 'bunch of built-in programs ready-made to do stuff'.

Lets see what a 'library' does:

1. offers books for customers
1.1 lets user select a book by genre, etc
1.2 lets user to use different books of same genre, etc
1.3 lets user to use books by same author, etc for different genre

2. keeps track of all the books + their genre
2.1 first knows what all books it has at present
2.2 when new book comes it is added to the particular shelf sorted by
genre,author,edition, etc.
2.3 when books become old they are kept separately for future
reference
2.4 very old books can be sent to a museum/discarded

I guess no standard library does the minimum of this but wants to be
called a library.

As a python user I always wanted the standard library to have such
features so the user/developer decides to use what set of libraries he
want.

consider the libraries for 2.5 ,2.6, 3K are all available to the user,
the user selects what he wants with something like.

use library 2.5 or use library 2.6 etc.

The 2 main things that the library management interface has to do is
intra library management and inter library management.

intra library mgmt- consider books to be different libraries
(standard, commercial, private, hobby, etc)
inter library mgmt- consider books to be modules inside a library
( standard, commercial, private, hobby, etc)

if somehow we could accomplish this kind of mother of a all plugin/ad-
hoc system that is a real breakthrough.

Advantages:

1. new modules can be added to the stream quickly
2. let the user select what he want to do
3. modules (that interdepend on each other) can be packed into small
distribution and added to the stream quickly without waiting for new
releases
4. solution to problems like py 2.x and 3.x
5. users can be up to date
6. documentation becomes easy + elaborate to users
7. bug managing is easy too
8. more feed back
9. testing also becomes easy
10. many more , i don't know.. you have to find.

Python already has some thing like that __future__ stuff. but my
question is how many people know that? and how many use that? most of
them wait until old crust gets totally removed. that is bad for user
and python. that is why problems like py2.x py3.x originate. If there
is a newer book collection it must always be available at the library.
i must not go to another library to get that book.

These are just my views on the state of the standard libraries and to
make them state-of-the-art..! ;)
 
D

Diez B. Roggisch

Sriram said:
I guess why every programming language has some kind of a 'standard
library' built in within it.
In my view it must not be called as a 'library' at all. what it does
is like a 'bunch of built-in programs ready-made to do stuff'.

Lets see what a 'library' does:

1. offers books for customers
1.1 lets user select a book by genre, etc
1.2 lets user to use different books of same genre, etc
1.3 lets user to use books by same author, etc for different genre

2. keeps track of all the books + their genre
2.1 first knows what all books it has at present
2.2 when new book comes it is added to the particular shelf sorted by
genre,author,edition, etc.
2.3 when books become old they are kept separately for future
reference
2.4 very old books can be sent to a museum/discarded

I guess no standard library does the minimum of this but wants to be
called a library.

As a python user I always wanted the standard library to have such
features so the user/developer decides to use what set of libraries he
want.

consider the libraries for 2.5 ,2.6, 3K are all available to the user,
the user selects what he wants with something like.

use library 2.5 or use library 2.6 etc.

The 2 main things that the library management interface has to do is
intra library management and inter library management.

intra library mgmt- consider books to be different libraries
(standard, commercial, private, hobby, etc)
inter library mgmt- consider books to be modules inside a library
( standard, commercial, private, hobby, etc)

if somehow we could accomplish this kind of mother of a all plugin/ad-
hoc system that is a real breakthrough.

Advantages:

1. new modules can be added to the stream quickly
2. let the user select what he want to do
3. modules (that interdepend on each other) can be packed into small
distribution and added to the stream quickly without waiting for new
releases
4. solution to problems like py 2.x and 3.x
5. users can be up to date
6. documentation becomes easy + elaborate to users
7. bug managing is easy too
8. more feed back
9. testing also becomes easy
10. many more , i don't know.. you have to find.

Python already has some thing like that __future__ stuff. but my
question is how many people know that? and how many use that? most of
them wait until old crust gets totally removed. that is bad for user
and python. that is why problems like py2.x py3.x originate. If there
is a newer book collection it must always be available at the library.
i must not go to another library to get that book.

You are greatly oversimplifying things, and ignoring a *lot* of issues
here. The reason for __future__ is that it can *break* things if new
features were just introduced. Take the with-statement, reachable in
python2.5 throug

from __future__ import with_statement

It introduces a new keyword, which until then could be happily used as
variable name.

So you can't arbirtarily mix code that is written with one or the other
feature missing.

Then there is the issue of evolving C-APIs (or ABI), wich makes modules
incompatible between interpreters.

And frankly, for most of your list I don't see how you think your
"approach" reaches the stated advantages. Why is documentation becoming
easier? Why bug managing? Why testing?

I'm sorry, but this isn't thought out in any way, it's just wishful
thinking IMHO.

Diez
 
S

Sriram Srinivasan

Sriram Srinivasan schrieb:















You are greatly oversimplifying things, and ignoring a *lot* of issues
here. The reason for __future__ is that it can *break* things if new
features were just introduced. Take the with-statement, reachable in
python2.5 throug

   from __future__ import with_statement

It introduces a new keyword, which until then could be happily used as
variable name.

So you can't arbirtarily mix code that is written with one or the other
feature missing.

Then there is the issue of evolving C-APIs (or ABI), wich makes modules
incompatible between interpreters.

And frankly, for most of your list I don't see how you think your
"approach" reaches the stated advantages. Why is documentation becoming
easier? Why bug managing? Why testing?

I'm sorry, but this isn't thought out in any way, it's just wishful
thinking IMHO.

Diez

I don't know if you have used Dev-C++.<http://www.bloodshed.net/dev/
packages/index.html> It has a 'package management' mechanism for the
standard libraries.
please see the <http://devpaks.org/> webpage where all the packaged
libraries are stored.

In python we have the PyPI which is equivalent to the http://devpacks.org
but in PyPI the packages are mostly user made applications.
What I want is similar to PyPI but for the python standard libraries,
so that they (libraries) are as add-on as possible.
check this out too.. <http://molhanec.net/devcpphelp/packages.php>


I guess you understand what I am thinking... and do pardon my english
too..
 
D

Diez B. Roggisch

Sriram said:
I don't know if you have used Dev-C++.<http://www.bloodshed.net/dev/
packages/index.html> It has a 'package management' mechanism for the
standard libraries.

No, it hasn't. It has packages for *additional* libraries. C++ has only
a very dim concept of standard-libraries. And those usually ship with
the compiler, as standard-libraries shipped with python.
please see the <http://devpaks.org/> webpage where all the packaged
libraries are stored.

In python we have the PyPI which is equivalent to the http://devpacks.org
but in PyPI the packages are mostly user made applications.
What I want is similar to PyPI but for the python standard libraries,
so that they (libraries) are as add-on as possible.
check this out too.. <http://molhanec.net/devcpphelp/packages.php>

Why do you want that? What is the actual issue you are addressing?
Python's strength is that it comes with a certain set of libs bundled.
Artificially to unbundle them, forcing users to install them separatly
makes little sense, if any. Sure, updating a bug might be *slightly*
easier, but then the standard-libraries are well-tested and decoupling
their evolving from the releases of the core interpreter opens up a
whole can of worms of problems.

There is a tradeoff for both approaches, and one can argue if the
current balance is the right one - but if so, then over concrete
packages, not over the system in general.


Diez
 
S

Sriram Srinivasan

Sriram Srinivasan schrieb:















You are greatly oversimplifying things, and ignoring a *lot* of issues
here. The reason for __future__ is that it can *break* things if new
features were just introduced. Take the with-statement, reachable in
python2.5 throug

   from __future__ import with_statement

It introduces a new keyword, which until then could be happily used as
variable name.

So you can't arbirtarily mix code that is written with one or the other
feature missing.

Then there is the issue of evolving C-APIs (or ABI), wich makes modules
incompatible between interpreters.

And frankly, for most of your list I don't see how you think your
"approach" reaches the stated advantages. Why is documentation becoming
easier? Why bug managing? Why testing?

I'm sorry, but this isn't thought out in any way, it's just wishful
thinking IMHO.

Diez

__future__ as you said helps in stop breaking things. what i suggest
that if both the libraries (in yr case-with is defined in one and
other with is not defined is a simple one) like py2.x and py3.x exists
and i want to use 3.x features in the morning then in the evening i
want to use 2.x or something like that just add on the library and i
get the require functionality
 
D

Diez B. Roggisch

Sriram said:
__future__ as you said helps in stop breaking things. what i suggest
that if both the libraries (in yr case-with is defined in one and
other with is not defined is a simple one) like py2.x and py3.x exists
and i want to use 3.x features in the morning then in the evening i
want to use 2.x or something like that just add on the library and i
get the require functionality

This doesn't make sense to me. What are you doing in the morning, and
what in the evening, and to what extend is the code the same or are you
talking about different pieces of code?

Diez
 
S

Sriram Srinivasan

Sriram Srinivasan schrieb:





This doesn't make sense to me. What are you doing in the morning, and
what in the evening, and to what extend is the code the same or are you
talking about different pieces of code?

Diez


ok let me make it more clear..
forget how you use python now.. i am talking about __futuristic__
python programming.

there is no more python2.x or python3.x or python y.x releases. there
is only updates of python and standard library say 1.1.5 and 1.1.6.
let the difference be an old xml library updated with a new regex
support.

i am coding my program now.
i want my application to be compatible with 1.1.5 library

use library 1.1.5
import blah form blahblah
....
....

i cannot use regex feature of xml in this application
i then update my library in the evening
now both the libraries are present in my system.
now i try using the new feature

use library 1.1.6 #thats all now i get all features
import blah from blahblah
....
....
 
D

Diez B. Roggisch

ok let me make it more clear..
forget how you use python now.. i am talking about __futuristic__
python programming.
>

there is no more python2.x or python3.x or python y.x releases. there
is only updates of python and standard library say 1.1.5 and 1.1.6.
let the difference be an old xml library updated with a new regex
support.

i am coding my program now.
i want my application to be compatible with 1.1.5 library

use library 1.1.5
import blah form blahblah
...
...

i cannot use regex feature of xml in this application
i then update my library in the evening
now both the libraries are present in my system.
now i try using the new feature

use library 1.1.6 #thats all now i get all features
import blah from blahblah

This is not futuristic, this is state of the art with PyPI & setuptools.

You still haven't addressed all the issues that arise though, regarding
different python interpreter versions having different syntax and ABI.

Diez
 
S

Steven D'Aprano

I guess why every programming language has some kind of a 'standard
library' built in within it.
In my view it must not be called as a 'library' at all. what it does is
like a 'bunch of built-in programs ready-made to do stuff'.

Lets see what a 'library' does:

1. offers books for customers
[...]


You are describing a lending library, which is not the only sort of
library. My personal library doesn't do any of those things. It is just a
room with shelves filled with books.

Words in English can have more than one meaning. Horses run,
refrigerators run, and even though they don't have either legs or motors,
programs run too. The argument that code libraries don't behave like
lending libraries won't get you anywhere.

As a python user I always wanted the standard library to have such
features so the user/developer decides to use what set of libraries he
want.

consider the libraries for 2.5 ,2.6, 3K are all available to the user,
the user selects what he wants with something like.

use library 2.5 or use library 2.6 etc.

Since library features are tied closely to the features available in the
Python interpreter, the way to use library 2.5 is to use Python 2.5. You
*might* be able to use library 2.5 with Python 2.6 or 2.4; you absolutely
won't be able to use it with Python 1.5 or 3.1.
 
S

Sriram Srinivasan

This is not futuristic, this is state of the art with PyPI & setuptools.

You still haven't addressed all the issues that arise though, regarding
different python interpreter versions having different syntax and ABI.

Diez

haha... it would be awesome if they implement it in the 'future' .. i
posted the same to (e-mail address removed), it seems the distutil is
getting a big overhaul. (i hope they make a new uninstall option for
setuptools n easy_install) they say there are many ways to do that
using pkg tools... but python wants one and only one way- the python
way.
regarding issues:

1.security is always a issue
2. regarding problems like with statement you mentioned earlier.. i
think there will be a basic feature set that is common for all
versions of add-on libraries.
3.when you want the new feature you have to update your python
interpreter

use interpreter 1.5.2

may trigger the proper interpreter plugin(responsible for additional
feature) to load and add functionality.. its simple to say but it is
hard to make the compiler pluggable, may be they figure out.

use library x.y.z

while issuing this command the default interpreter has to
automatically resolve dependencies of the core c/cpp static libraries
and other shared libraries. so that must not be an issue. if they have
implemented this much, dep solving is nothing.

futuristic python may also contain an option for compiling a module
into a static library. so we can code python libraries in python
(mostly) itself. think of pypy. it is already state of the art.
 
S

Sriram Srinivasan

You are describing a lending library, which is not the only sort of
library. My personal library doesn't do any of those things. It is just a
room with shelves filled with books.

how i see is all libraries are libraries, for a personal library you
are the only customer and you are the management too.!
Words in English can have more than one meaning. Horses run,
refrigerators run, and even though they don't have either legs or motors,
programs run too. The argument that code libraries don't behave like
lending libraries won't get you anywhere.

first this is not an argument at all...i clearly stated these were my
imaginations cum ideas.
Since library features are tied closely to the features available in the
Python interpreter, the way to use library 2.5 is to use Python 2.5. You
*might* be able to use library 2.5 with Python 2.6 or 2.4; you absolutely
won't be able to use it with Python 1.5 or 3.1.

i am not talking about the past..past were experiences! that the
developers had and what is happening today *might be* another
experience for the future.
 
D

Diez B. Roggisch

Sriram said:
haha... it would be awesome if they implement it in the 'future' .. i
posted the same to (e-mail address removed), it seems the distutil is
getting a big overhaul. (i hope they make a new uninstall option for
setuptools n easy_install) they say there are many ways to do that
using pkg tools... but python wants one and only one way- the python
way.
regarding issues:

1.security is always a issue
2. regarding problems like with statement you mentioned earlier.. i
think there will be a basic feature set that is common for all
versions of add-on libraries.

So all libraries written have to use the common subset, which - unless
things are *removed*, which with python3 actually happened - is always
the oldest interpreter. And if a feature goes away, they have to be
rewritten with the then common subset.

In other words: as a library writer, I can't use shiny, new & better
features, I'm stuck with the old stuff, and whenever the intepreter
evolves I have to rewrite even the old ones. Not appealing. Why develop
the interpreter at all then?
3.when you want the new feature you have to update your python
interpreter

use interpreter 1.5.2

may trigger the proper interpreter plugin(responsible for additional
feature) to load and add functionality.. its simple to say but it is
hard to make the compiler pluggable, may be they figure out.
>

use library x.y.z

while issuing this command the default interpreter has to
automatically resolve dependencies of the core c/cpp static libraries
and other shared libraries. so that must not be an issue. if they have
implemented this much, dep solving is nothing.

In other words: the problem is solved by somehow solving the problem -
but not by a concrete solution you propose?
futuristic python may also contain an option for compiling a module
into a static library. so we can code python libraries in python
(mostly) itself. think of pypy. it is already state of the art.
PyPy is cool, but by far not advanced enough to replace external,
C-based libraries such as NUMPY and PyQt and whatnot.

I don't say that the current situation is by any means ideal. There is a
lot to be said about package creation, distribution, installation,
removal, dependency handling, and even system-packaging-integration if
one likes.

You IMHO just add another layer of complexity on top of it, without
proposing solutions to any of the layers in between, nor your new one -
namely, the interpreter version agnosticism.

Diez
 
S

Sriram Srinivasan

So all libraries written have to use the common subset, which - unless
things are *removed*, which with python3 actually happened - is always
the oldest interpreter. And if a feature goes away, they have to be
rewritten with the then common subset.

you see that's the problem with py3. instead of slowly removing the
features one by one till the end they made a jump. the problem is not
with the jump but with other libraries/packages that depend on py2.
what i felt was if the new version was available as soon as it is into
the stream, developers/users try to use them and get accustomed. now
they follow this. they have frozen py3 for may be 2-3 years so ppl
would move slowly to py2.6 ,2.7, 2.8... most. also the corporate
companies are the most influential.

also a point to note: if new updates keep on coming (for eg. daily
antivirus update) developers tend to accept and move on *easily*.

i am not an expert in programming/interpreter design/library writing/
packaging. i just wanted the standard library *too* to be pluggable,
not just the applications based on it.
i am not even experienced in python to make bold accusations so all i
can say is 'how it would be if?' and i always stick to one thing 'why
not?'.
In other words: as a library writer, I can't use shiny, new & better
features, I'm stuck with the old stuff, and whenever the interpreter
evolves I have to rewrite even the old ones. Not appealing. Why develop
the interpreter at all then?

if you are app developer you needn't rewrite anything. that is the
main point here! just specifying the interpreter and library version
is enough every thing has to work like magic. as the previous
library&interpreter wont be removed (unless and until it is your will)
u can use them. thats how everybody would like it to be.

as for a library writer is concerned: as you say it depends on certain
things like other libraries,interpreter,etc. but i would like to say
that change is inevitable. if some core developer has to change
something he might change. and if you have to update your library you
have to. this applies to the interpreter too. what python now does is
make every update work with the minimal set (giving minimal backward
compatibility) plus the new features too. which is exactly the right
thing. what i wanted is the users have to be aware now and then when
of the modification, deletion, addition stuff not in every release as
in py3k. then the whole backward incompatibility issue would be
*dissolved* or what i mean is *diffused* and ppl dont even know that
they have switched over to a very new thing.


In other words: the problem is solved by somehow solving the problem -
but not by a concrete solution you propose?

as i told before i neither know the problem nor the solution. i just
wrote my ideas/imagination down
PyPy is cool, but by far not advanced enough to replace external,
C-based libraries such as NUMPY and PyQt and whatnot.

I don't say that the current situation is by any means ideal. There is a
lot to be said about package creation, distribution, installation,
removal, dependency handling, and even system-packaging-integration if
one likes.

You IMHO just add another layer of complexity on top of it, without
proposing solutions to any of the layers in between, nor your new one -
namely, the interpreter version agnosticism.

yes you are right. python has a lot of bindings to a lot of stuff. but
it is the strength and weakness. not exactly pythons weakness but with
the thing on the other side. yes it may be looking complex because
most of the 'standard library' system was not designed to be adhoc/add-
on/pluggable from the start.
 
B

Benjamin Kaplan

you see that's the problem with py3. instead of slowly removing the
features one by one till the end they made a jump. the problem is not
with the jump but with other libraries/packages that depend on py2.
what i felt was if the new version was available as soon as it is into
the stream, developers/users try to use them and get accustomed. now
they follow this. they have frozen py3 for may be 2-3 years so ppl
would move slowly to py2.6 ,2.7, 2.8... most. also the corporate
companies are the most influential.

The freeze was put in place so that the other implementations of
Python, like Jython and IronPython, have a chance to catch up with the
reference CPython implementation. It's not so people will slowly move
up. FWIW, people knew for years what was going to change in Python 3.
also a point to note: if new updates keep on coming (for eg. daily
antivirus update) developers tend to accept and move on *easily*.

i am not an expert in programming/interpreter design/library writing/
packaging. i just wanted the standard library *too* to be pluggable,
not just the applications based on it.
i am not even experienced in python to make bold accusations so all i
can say is 'how it would be if?' and i always stick to one thing 'why
not?'.


if you are app developer you needn't rewrite anything. that is the
main point here! just specifying the interpreter and library version
is enough every thing has to work like magic. as the previous
library&interpreter wont be removed (unless and until it is your will)
u can use them. thats how everybody would like it to be.

So you're saying that we'd have multiple versions of the interpreter
running at the same time, but they all have access to the same memory.
That wouldn't just require a change to Python, that would require a
change to the way all modern operating systems work.
as for a library writer is concerned: as you say it depends on certain
things like other libraries,interpreter,etc. but i would like to say
that change is inevitable. if some core developer has to change
something he might change. and if you have to update your library you
have to. this applies to the interpreter too. what python now does is
make every update work with the minimal set (giving minimal backward
compatibility) plus the new features too. which is exactly the right
thing. what i wanted is the users have to be aware now and then when
of the modification, deletion, addition stuff not in every release as
in py3k. then the whole backward incompatibility issue would be
*dissolved* or what i mean is *diffused* and ppl dont even know that
they have switched over to a very new thing.

Actually, that's not entirely true. New versions to break things
Consider these statements

as = ['a','a','a']
with = [1,2,3]
It's legal in Python 2.4 or earlier, a warning in Python 2.5, but
illegal in Python 2.6
as i told before i neither know the problem nor the solution. i just
wrote my ideas/imagination down


yes you are right. python has a lot of bindings to a lot of stuff. but
it is the strength and weakness. not exactly pythons weakness but with
the thing on the other side. yes it may be looking complex because
most of the 'standard library' system was not designed to be adhoc/add-
on/pluggable from the start.


It is pluggable. The standard library consists of Python modules like
any other. 2.6's multiprocessing module is just an incorporation of
the pyprocessing module for instance. The point of the standard
library is that you can count on it being there, and you can count on
it having certain features, given your interpreter version. You can
also be sure that anytime a new version of Python comes out, those
modules will be compatible.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,832
Latest member
GlennSmall

Latest Threads

Top