improving the Python docs -- a wiki? copy PHP's model?

[Stephen Ferg]

I've been wondering how we can improve the quality of the Python
documentation.

The problem with the Python documentation is that it is not being
handled like an open source product. The hallmark of open-source
development is that many people contribute. The Python docs in
contrast are being maintained by a very small group.

I'm sure that many people would like to contribute to the improvment
of the documentation... the problem is that there is no way that they
can do so easily. The solution is to move to a different model of the
documentation process, by developing a way to make it easy for
everyone in the Python community to contribute to the documentation.

One solution is a documentation wiki: see the request (below) for a
"corresponding wikiable page for every page here in the pydocs." A
similar but better solution is to follow the model created by the PHP
community. If you follow the link to
http://us2.php.net/manual/en/function.sort.php you'll see how
impressive the results can be. Something like this is clearly what we
need.

So I put this question to the entire Python community. Can we create
an interactive site, and the supporting organizational infrastructure,
that will enable everyone in the Python community easily to contribute
to the Python documentation?

-- Steve (Stephen Ferg ... (e-mail address removed))

---------------------------------------------------------------

Here are some recent posts on this subject.

---------------------------------------------------------------


from:
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&[email protected]

I have found *by far* that the best docs are for PHP. It has little to
do with the actual official content (which is fairly complete and well
written and almost always includes an example), but a lot to do with
how anybody can nnotate the pages, wiki style. Here is an example for
the sort function:
http://us2.php.net/manual/en/function.sort.php
Having all those comments is really helpful, and they also help for
updating the documentation. If there are a lot of comments against a
page then the main body really needs some more work!

-------------------------------------------------------------

from:
http://groups.google.com/[email protected]

One thing I like about PHP's docs is that it allows users to post
comments, sugestions, extra information, etc to the docs online. Now
and then someone goes through and edits the useful bits into the
official documentation and flushes the comments all back out leaving
them clear for new people to post their own. Several times I've
attached comments as to how to compile a module or use some poorly
documented feature and have had lots of people write to thank me for
how helpful those comments were. I'd imagine the same process could be
useful in Python.

-----------------------------------------------------------

from:
http://www.python.org/cgi-bin/moinmoin/PythonDocumentation

Discussion

Please!! get a wiki running side-by-side with the docs: e.g. one
corresponding wikiable page for every page here in the pydocs. The
startup data is simply that of the docs, so just mirror their
structure. Then after the wiki docs start offering lots of useful
information, the actual doc writers can incorporate this. To
summarize, I think Python's documentation (like the library reference)
is basically a big melange of api data and some snippets of useful
info. This wiki will help replace those places that just use API data
to fill up space, and turn it into something better.

Like the PHP Manual at http://www.php.net/.


I think that's a great idea. But I think it should appear on a
separate wiki. We're going to need to do some steering for this wiki:
If every python-related project in the world tries to make their wiki
a walled garden within this wiki, we're going to run into problems.
RecentChanges will be a total mess of a zillion projects. Let's talk.
-- LionKimbro 2003-10-02 03:15:01

 

[Andrew Bennetts]

I've been wondering how we can improve the quality of the Python
documentation.

The problem with the Python documentation is that it is not being
handled like an open source product. The hallmark of open-source
development is that many people contribute. The Python docs in
contrast are being maintained by a very small group.

The Python docs accepts contributions exactly the same way the Python
interpreter and library does: by patches and bugs submitted to the
SourceForge tracker (take a look, there have been quite a few there).

I'm sure that many people would like to contribute to the improvment
of the documentation... the problem is that there is no way that they
can do so easily. The solution is to move to a different model of the
documentation process, by developing a way to make it easy for
everyone in the Python community to contribute to the documentation.

Is submitting a bug or patch to sourceforge really that much of a barrier?

One solution is a documentation wiki: see the request (below) for a
"corresponding wikiable page for every page here in the pydocs." A
similar but better solution is to follow the model created by the PHP
community. If you follow the link to
http://us2.php.net/manual/en/function.sort.php you'll see how
impressive the results can be. Something like this is clearly what we
need.

Sorry, if the official docs lack sufficient description and examples to be
clear on how to use a sort function, then the docs should be fixed by
someone willing to spend the time to write a coherent addition to (or even
just rewording of) the existing documentation, not augmented by twenty times
as much commentary on randomly varying related topics. As a reader, I
prefer properly written and edited and officially approved documentation to
ten pages of advice from people I don't know and trust -- I've often seen
well-meaning novices give bad advice on newsgroups and mailing lists.

As a reader, I also don't want to see a tutorial when I'm looking for
reference documentation. Thus most of those comments are completely
irrelevant to me a lot of the time.

In fact, those comments look like conversations that already occur on the
newsgroup -- which is already a community-driven documentation resource,
handily indexed by google. Some of them look like recipes -- but we already
have http://aspn.activestate.com/ASPN/Python/Cookbook/ for that (and I've
never seen 30+ line functions for case-insensitive sorting in our
cookbook).

So I put this question to the entire Python community. Can we create
an interactive site, and the supporting organizational infrastructure,
that will enable everyone in the Python community easily to contribute
to the Python documentation?

I've never felt that the Python documentation was particularly lacking,
and searching google has always filled in few the gaps for me. I guess
you're trying to solve a problem I don't have. Maybe I'm unusual.

-Andrew.

 

[John Roth]

I've been wondering how we can improve the quality of the Python
documentation.

Frankly, I think it's pretty good as it stands. There are some places
where it could be improved: documenting new style classes is an
outstanding example, but by and large it's better than most efforts
I've seen.

The problem with the Python documentation is that it is not being
handled like an open source product. The hallmark of open-source
development is that many people contribute. The Python docs in
contrast are being maintained by a very small group.

You're confusing process with result.

I'm sure that many people would like to contribute to the improvment
of the documentation... the problem is that there is no way that they
can do so easily. The solution is to move to a different model of the
documentation process, by developing a way to make it easy for
everyone in the Python community to contribute to the documentation.

One solution is a documentation wiki: see the request (below) for a
"corresponding wikiable page for every page here in the pydocs." A
similar but better solution is to follow the model created by the PHP
community. If you follow the link to
http://us2.php.net/manual/en/function.sort.php you'll see how
impressive the results can be. Something like this is clearly what we
need.

I've looked at it. I'll certainly say it's impressive, but I doubt if
my impression is what you though it would be.

So I put this question to the entire Python community. Can we create
an interactive site, and the supporting organizational infrastructure,
that will enable everyone in the Python community easily to contribute
to the Python documentation?

As Andrew commented, there's nothing stopping you from
contributing. There's also nothing stopping you from setting
up a wiki yourself so people can comment on various pages
of the official documentation.

John Roth

 

[Roger Binns]

As Andrew commented, there's nothing stopping you from

contributing.

Ask anyone who works on PHP. Their system is really good
and greatly helps developer productivity. The people who
work on the doc incorporate the comments into the next
doc releases and usually wipe them clean.

It is trivial to add new comments. Sure one could do
it via SourceForge instead, but that is a far greater
hassle. Every single little step you put in the way
makes it less likely people will contribute. If you
are running cathedral style development, then that is
fine.

There's also nothing stopping you from setting
up a wiki yourself so people can comment on various pages
of the official documentation.

Yes, except the official doc would then need a link
at the bottom of each page pointing to the wiki. The
PHP docs have it all there in one page, no need to look
in two locations etc.

Roger

 

[Ville Vainio]

Roger> It is trivial to add new comments. Sure one could do
Roger> it via SourceForge instead, but that is a far greater
Roger> hassle. Every single little step you put in the way
Roger> makes it less likely people will contribute. If you
Roger> are running cathedral style development, then that is
Roger> fine.

I'm not sure I would like to see the official Python documentation
"improved" too much. At least the library reference is top notch as it
stands, and some other areas (distutils/embedding) should be covered
in seperate volumes of documentation, e.g. some wiki-maintained
tutorials.

Documentation just isn't a problem for core Python. The same can't be
said for various UI toolkits or third party libs, but that's a
separate problem.

 

[Peter Hansen]

Ask anyone who works on PHP. Their system is really good
and greatly helps developer productivity. The people who
work on the doc incorporate the comments into the next
doc releases and usually wipe them clean.

Clearly the example presented (for function.sort.php) is
an extreme exception to that rule then, containing as it
does user-contributed, uh, "assistance" dating back to March
of 2000...

-Peter

 

[Stephen Ferg]

Is submitting a bug or patch to sourceforge really that much of a barrier?

Yes, it is a significant barrier. It requires knowledge of both Latex
and a certain way of using diff. There are many, many Python users --
especially Python-on-Windows -- who are not familiar with these
technologies and don't have the time to learn them for the sole
purpose of making a contribution to the documentation.

As a reader, I
prefer properly written and edited and officially approved documentation to
ten pages of advice from people I don't know ...

I agree. But there are gaps in the documentation, where the
officially approved documentation is silent. In such cases, I prefer
advice from others, even though I've never met them, to no help at
all.

After all, what is comp.lang.python except advice from people that
you've (for the most part) never met.

I've never felt that the Python documentation was particularly lacking...

In general, I'd agree that the Python documentation is outstanding.
But there are gaps -- a qualifying phrase here, a missing reference
there, a small example in many places. Filling those gaps would make
the difference in those places between documentation that is just
barely adequate and documentation that is really excellent.

For experienced Pythonistas, this difference may be merely a minor
matter of convenience, but for newbie Pythonistas this may be the
difference between usable and not usable. And we want to support
newbies. We want to attract as many of them as possible. The larger
the newbie population today, the larger the active, experienced Python
community tomorrow.

And remember, it is the newbies who are finding the problems in the
documentation... they are the ones reading it, because they are the
ones that need it. As newbies, they may not yet be masters of all the
supporting technologies we use (see my earlier point about the
technical barriers being too high), but they are certainly the people
in the best position to find gaps in the documentation and to supply
material that others would find useful.

WE NEED TO MAKE IT EASIER FOR MANY HANDS -- INCLUDING NEWBIE HANDS --
TO CONTRIBUTE TO THE DOCUMENTATION.

The PHP model is a good one. Many hands contribute online.
Periodically, expert hands incorporate the best of the online
contributions into the mainline docs.

 

[Aahz]

Stephen deleted someone's attribution:

Yes, it is a significant barrier. It requires knowledge of both Latex
and a certain way of using diff. There are many, many Python users --
especially Python-on-Windows -- who are not familiar with these
technologies and don't have the time to learn them for the sole
purpose of making a contribution to the documentation.

Huh. Fred Drake has said for years that plain-text contributions are
welcome. I've submitted an SF bug to make that clear in the docs. ;-)

 

[Terry Reedy]

barrier?

Yes, it is a significant barrier. It requires knowledge of both Latex
and a certain way of using diff.

Neither are required to submit a doc bug report, which can then be used by
someone else with the required knowledge to actually make the fix. Typos
currently get fixed pretty quickly once reported.

A suggested sentence to replace an incorrect sentence may take longer to
get inserted. But I have helped fix a few reported more-than-one-word-typo
doc glitches, without knowing either Latex or diff, by submitting comments
with a suggested sentence or two and with specific suggestion locations.

In general, I'd agree that the Python documentation is outstanding.
But there are gaps -- a qualifying phrase here, a missing reference
there, a small example in many places. Filling those gaps would make
the difference in those places between documentation that is just
barely adequate and documentation that is really excellent.

I admit I don't know whether 'text patches' would be as welcome for
enhancements as for fixes. If I were inclined to submit such, I might
inquire first.

Terry J. Reedy

 

[Doug Holton]

Yes, except the official doc would then need a link
at the bottom of each page pointing to the wiki. The
PHP docs have it all there in one page, no need to look
in two locations etc.

I think it would be great to have moderated commentable docs like PHP.

I'm not sure what docs to start off with though. The Python reference
manual isn't written in a way that separates a page for each keyword or
builtin like PHP's.

Or should we start off with a combination of the manuals with an
automatically generated reference from the standard library like this
epydoc one: http://epydoc.sourceforge.net/stdlib/
or pydoc: http://pydoc.org/

Perhaps we could take all the reference documents and convert them from
html to wiki and load them into the python wiki or another wiki, with a
"comments" section added at the bottom of each page. We're talking
hundreds of pages though.

 

[Thomas Guettler]

Am Wed, 05 May 2004 06:16:06 -0700 schrieb Stephen Ferg:

Yes, it is a significant barrier. It requires knowledge of both Latex
and a certain way of using diff.

Hi Stephen,

You don't need to submit a patch to the latex source.

You can send plain text as a "feature request".

I already send some feature request which
were added to the docs (deadlock
problem, if not using popen4)

If you are unsure if it should be in the docs,
you can ask here before sending it to the
bug tracking system.

Thomas

 

[Marcel Molina Jr.]

I've been wondering how we can improve the quality of the Python
documentation.
[..snip..]

One solution is a documentation wiki: see the request (below) for a
"corresponding wikiable page for every page here in the pydocs." A
similar but better solution is to follow the model created by the PHP
community. If you follow the link to
http://us2.php.net/manual/en/function.sort.php you'll see how
impressive the results can be. Something like this is clearly what we
need.

if php's documentation is being presented as a sort of model in this
thread, then i would like to mention what *i* find the most helpful aspect
of its documentation system, far apart from there being a comment posting
system at the bottom of each page, which has been (as far as i can tell)
the primary feature that people would like adopted by python.

what really makes sense for me with php's documentation is how very
intuitive (and therefore easy and fast) it is to get to what one is
looking for. this is particularly true of its online documentation of
its built in functions. this ease of access is leveraged through their
websites way of dealing with 404 requests.

if you go to php.net/<function name> you are automatically redirected
to the page that documents that function.

this is often more efficient than the three step process of going to
the page, doing a search, clicking on the appropriate search result. it
is often faster than have a bookmark to a page that lists all functions
and then scrolling or even Ctrl-Fing your way to the appropriate one.

if you mistype <function name> (by putting an underscore, say, where there
is none, or visa versa), then you are presented with a page of the 20
closest matches. this can be taken advantage of when there are a set of
functions that all share the same prefix (say 'str' or 'pg' or 'array').

in addition, once you have made it to the function you are looking for,
the notion of categories is reflected in the left menu which will note
that you are on an "str' type function and will then list all string
functions for you.

the sites 404 handling also extends to more than built in function
detection. if you go to php.net/operators you are redirected to the
gateway to all documentation on operators. on the left, it provides a
menu of all other such gateway pages in the same categorical section
of the languages documentation, so there are links to similar gateway
pages such as "types", "variables", "constants", "control structures",
"classes and objects", etc. it recognizes from what "height" you are
approaching the language, which is helpful. in the case of a specific string
function (as mentioned above) it lists all string functions, in the case of
being on the operators page, it lists all other fundamental language
documentation pages.

this is not to say that this is revolutionary or unique. i suspect
(without the aid of explicitly *remembering*) that most language
documentation provides these things. my point about php's iteration of
this, though, is that the interface is very intuitive. it does a good job of
being psychic. this approach may used with other documentation systems, i
just don't know about them.

going to google will often get you within three clicks of what you are
looking for. that is pretty good. php's website often gets me to what
i am looking for in one try, though. when one is focusing on the problem
at hand, and making frequent documentation enquiries, getting to the
answers as quick as possible allows one to keep their thoughts primarily
on what the problem/question is, rather than having to break off and think
about the best ways of trying to find a given piece of documentation.

online documentation shouldn't be one's only recourse though, and
this is a place where python has a major leg up on php. one aspect to
python's documentation that i really appreciate is having the help()
function interface to pydoc now in the interpreter. whether it be ruby's
'ri', perl's 'perldoc' or python's 'pydoc', command line interfaces to
documentation are very helpful, both as a learning tool and a reference,
especially when they can be accessed from within an editor such as with
vi's Shift-K macro. php has no such interface.

pydoc from the interpreter through 'help()' often gets me to where i want
to go in one step, so no complaints there. it would be nice, though,
if there was the same kind of web site functionality at python.org
as there is at php.net (though i am not familiar with all the ways to
access documentation for either and there may be many means of getting
documentation that address my need that i am just not aware of).

i think ruby's "ri" is really great, though much of that functionality
seems to be more or less built into python through dir() and __doc__,
et al.

Summary:

php's web site affords the user a flexible approach toward accessing
documentation. once you actually *get there* you might like to have
comments at the bottom of the page, or you might like, on the other hand,
a more "official" reference style approach. regardless, though, you have
to *get there* first and php's website makes that very fast and intuitive.

i don't have issues with python's documentation, per se, the way that
the person who started this thread does. i just wanted to mention the
thing that i like about php's documentation which no one else has seemed
to mention.

thanks,
marcel

 

[Paul Boddie]

Clearly the example presented (for function.sort.php) is
an extreme exception to that rule then, containing as it
does user-contributed, uh, "assistance" dating back to March
of 2000...

It looks like a cross between a man page for a UNIX function that
either never really got standardised across System V, BSD and umpteen
other variants, or got overstandardised (hence the proliferation of
sort functions), combined with a developer's cookbook (eg. the
O'Reilly Python cookbook) but with less Reilly and more "Oh?".

Paul

 

[Michael Geary]

I've never felt that the Python documentation was

In general, I'd agree that the Python documentation is
outstanding. But there are gaps -- a qualifying phrase here,
a missing reference there, a small example in many places.
Filling those gaps would make the difference in those places
between documentation that is just barely adequate and
documentation that is really excellent.

For experienced Pythonistas, this difference may be merely a
minor matter of convenience, but for newbie Pythonistas this
may be the difference between usable and not usable. And
we want to support newbies. We want to attract as many of
them as possible. The larger the newbie population today, the
larger the active, experienced Python community tomorrow.

A good example of this just came up in this group a few days ago. A new
Python programmer was puzzled over this result:
'wsouthpine.com" {'

He did read the documentation for lstrip, but the doc doesn't make it at all
clear that the argument to lstrip is treated as a set of characters.

With the PHP documentation model, there would have been a user comment and
example attached to the lstrip doc explaining this.

Asking for patches and comments via Sourceforge is nowhere near as effective
as having a comment box right on the doc page. Someone has to be really
motivated to take the trouble to go to SF to submit a comment. A comment box
makes it easy.

-Mike

 

[John Roth]

what really makes sense for me with php's documentation is how very
intuitive (and therefore easy and fast) it is to get to what one is
looking for. this is particularly true of its online documentation of
its built in functions. this ease of access is leveraged through their
website's way of dealing with 404 requests.

if you go to php.net/<function name> you are automatically redirected
to the page that documents that function.

this is often more efficient than the three step process of going to
the page, doing a search, clicking on the appropriate search result. it
is often faster than have a bookmark to a page that lists all functions
and then scrolling or even Ctrl-Fing your way to the appropriate one.
[snip]

online documentation shouldn't be one's only recourse though, and
this is a place where python has a major leg up on php. one aspect to
python's documentation that i really appreciate is having the help()
function interface to pydoc now in the interpreter. whether it be ruby's
'ri', perl's 'perldoc' or python's 'pydoc', command line interfaces to
documentation are very helpful, both as a learning tool and a reference,
especially when they can be accessed from within an editor such as with
vi's Shift-K macro. php has no such interface.

I kind of like this notion. There are a lot of times I'm going for a
module that I know is there, but I'm not sure of either the name or
the location in the TOC. (And the trick of cleverly hiding
the module index at the back of the Library Reference instead of
leaving it as a separate document got me confused for a while, but
that may be a WinHelp issue.)

pydoc from the interpreter through 'help()' often gets me to where i want
to go in one step, so no complaints there. it would be nice, though,
if there was the same kind of web site functionality at python.org
as there is at php.net (though i am not familiar with all the ways to
access documentation for either and there may be many means of getting
documentation that address my need that i am just not aware of).

I vaguely remember that there were ways of rigging WinHelp so that
it would do something similar, at least if you asked nicely. The
difficulty is probably getting the doc process to produce the right
incantations.

John Roth

 

[Gyro Funch]

He did read the documentation for lstrip, but the doc doesn't make it at all
clear that the argument to lstrip is treated as a set of characters.

With the PHP documentation model, there would have been a user comment and
example attached to the lstrip doc explaining this.

Asking for patches and comments via Sourceforge is nowhere near as effective
as having a comment box right on the doc page. Someone has to be really
motivated to take the trouble to go to SF to submit a comment. A comment box
makes it easy.

-Mike

Hi,
Would it make sense to work toward a built-in function 'example()'
analogous to Python's built-in help facility?

Help on method_descriptor:

lstrip(...)
S.lstrip([chars]) -> string or unicode

Return a copy of the string S with leading whitespace removed.
If chars is given and not None, remove characters in chars
instead.
If chars is unicode, S will be converted to unicode before
stripping

'my test'


Perhaps the 'example' facility could present one or more examples
of typical usage and illustrate any relevant 'gotchas'.

-g

 

[Skip Montanaro]

Mike> A good example of this just came up in this group a few days
Mike> ago. A new Python programmer was puzzled over this result:
Mike> 'wsouthpine.com" {'

Mike> He did read the documentation for lstrip, but the doc doesn't make
Mike> it at all clear that the argument to lstrip is treated as a set of
Mike> characters.

Mike> With the PHP documentation model, there would have been a user
Mike> comment and example attached to the lstrip doc explaining this.

As a first step perhaps someone can it upon themselves to mirror the library
reference manual's structure in the wiki and start adding missing examples,
comments, etc. If what's there becomes significant enough, medium term I
think one or both of these will happen:

1. Content will migrate from the wiki to the libref manual.

2. Links from the online libref manual will be added that refer to the
comment/example pages in the wiki.

Longer term, perhaps a more tightly integrated system can be developed.
Note however that the current system uses latex2html to generate the online
documentation and that there is very limited extra time with the current
people actively doing Python documentation.

I created a Python Glossary wiki awhile back:

http://manatee.mojam.com/python-glossary

with the intention that it would serve as raw material for the glossary in
the Python tutorial. I return to it ever-so-rarely to migrate content from
it into the glossary in the tutorial. It would be great if someone could at
least take over the review process.

Skip

 

[Andrew Bennetts]

Yes, it is a significant barrier. It requires knowledge of both Latex
and a certain way of using diff. There are many, many Python users --
especially Python-on-Windows -- who are not familiar with these
technologies and don't have the time to learn them for the sole
purpose of making a contribution to the documentation.

This is demonstrably false. I've contributed to the docs with a bug where I
suggested a sentence or two that should be inserted, and that was sufficient
for my text to make it into the docs. No diff, no LaTeX. I doubt I'm
alone.

I agree. But there are gaps in the documentation, where the
officially approved documentation is silent. In such cases, I prefer
advice from others, even though I've never met them, to no help at
all.

After all, what is comp.lang.python except advice from people that
you've (for the most part) never met.

comp.lang.python is also a place where you can already most likely find
advice on pretty much any un- or under-documented standard library module,
if you check the archives. So even here we have everything the PHP docs do.

In general, I'd agree that the Python documentation is outstanding.
But there are gaps -- a qualifying phrase here, a missing reference
there, a small example in many places. Filling those gaps would make
the difference in those places between documentation that is just
barely adequate and documentation that is really excellent.

You seem to know of several cases of these -- so submit bugs about them! No
diff or LaTeX required, as explained above.

[...]

WE NEED TO MAKE IT EASIER FOR MANY HANDS -- INCLUDING NEWBIE HANDS --
TO CONTRIBUTE TO THE DOCUMENTATION.

This is problematic. You're quite right that newbies rely more heavily on
docs than experienced users, and so they're more likely to find any problems
in the docs. But they're also the least qualified to know how to fix it
(as amply demonstrated by your, ahem, "impressive" example of PHP's docs on
sorting arrays). Because of this, newbies generally should be submitting
bugs, rather than offering patches, an activity that clearly needs no diff
or LaTeX.

I realise I'm sounding repetitive here, but: The only barrier is being able
to submit a bug to the bug tracker.

The PHP model is a good one. Many hands contribute online.
Periodically, expert hands incorporate the best of the online
contributions into the mainline docs.

How long is that period? The example you gave had comments that were
several years old. How many releases of PHP have there been since 2000?

-Andrew.

 

[Roger Binns]

Asking for patches and comments via Sourceforge is nowhere near as effective
as having a comment box right on the doc page. Someone has to be really
motivated to take the trouble to go to SF to submit a comment.

Not only that, there is a many month if not year lead time between supplying
information via SF and actually having the results available (the next release
of Python).

The PHP docs give instant gratification. I know I am significantly less
likely to contribute to projects the longer the lead time between me making
a contribution and the results being available.

Roger

 

[Aahz]

I realise I'm sounding repetitive here, but: The only barrier is being able
to submit a bug to the bug tracker.

It's not even a particularly large barrier; SF finally fixed things so
that Lynx can be used to submit bugs. ;-)