Quick Reference from module doc strings.

[Ron Adam]

Does anyone have suggestions on how to improve this further?


Cheers,
Ron_Adam



def getobjs(object, dlist=[], lvl=0, maxlevel=1):
""" Retrieve a list of sub objects from an object. """
if object not in dlist:
dlist.append(object)
if lvl<maxlevel:
dobj = dir(eval(object))
for item in dobj:
try:
dlist = getobjs(object+'.'+item, dlist, lvl+1)
except:
pass
return dlist

def printdoc(objectlist):
""" Return a sorted printable quick reference
guide from a list of objects. """
outtext = []
objectlist.sort(lambda x, y: cmp(x.lower(), y.lower()))
for obj in objectlist:
object = eval(obj)
object_type = type(object)
outtext.append('-'*40+'\n')
outtext.append(str(obj)+'\n')
if hasattr(object, '__module__'):
outtext.append("Module:"+str(object.__module__)+'\n')
if hasattr( object,'__class__'):
outtext.append("Class:"+str(object.__class__)+'\n\n')
else:
outtext.append("Type:"+str(object_type)+'\n\n')
if isinstance(object,str):
if len(object)>200:
s = object[0:200]+"......"
else:
s = object
outtext.append(obj+'=')
if '\n' in s: quotes='"""'
else: quotes ='"'
if len(s)>60: print
outtext.append(quotes+s+quotes+'\n\n')
elif (isinstance(object,str)
or isinstance(object,int)
or isinstance(object,bool)
or isinstance(object,tuple)
or isinstance(object,list)
or isinstance(object,dict)):
s = str(object)
if len(s)<200:
outtext.append(obj+'='+s+'\n\n')
else:
outtext.append(obj+'='+s[0:200]+'......\n\n')
if hasattr(object,'__doc__'):
if object.__doc__ != type(object).__doc__:
outtext.append(str(object.__doc__)+'\n\n')
return ''.join(outtext)

def quick_ref(name):
"""
quick_ref(module_name) -> printable string

Generate a sorted quick reference guide from an objects
doc strings. The module_name is a string with the name of
the module or class to get documents string from.

Example:
import os
print quick_ref('os')
"""
objlist = getobjs(name)
return printdoc(objlist)

if __name__ == "__main__":
#import module before calling in most cases.
print quick_ref('__builtins__')

 

[John Machin]

Does anyone have suggestions on how to improve this further?

Not functionally (from me, yet). However if you can bear a stylistic
comment, do read on

elif (isinstance(object,str)
or isinstance(object,int)
or isinstance(object,bool)
or isinstance(object,tuple)
or isinstance(object,list)
or isinstance(object,dict)):

Since Python version 2.2, the 2nd arg of isinstance may be a tuple. You
could define this up front, with a meaningful name:

TYPES_WHICH_whatever = (str, int, bool, etc etc)

Cheers,
John

 

[Ron Adam]

Not functionally (from me, yet). However if you can bear a stylistic
comment, do read on




Since Python version 2.2, the 2nd arg of isinstance may be a tuple. You
could define this up front, with a meaningful name:

TYPES_WHICH_whatever = (str, int, bool, etc etc)

Cheers,
John

Actually I'm begging for comments, it's the beginning of a project not
the end. So thanks! ;-)

I changed it to:

if type(object)==str:
....
elif type(object) in (str,int,bool,tuple,list,dict):
....

Thanks, I don't need the isinstance(), type works here just as well.

What would it take to make it useful? I'm thinking of putting it to use
in a database where you can get quickly get info by subject and
keywords, not just by module. I'm also thinking of using it to generate
web pages.

As far as the actual content of doc strings for each item, we can submit
requests for improvements where it's needed.

Cheers,
_Ron

 

[Scott David Daniels]

Actually I'm begging for comments, it's the beginning of a project not
the end. So thanks! ;-)

I changed it to:

if type(object)==str:
....
elif type(object) in (str,int,bool,tuple,list,dict):
....

Althought object is a horrible name for your own value (there is a builtin
object which you use for defining new-style classes), you probably want:

if isinstance(object, (str,int,bool,tuple,list,dict)):
...
or (as John Machin was trying to suggest):

if isinstance(object, TYPES_WHICH_whatever):
...

This allows you to use instances of those builtin types and any
user-defined subtypes.

Thanks, I don't need the isinstance(), type works here just as well.

But the isinstance version is better than the type(...) in ... version.

--Scott David Daniels
(e-mail address removed)

 

[Ron Adam]

Althought object is a horrible name for your own value (there is a builtin
object which you use for defining new-style classes), you probably want:

Good point, I agree. It's a bad habit to start, sooner or later it
would cause a problem. I'll find something else to call it.

if isinstance(object, (str,int,bool,tuple,list,dict)):
...
or (as John Machin was trying to suggest):

if isinstance(object, TYPES_WHICH_whatever):
...

This allows you to use instances of those builtin types and any
user-defined subtypes.




But the isinstance version is better than the type(...) in ... version.

Ok, done. I didn't think to look it up in the quick reference this
prints out. LOL, Thanks Scott

I'm not sure if isinstance() would make a difference, it doesn't for
__builtins__, but it might in some other modules.

--Scott David Daniels
(e-mail address removed)

Do you have any feature suggestions, additional information that could
go in, something that would extend the content in some way and make it
more useful?

As it stands now, it could be just a module, so you could...


Regards,
_Ron



The current version is now....


# qref.py
"""
Generate a quick reference for a module by analyzing
a modules contents.

Use example:
import qref
import sys
print qref.quickref("sys")
"""

def getobjs(obj, dlist=[], lvl=0, maxlevel=1):
""" Retreive a list of sub objects from an object. """
if obj not in dlist:
dlist.append(obj)
if lvl<maxlevel:
dobj = dir(eval(obj))
for item in dobj:
try:
dlist = getobjs(obj+'.'+item, dlist, lvl+1)
except:
pass
return dlist

def printdoc(objectlist):
""" Return a sorted printable quick reference
guide from a list of objects. """
outtext = []
objectlist.sort(lambda x, y: cmp(x.lower(), y.lower()))
for obj in objectlist:
objt = eval(obj)
objt_type = type(objt)
outtext.append('-'*40+'\n')
outtext.append(str(obj)+'\n')
if hasattr(objt, '__module__'):
outtext.append("Module:"+str(objt.__module__)+'\n')
if hasattr( objt,'__class__'):
outtext.append("Class:"+str(objt.__class__)+'\n\n')
else:
outtext.append("Type:"+str(objt_type)+'\n\n')
if isinstance(objt,str):
if len(objt)>200:
s = objt[0:200]+"......"
else:
s = objt
outtext.append(obj+'=')
if '\n' in s: quotes='"""'
else: quotes ='"'
if len(s)>60: print
outtext.append(quotes+s+quotes+'\n\n')
elif isinstance(objt,(int,bool,tuple,list,dict)):
s = str(objt)
if len(s)<200:
outtext.append(obj+'='+s+'\n\n')
else:
outtext.append(obj+'='+s[0:200]+'......\n\n')
if hasattr(objt,'__doc__'):
if objt.__doc__ != type(objt).__doc__:
outtext.append(str(objt.__doc__)+'\n\n')
outtext.append('-'*40+'\n')
return ''.join(outtext)

def quickref(name):
"""
quickref(module_name) -> printable string

Generate a sorted quick reference guide from an objects
doc strings. The module_name is a string with the name of
the module or class to get documents string from.

Example:
import os
print quickref('os')
"""
objlist = getobjs(name)
return printdoc(objlist)

if __name__ == "__main__":
print quickref('__builtins__')

 

[Scott David Daniels]

Do you have any feature suggestions, additional information that could
go in, something that would extend the content in some way and make it
more useful?

As it stands now, it could be just a module, so you could...

The style is still a sticking point for me -- too many evals (a nasty
lure for evil-doers).

Recall that sys.modules is a dictionary from module names to modules.

Any python object (including a module) will return attributes listed
in a dir(obj) with getattr(obj, attributename). So, no evals are
needed in this program at all.

I also wonder if you've ever used maxlevel>1. It seems you'd expand
to way too many names to be useful.

Finally, if you use StringIO (or cStringIO), most of your string
construction can be turned into prints in a way similar to:

def printnames(namelist, dest=None):
for name in sorted(namelist, key=str.lower):
print >>dest, name

from cStringIO import StringIO

def textnames(namelist):
hold = StringIO()
printnames(namelist, hold)
return hold.getvalue()

--Scott David Daniels
(e-mail address removed)

 

[Ron Adam]

The style is still a sticking point for me -- too many evals (a nasty
lure for evil-doers).

I suppose someone could create an "evil" name in a module that would
either cause and error when evaluated or evaluate to something other
than the actual name.

Recall that sys.modules is a dictionary from module names to modules.

Any python object (including a module) will return attributes listed
in a dir(obj) with getattr(obj, attributename). So, no evals are
needed in this program at all.

Thanks for suggesting the concise way to to this. It was a todo item.

I also wonder if you've ever used maxlevel>1. It seems you'd expand
to way too many names to be useful.

Yes, it does quickly explode in some cases, I left the maxlevel keyword
in there because it might be useful as a general function for other
purposes like building a dependency tree for example. I haven't
actually tried that yet, and there's probably better ways to do it.

Finally, if you use StringIO (or cStringIO), most of your string
construction can be turned into prints in a way similar to:

def printnames(namelist, dest=None):
for name in sorted(namelist, key=str.lower):
print >>dest, name

from cStringIO import StringIO

def textnames(namelist):
hold = StringIO()
printnames(namelist, hold)
return hold.getvalue()

--Scott David Daniels
(e-mail address removed)

What would be the advantage of using StringIO over list.append with
''.join()?

I rearranged it somewhat to create a dictionary of object docs. I think
this approach will be better for creating lists in different orders
without having to regenerate the whole thing.

This is where it is now... (No evals, better organized overall I think)

Cheers, _Ron


# qref.py
"""
Generate a quick reference for a module by analyzing
a modules contents.
"""

def getobjs(obj, objlist=[]):
"""Retreive a list of sub-object (name,object) pairs
from an object."""
objlist.append((obj.__name__,obj))
for name in dir(obj):
obj2 = getattr(obj,name)
name = obj.__name__+'.'+name
if (obj2,name) not in objlist:
objlist.append((name,obj2))
return objlist

def getdocdict(objlist):
"""Build a dictionary of docs from an list of
(name,object) tuples."""
doc_dict = {}
for item in objlist:
doc_dict[item[0]] = getobjinfo(*item)
return doc_dict

def getobjinfo(name,obj):
"""Build a formated document string from an objects
__doc__ string, and attribues."""
strout = []
strout.append(name+'\n')
if hasattr(obj, '__module__'):
strout.append("Module:"+str(obj.__module__)+'\n')
if hasattr(obj, '__class__'):
strout.append(str(obj.__class__))
else:
strout.append(str(type(obj)))
strout.append('\n\n')
if isinstance(obj,str):
if '\n' in obj:
quotes='\n"""'
else:
quotes ='"'
s = name+' = '+quotes+obj+quotes
if len(s)>200:
s = s[0:200]+"......"
strout.append( s+'\n\n')
elif isinstance(obj,(int,bool,tuple,list,dict)):
s = name+' = '+str(obj)
if len(s)>200:
s = s[0:200]+"......"
strout.append( s+'\n\n')
if hasattr(obj,'__doc__'):
d = ''
if obj.__doc__ == None:
d = 'None'
elif obj.__doc__ != type(obj).__doc__:
d = obj.__doc__
elif obj is type(type):
d = obj.__doc__
if d != '':
strout.append(d+'\n\n')
return ''.join(strout)

def quickref(name):
"""
quickref(module_name) -> printable string

Generate a sorted quick reference guide from an objects
doc strings. The module_name is a string with the name of
the module or class to get documents string from.

Example:
import os
print quickref('os')
"""
objlist = getobjs(name)
docs = getdocdict(objlist)
dlist = docs.keys()
dlist.sort(lambda x,y: cmp(x.lower(),y.lower()))
outstr = []
outstr.append('-'*40+'\n')
for d in dlist:
outstr.append(docs[d])
outstr.append('-'*40+'\n')
return ''.join(outstr)

if __name__ == "__main__":
print quickref(__builtins__)

 

[Michele Simionato]

Do you have any feature suggestions, additional information that
could

go in, something that would extend the content in some way and make it
more useful?

I have written something similar which I use all the time. It generates
ReST
output which I can browse with "less" from the command line, as well as
HTML
output and PDF output that I can print. The hard work is done by
docutils, of course.
I have options to specify if I want to least private names or not, and
to specify
if I want an analytical index to be generated or not. Also, the HTML
output
contains hyperlinks to the original source code, so I can see it with
a click.
I can feed to "minidoc" whole packages (it works recursively on
subpackages, so everything is documented).

It was unvaluable in my struggle with Zope.

Michele Simionato

 

[Ron Adam]

I have written something similar which I use all the time. It generates
ReST
output which I can browse with "less" from the command line, as well as
HTML
output and PDF output that I can print. The hard work is done by
docutils, of course.
I have options to specify if I want to least private names or not, and
to specify
if I want an analytical index to be generated or not. Also, the HTML
output
contains hyperlinks to the original source code, so I can see it with
a click.
I can feed to "minidoc" whole packages (it works recursively on
subpackages, so everything is documented).

It was unvaluable in my struggle with Zope.

Michele Simionato

Hi Michele,

Sound great! Adding a command line parser, I'm going to add a brief
command line parser to it today, but nothing as elaborate as you have
already. Could you post a part of the output as an example? How is the
index built?

The direction I'm going is to build a database/dictionary with each
individual item as a record. Maybe feed it the whole lib directory, then
to be able to generate cross module/package lists by subject or keyword.
I'm not exactly sure how to connect everything together yet. This is
a figure it out as I go project, but I think it's worth trying.

Cheers,
_Ron

 

[Michele Simionato]

Ron Adam:

Sound great! Adding a command line parser, I'm going to add a brief
command line parser to it today, but nothing as elaborate as you have
already. Could you post a part of the output as an example? How is the
index built?

For the command line parser, see
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278844

Here is an example of output:

http://www.phyast.pitt.edu/~micheles/python/ms.html

(it is a package called "ms" (as "My Stuff") where I put my utilities).

The system works for module of the standard library too, but since
most of the time they do not have docstrings in reST format, there
are ugly formatting errors. But this is a bug of the standard library,
not of my tool

For the index: "minidoc" associates a footnote number to every name,
and then
prints the names in alphabetical order. You can reach the documentation
for that name with a click.

Michele Simionato

 

[Ron Adam]

Ron Adam:

^---------------------------^

That part should have been deleted, I meant your whole program sounded
good, not just that part.

For the command line parser, see
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278844

I got this one covered, just haven't done it yet. ;-)

Here is an example of output:

http://www.phyast.pitt.edu/~micheles/python/ms.html

(it is a package called "ms" (as "My Stuff") where I put my utilities).

Good idea, I think I'll follow your lead. Currently my file are not too
organized.

The system works for module of the standard library too, but since
most of the time they do not have docstrings in reST format, there
are ugly formatting errors. But this is a bug of the standard library,
not of my tool

Thats part of what I'm trying to resolve, the doc strings a lot of time
isn't enough by itself or is missing. So I'm trying to build up a
complete enough record so if there is no doc string, at least some sense
of what it is can be figured out without a lot browsing or looking at
the source code.

Then to enable different searches by subject and keywords on these
instead of by package or module.

For the index: "minidoc" associates a footnote number to every name,
and then
prints the names in alphabetical order. You can reach the documentation
for that name with a click.

Michele Simionato

That's a nice format. It took me a while before I realized the whole
page *is* the output and not a part of it as I expected.

Did you use the inspect module to get the class/function name and args?

BTW, I presume you're aware that the "source links" on the web page link
to your computer and not to a file on the web site. Just letting know
in case it was an over site.

Cheers,
_Ron

Ps... I like your physics link page.

 

[Scott David Daniels]

...What would be the advantage of using StringIO over list.append with
''.join()?

The advantage is more in using a function that prints as it goes
rather than building up a large string to print. I would call the
print function at the bottom (with None as the print destination),
rather than printing the result of calling the string function.

I just did the StringIO thing to show you that printing as you go
needn't mean you cannot get the string value without duplicating code.

--Scott David Daniels
(e-mail address removed)

 

[Ron Adam]

The advantage is more in using a function that prints as it goes
rather than building up a large string to print. I would call the
print function at the bottom (with None as the print destination),
rather than printing the result of calling the string function.

The reason I return the string to the calling function instead of
printing it, is the person using it may not want to print it. They may
want to send it to a file, or process it further before printing it.
Same with the individual records.

I just did the StringIO thing to show you that printing as you go
needn't mean you cannot get the string value without duplicating code.

--Scott David Daniels
(e-mail address removed)

I've used IO Streams,(">>" and "<<"), before in C, where I was
continually reading to and writing from a buffer as I went. (Back when
memory was often smaller than most files.)

I haven't played around with the StringIO module yet, and this will
probably be more of a database application rather than just formatting
input for output. So for now the strings operations are fine until I
better decide how they will get used.

Maybe further down the road I'll convert them to, or have a need for
serialized objects, so I'll keep the STringIO module in mind.

Cheers,
_Ron

 

[Michele Simionato]

Ron Adam:

Thats part of what I'm trying to resolve, the doc strings a lot of time
isn't enough by itself or is missing. So I'm trying to build up a
complete enough record so if there is no doc string, at least some sense
of what it is can be figured out without a lot browsing or looking at
the source code.

I have a command line option so I can decide if I want to display only
the
documented objects or every object, including the ones without
docstring.
I could add an option to tell "minidoc" to interpret docstrings are
plain
text, not restructured text; however I don't want to do that (part of
the reason for "minidoc" is to force me to write good reST docstrings
in my own modules).

Then to enable different searches by subject and keywords on these
instead of by package or module.

Well, my browser has a CTRL-F functionality

That's a nice format. It took me a while before I realized the whole
page *is* the output and not a part of it as I expected.

You should see how nice is the PDF when printed

Did you use the inspect module to get the class/function name and

args?

Yes, the inspect module is amazing.

BTW, I presume you're aware that the "source links" on the web page link
to your computer and not to a file on the web site. Just letting know
in case it was an over site.

Yes, I am aware of it, it was on purpose

Cheers,
_Ron

Ps... I like your physics link page.

That page is old, I am no more in physics, but I still have that
account for free
and I am lazy, so I am keeping it

Michele Simionato

 

[Michele Simionato]

These days I use generators instead of StringIO, i.e.
instead of

print >> out, mystring

I just write

yield mystring

and then I "".join the generator.

Michele Simionato

 

[Ron Adam]

Ron Adam:



I have a command line option so I can decide if I want to display only
the
documented objects or every object, including the ones without
docstring.
I could add an option to tell "minidoc" to interpret docstrings are
plain
text, not restructured text; however I don't want to do that (part of
the reason for "minidoc" is to force me to write good reST docstrings
in my own modules).

If it works, don't fix it. ;-)

Well, my browser has a CTRL-F functionality

I recently switched over to using the GVIMs Cream version as an editor,
So it should be possible to map a function key to grab the import
statements from the currently edited document, feed the imported module
names to quickref and capture the output in a window for searching in
the same way.

It will be fun to play around with once I get back from a week long trip
starting Thursday.

Yes, I am aware of it, it was on purpose

I was pretty sure it was.

That page is old, I am no more in physics, but I still have that
account for free
and I am lazy, so I am keeping it

Michele Simionato

I'm not in physics in *any* way, but I have a few pet theories (more
like opinions really). So I like to check what's going on and see if
anyone or anything proves or disproves them. ;-)