Naive idiom questions

[Terran Melconian]

* Why are there no real mutable strings available?

I found MutableString in UserString, but further research indicates
that it is horribly inefficient and actually just wraps immutable
strings for the implementation.

I want to be able to accumulate a string with +=, not by going
through an intermediate list and then doing ''.join(), because I
think the latter is ugly. There are also times when I'd like to use
the string as a modifiable buffer.

Is the python idiom that this is actually the Right Thing for
reasons I'm not seeing? Is there a fundamental reason it would be
hard to implement a mutable string in cpython?

* What's the best way to initialize a list of lists?

I keep wanting to do this:

l=[[None]*5]*5

as do many other Python novices, but of course it quickly becomes
apparent why this is a bad idea as soon as one modifies a value.
The best I've found is:

l=[[None]*5 for i in range(5)]

I don't particularly like it, though. It bothers me to have to
explain list comprehensions, which are a moderately advanced feature
conceptually, just to initialize an array. We could get around this
if we had a defaultlist, but we don't. Is there a more elegant
solution here?

* Is there a way to get headings in docstrings?

I want to create my own sections, like "OVERVIEW", "EXAMPLES",
"AUTHORS", "BUGS", etc. I can't figure out any way to do this. In
perldoc, I can easily use =head1, but I can't figure out the Python
equivalent. At first I thought I could use (re)structuredtext, but
then it became apparent that pydoc(1) does not parse any of this
markup. Am I missing something, or is it just not possible to
achieve this effect other than by writing separate, independent
manpages?

Thanks for any suggestions or thoughts.

 

[Grant Edwards]

* Why are there no real mutable strings available?
[...]

I want to be able to accumulate a string with +=, not by going
through an intermediate list and then doing ''.join(),

So what's stopping you?

because I think the latter is ugly.

Then don't do it.

There are also times when I'd like to use the string as a
modifiable buffer.

That would be an array of characters or bytes:

http://docs.python.org/lib/module-array.html

Is the python idiom that this is actually the Right Thing for
reasons I'm not seeing?

I'm not sure what you're asking. AFAIK, the main reason that
strings are immutable is so they can be used as dict keys.

Is there a fundamental reason it would be hard to
implement a mutable string in cpython?

What problem would a mutable string solve that an array of
chars won't?

* What's the best way to initialize a list of lists?

Hmm. I guess I never need to do that.

* Is there a way to get headings in docstrings?

Docstrings? Real Men Don't Write Docstrings!

 

[Bjoern Schliessmann]

* Why are there no real mutable strings available?

I found MutableString in UserString, but further research
indicates that it is horribly inefficient and actually just
wraps immutable strings for the implementation.

Did you measure such impact on your application?

Also see http://www.skymind.com/~ocrow/python_string/

I want to be able to accumulate a string with +=, not by going
through an intermediate list and then doing ''.join(), because
I think the latter is ugly. 'hameggs'

There are also times when I'd like to use the string as a
modifiable buffer.

Use a list instead. When done with modifications you may concatenate
its contents to a string using "".join(my_list).

* What's the best way to initialize a list of lists?
[...]
l=[[None]*5 for i in range(5)]

I don't particularly like it, though. It bothers me to have
to explain list comprehensions, which are a moderately
advanced feature conceptually, just to initialize an array.

Look at "lower level" HLLs. In C++, you'd have to use

int my_array[5][5];

for (int i=0; i<5; ++i)
for (int j=0; j<5; j++)
my_array[j] = -1;

Personally, I like list comprehensions much better.

Regards,


Björn

 

[Paul Rubin]

I want to be able to accumulate a string with +=, not by going
through an intermediate list and then doing ''.join(), because I
think the latter is ugly. There are also times when I'd like to use
the string as a modifiable buffer.

See the StringIO, cStringIO, and array modules.

l=[[None]*5 for i in range(5)]

This is the usual way.

* Is there a way to get headings in docstrings?

I think this is normally not done.

 

[Roger Miller]

I'm not sure what you're asking. AFAIK, the main reason that
strings are immutable is so they can be used as dict keys.

I think its more fundamental than that. If strings were mutable
you would be constantly worrying about whether changing a string
here might affect something there, or whether to write x = y or
x = copy.copy(y). Or condsider

 

[Terran Melconian]

I think its more fundamental than that. If strings were mutable
you would be constantly worrying about whether changing a string
here might affect something there, or whether to write x = y or
x = copy.copy(y).

Sure. I don't think any of those considerations are fundamentally
different than what we face now with lists, though, are they? Lists can
be thought of as the mutable version of the tuple, and sets come in a
pair with frozensets so you can get whichever flavor you need for your
application. I'm all for the default string implementation being
immutable, since it facilitations certain operations and avoids certain
classes of problems, but it isn't obvious to me why there isn't a class
of a different name, such as "mutablestring", for when one wants that.
Even having just the += operator and slicing make modifications in place
could be a win.

I guess for complete symmetry, there should then be frozendicts as well,
although I don't envision a lot of use for them at the moment.

 

[Carl Banks]

* Why are there no real mutable strings available?

I found MutableString in UserString, but further research indicates
that it is horribly inefficient and actually just wraps immutable
strings for the implementation.

I want to be able to accumulate a string with +=, not by going
through an intermediate list and then doing ''.join(), because I
think the latter is ugly.

If it's just the weird spelling, you can always write

str.join("",acc)

There are also times when I'd like to use
the string as a modifiable buffer.

array module has already been mentioned.

You can also use an mmap (pass it a file handle of -1 I believe to get
an anonymous buffer), or even a numpy array of type character if
you're feeling frisky.

Python 3.0 is scheduled to have a mutable buffer type. (BTW,
originally there was going to be a mutable bytes type, but this proved
to cause many inconveniences so bytes is now an immutable, and buffer
is a now a lessened version of what bytes would have been.)

Is the python idiom that this is actually the Right Thing for
reasons I'm not seeing? Is there a fundamental reason it would be
hard to implement a mutable string in cpython?

It might seem a little weird for someone coming from Perl, but Python
isn't the sort of language that implements every possible feature that
isn't "fundamentally hard". Python carefully considers the benefits
and drawbacks of adding something, and if there is too much drawback
it won't be added, no matter how obvious or basic it seems.

In practice, very few string operations really need the benefit of
mutability. (How large are most strings, really?) OTOH, the
drawbacks of mutability are severe: once you allow string buffer data
to change, then the language has to take defensive steps in case
string values are mutated. Strings could no longer be keys for dicts,
for instance, which would be unfortunate since that's how Python looks
up global variables and attributes.

Python wisely made the built-in string type immutable--saving all
kinds of grief for the most typical usages--while providing modules
such as cStringIO, array, and mmap to handle the special cases where
you really need a mutable type.

* What's the best way to initialize a list of lists?

I keep wanting to do this:

l=[[None]*5]*5

as do many other Python novices, but of course it quickly becomes
apparent why this is a bad idea as soon as one modifies a value.
The best I've found is:

l=[[None]*5 for i in range(5)]

I don't particularly like it, though. It bothers me to have to
explain list comprehensions, which are a moderately advanced feature
conceptually, just to initialize an array. We could get around this
if we had a defaultlist, but we don't. Is there a more elegant
solution here?

No. Since 2D arrays aren't commonplace, since listcomps work passably
here, and since there's a popular thrid-party library (numpy) that
handles multidimensional arrays, it's unlikely that Python would add
any means to do this more elegantly.

You can put it in a function and forget about how it's implemented if
you don't like the listcomps.

def create_empty_2d_list(rows,cols):
return [[None]*cols for i in xrange(rows)]

* Is there a way to get headings in docstrings?

I want to create my own sections, like "OVERVIEW", "EXAMPLES",
"AUTHORS", "BUGS", etc. I can't figure out any way to do this. In
perldoc, I can easily use =head1, but I can't figure out the Python
equivalent. At first I thought I could use (re)structuredtext, but
then it became apparent that pydoc(1) does not parse any of this
markup. Am I missing something, or is it just not possible to
achieve this effect other than by writing separate, independent
manpages?

Not a big expert on docstrings (they seem so superfluous...) but
perhaps there are third-party packages that let you specify meta-
documentation with function decorators. For example:

@overview("Blah blah blah")
@authors("So and so")
def some_function():
pass

Try looking on PyPI (cheeseshop.python.org). Python's in-code
documentation is pretty basic; I think Python is happy to outsource
more elaborate schemes to third party packages.


Carl Banks

 

[Paul Rubin]

I guess for complete symmetry, there should then be frozendicts as well,
although I don't envision a lot of use for them at the moment.

There should definitely be frozendicts and I've been wanting to write
an implementation. The main thing about them is if they are
implemented as tree structures instead of hashes, you can efficiently
make differing versions of them that share structure, basically giving
you version rollback for free. And since they are immutable, you can
share them between threads without needing to mess with locks. I've
posted about this a couple times in the past.

 

[Stargaming]

* Is there a way to get headings in docstrings?

I want to create my own sections, like "OVERVIEW", "EXAMPLES",
"AUTHORS", "BUGS", etc. I can't figure out any way to do this. In
perldoc, I can easily use =head1, but I can't figure out the Python
equivalent. At first I thought I could use (re)structuredtext, but
then it became apparent that pydoc(1) does not parse any of this
markup. Am I missing something, or is it just not possible to
achieve this effect other than by writing separate, independent
manpages?

I could imagine several solutions.

Do it like normal manpages, just use indentation::

OVERVIEW
So this is my script.

AUTHORS
Written by me.

Use ReStructuredText -- it's readable even if it is not parsed by pydoc::

Overview
========

So this is my script.


Authors
=======

Written by me.

Despite being readable even without transformations, you can use
additional third-party packages to get nice and shiny documentation (HTML
for most purposes). My favorite is Epydoc_, even though there are a whole
lot of others (for example the tool used for the CPython docs, Sphinx, or
just plain docutils_).

... Epydoc: http://epydoc.sourceforge.net/
... _docutils: http://docutils.sourceforge.net/

And, uh, if you abandon pydoc and roll your own docstring parser, you
could basically use *any* markup you like. ``<h1>Overview</h1>``, for
example.

HTH,

 

[Wildemar Wildenburger]

Not a big expert on docstrings (they seem so superfluous...)

Say whaaaat?!

/W

 

[kdwyer]

I want to be able to accumulate a string with +=, not by going
through an intermediate list and then doing ''.join(), because I
think the latter is ugly.

As others have observed, you can build a string using += instead of
''.join(), but += runs in quadratic time so it's not terribly
efficient if you're building strings out of a lot of elements, or if
the number of elements is arbitrary.

Cheers,

Kev