a ConfigParser wtf moment

[Eric S. Johansson]

I'm not sure if this is a real problem or if I have been staring at code
too long. given this code

#!/usr/bin/python

from ConfigParser import *

configuration_file = "test.conf"
substitution = {"xyzzy":"maze"}
configuration = SafeConfigParser()
configuration.readfp(file(configuration_file))


list = configuration.items("core")
print list

list = configuration.items("core",0, substitution)
print list

---------------

and this configuration file

-----------

[core]
xyzzy=little bird
dwarf = greasy smoke %(xyzzy)s plugh

------------
why do I get the following results?

-------------

trial backup # python test.py
[('dwarf', 'greasy smoke little bird plugh'), ('xyzzy', 'little bird')]
[('dwarf', 'greasy smoke maze plugh'), ('xyzzy', 'maze')]
trial backup #

-------------

if you're having a hard time seeing it, before the substitution, xyzzy
is set to the value in the configuration file, afterwards, it is set to
the value of the substitution in the code. It seems to me that
substitutions should not affect any configuration file symbols of the
same name.

anyway to fix this problem without python diving?

---eric

 

[grahamd]

Sort of hard to explain, but if you put another:

list = configuration.items("core")
print list

at the end of the script, you will find that the original config hasn't
been changed.
It is a quirk of how the items() method is implemented using 'yield'
that means that
you see what you do.

In particular to use 'yield' it it necessary to create a temporary
dictionary which
contains the key/value pairs from that section of the config and then
overlay it
with the user supplied vars. Ie., the items() code has:

.. d = self._defaults.copy()
.. try:
.. d.update(self._sections[section])
.. except KeyError:
.. if section != DEFAULTSECT:
.. raise NoSectionError(section)
.. # Update with the entry specific variables
.. if vars:
.. d.update(vars)

See the last line, that will replace the value of 'xyzzy' with that
passed in
as argument to items().

To avoid this, you need to write something like:

.. list = []
.. for key in configuration.options("core"):
.. list.append((key,configuration.get("core",substitution))
.. print list

This cause me problems for a different reason, ie., that user vars keys
appear in what items() returns. I avoid using items() for this reason.

 

[Eric S. Johansson]

To avoid this, you need to write something like:

. list = []
. for key in configuration.options("core"):
. list.append((key,configuration.get("core",substitution))
. print list

This cause me problems for a different reason, ie., that user vars keys
appear in what items() returns. I avoid using items() for this reason.

it turns out, I originally discovered this problem through the get with
substitutions. It's too late to muck with it now but if you are really
interested I will generate a test case showing the failure or else prove
I was hallucinating.

My current workaround is to look for a %( in every value returned and if
so do a string substitution.

value = base.get(section,item, 1)
if value.find("%(") != -1:
value = value% self.interpolation_symbols

yes, it is as ugly as a roadkill toad but it gets the job done. I've
spent away too much time on this problem for this particular project. I
think we all know the feeling.

---eric

 

[grahamd]

True, wasn't thinking. This will affect get() as well. My problem was a
slightly different problem.

In your case you would have got what you wanted if get()/items()
instead of being implemented as:

.. try:
.. value = d[option]
.. except KeyError:
.. raise NoOptionError(option, section)

Was implemented as:

.. try:
.. value = self._sections[section][option]
.. except KeyError:
.. raise NoOptionError(option, section)

That is, get the raw value from the original dictionary for the
section. That way you avoid picking up a value from the user
supplied vars. It does also avoid picking a key which has come
through from the default section, but that could easily be
accomodated if that was perceived to be expected behaviour
by having the except clause fall through to looking in the
default section. Similarly if seen that getting it from vars is
okay as well, fall back onto that as file step. All depends on
what the perceived overridding priority is.

At the moment vars overrides section which overrides default
and the document sort of does say that that is what one should
expect for vars at least:

.. Additional substitutions may be provided using the
.. `vars' argument, which must be a dictionary whose contents overrides
.. any pre-existing defaults.

Probably not what I would have expected either. I would have expected
vars to be available for substitution but not for option selection in
the
first place.
It is too late on a Friday, so I may be hallucinating now.