ConfigParser.get() defaults?

[Tim Chase]

With a normal dictionary, I can specify a default fallback value
in the event the requested key isn't present:

d = {}
print d.get(42, 'Some default goes here')

However, with the ConfigParser object, there doesn't seem to be
any way to do a similar

cp = ConfigParser(...)
# ...
print cp.get('MySection', 'MyValue', 'Some default goes here')

that, in the event either "MySection" doesn't exist in the config
file, or "MyValue" doesn't exist within "MySection", it simply &
quietly returns my default. At the moment, I have to mimic this with

try:
val = cp.get('MySection', 'MyValue')
except (NoSectionError, NoOptionError), e:
val = 'Some default goes here'
print val

which is a real pain when you have multiple options, some using
various combinations of get(), getboolean(), getint(), or
getfloat(). Yes, I can write some cheap wrappers to do these,
but it feels like something that should be built-in.


I've played with the DEFAULT, but that seems to just create its
own section that I have to *ask* for:

cp.get('DEFAULT', 'MyValue')



I've thumbed through the source to ConfigParser.py but don't see
any indication that it's possible to do what I'd like short of
wrapping all my cp.get*() calls in try/except blocks or creating
a forked version of ConfigParser.py locally. Is there anything
I'm missing, or a better workaround to this?

Thanks,

-tkc

 

[Jon Clements]

With a normal dictionary, I can specify a default fallback value
in the event the requested key isn't present:

   d = {}
   print d.get(42, 'Some default goes here')

However, with the ConfigParser object, there doesn't seem to be
any way to do a similar

   cp = ConfigParser(...)
   # ...
   print cp.get('MySection', 'MyValue', 'Some default goes here')

that, in the event either "MySection" doesn't exist in the config
file, or "MyValue" doesn't exist within "MySection", it simply &
quietly returns my default.  At the moment, I have to mimic this with

   try:
     val = cp.get('MySection', 'MyValue')
   except (NoSectionError, NoOptionError), e:
     val = 'Some default goes here'
   print val

which is a real pain when you have multiple options, some using
various combinations of get(), getboolean(), getint(), or
getfloat().  Yes, I can write some cheap wrappers to do these,
but it feels like something that should be built-in.

I've played with the DEFAULT, but that seems to just create its
own section that I have to *ask* for:

   cp.get('DEFAULT', 'MyValue')

I've thumbed through the source to ConfigParser.py but don't see
any indication that it's possible to do what I'd like short of
wrapping all my cp.get*() calls in try/except blocks or creating
a forked version of ConfigParser.py locally.  Is there anything
I'm missing, or a better workaround to this?

Thanks,

-tkc

Not convinced about this, but a quick "work-around" would be something
like:

def get_config(config, section, option, ctype=str, default=None):
if default is None:
ret = config.get(section, option)
else:
confdict = config.__dict__.get('_sections')
ret = confdict.get(section).get(option, default)
return ctype(ret)

That should cover most exceptions and the get* functions..., just
provide your own factory function to post-parse it...

Of course, I'm probably missing something as usual...

Anyway, Cheers,

Jon.

 

[Steven D'Aprano]

With a normal dictionary, I can specify a default fallback value in the
event the requested key isn't present: [...]
However, with the ConfigParser object, there doesn't seem to be any way
to do a similar

[...]


Sounds like a nice feature to have. When you submit a patch (*grin*),
please make sure the caller can specify what to do on missing section and
missing option independently.

I've thumbed through the source to ConfigParser.py but don't see any
indication that it's possible to do what I'd like short of wrapping all
my cp.get*() calls in try/except blocks or creating a forked version of
ConfigParser.py locally. Is there anything I'm missing, or a better
workaround to this?

Don't fork the source code, that's ridiculously overkill. Subclass
ConfigParser and add the required functionality.

 

[Tim Chase]

With a normal dictionary, I can specify a default fallback value in the
event the requested key isn't present: [...]
However, with the ConfigParser object, there doesn't seem to be any way
to do a similar

Sounds like a nice feature to have. When you submit a patch (*grin*),

I'm game to create and provide a patch, though I've had no
success in the past trying to push tweaks into the Python Core
(adding a subclass of optparse.HelpFormatter that respects
newlines for text-wrapping).

But if there's a straightforward way to do this, I'd love to
patch the file and submit the deltas. I just need to know

- what sort of patch (unified diff, context diff, ed)

- against which version? I'm currently running the 2.5 that
Debian Testing defaults to, though I have 2.6 available through
my repository.

- any tests needed/expected? run how? internal as an "if
__name__ == '__main__'" block, or an external test suite?

- where (or to whom) to I submit the patch (and possibly tests)


I asked these questions about my optparse patch and never heard
anything back (it was against 2.4 so I'm sure optparse has
changed enough to render my past patches somewhat defunct)

please make sure the caller can specify what to do on missing section and
missing option independently.

Could you detail what you envision here? For my use case, it's
that the option doesn't exist, whether because there's no
section, or there's a section but no option. Either way,
something failed and I want a default value back. I don't have a
mental model of what you expect in the "section exists but no
option" that would behave differently from the "section doesn't
exist" case.

Do you also have suggestions for how it should interact with the
weird [DEFAULT] section oddities (that don't really seem to
default the way I expect them to)?

Perhaps the two interrelate?

Don't fork the source code, that's ridiculously overkill. Subclass
ConfigParser and add the required functionality.

Sorry for such loaded terms -- my "forking" suggestion wasn't
intended to be Python, the Tim Version(tm), but rather, "copy the
ConfigParser.py into my project directory, make the edits I
need/want, and have it available to my project" (something
similar to what I did for one of my projects copying in the
python2.6 zipfile.py to my python2.4 code-base to supersede the
standard 2.4 library's zipfile.py).

Ideally, I'd tweak the base RawConfigParser class (and its
subclasses if needed) rather than subclass, as it's functionality
that would be useful in all cases (Raw, regular, and Safe).

But yes, doing this would allow me to proffer the above-suggested
patches.

Thanks for your input,

-tkc

 

[Tim Chase]

With a normal dictionary, I can specify a default fallback value in the
event the requested key isn't present: [...]
However, with the ConfigParser object, there doesn't seem to be any way
to do a similar

Sounds like a nice feature to have. When you submit a patch (*grin*),

I'm game to create and provide a patch,

Attached is my initial patch against 2.6's ConfigParser.py to
work in some defaults (though it should also apply fairly cleanly
against 2.5).

It doesn't differentiate between missing sections and missing
options, as both cases are the same in my use: the user didn't
provide a setting, so I want to specify a default without lots of
try/except wrapping.

For the type-specific get[int/float/boolean]() functions, it also
provides an optional parameter to raise the ValueError or just
return the default if it's provided-but-bogus (it may also yell
at you if your default creates a ValueError).

My distributed ConfigParser.py didn't seem to have any tests, so
I didn't include any; but would be glad to write a few if they'd
be integrated into the standard core testing.

Is this the sort of things that should also be sent to
python-dev, or are the powers-that-be lurking sufficiently on
this list to see this patch fly?

-tkc

 

[Steven D'Aprano]

With a normal dictionary, I can specify a default fallback value in
the event the requested key isn't present: [...]
However, with the ConfigParser object, there doesn't seem to be any
way to do a similar

Sounds like a nice feature to have. When you submit a patch (*grin*),

I'm game to create and provide a patch, though I've had no success in
the past trying to push tweaks into the Python Core (adding a subclass
of optparse.HelpFormatter that respects newlines for text-wrapping).

If your patch doesn't attract the interest of a Python-Dev developer, you
might need to give them a prod occasionally. Their time for reviewing
bugs and patches is always in short supply.

But if there's a straightforward way to do this, I'd love to patch the
file and submit the deltas. I just need to know

- what sort of patch (unified diff, context diff, ed)

- against which version? I'm currently running the 2.5 that Debian
Testing defaults to, though I have 2.6 available through my repository.

- any tests needed/expected? run how? internal as an "if __name__ ==
'__main__'" block, or an external test suite?

- where (or to whom) to I submit the patch (and possibly tests)

Submit the patch and tests to the bug tracker at http://bugs.python.org/

Look at the tests in (e.g.) /usr/local/lib/python2.6/test/ for an idea of
what you should do.

Because this is a new feature, and 2.5 (and 2.6?) are in bug fix only
mode, you should probably aim for 2.7 or 3.2 as the ideal. Patches
written against 2.6 or 3.1 are probably good enough though.

And you want a unified diff.

See also: http://www.python.org/dev/patches/

please make sure the caller can specify what to do on missing section
and missing option independently.

Could you detail what you envision here? For my use case, it's that the
option doesn't exist, whether because there's no section, or there's a
section but no option. Either way, something failed and I want a
default value back. I don't have a mental model of what you expect in
the "section exists but no option" that would behave differently from
the "section doesn't exist" case.

Do you also have suggestions for how it should interact with the weird
[DEFAULT] section oddities (that don't really seem to default the way I
expect them to)?

Perhaps the two interrelate?

I haven't really given it any detailed thought, but what I was thinking
was something like:

config.set_missing_section("FOO")
config.set_missing_value("missing")

Then if you ask for the value of key 'x' in section BAR, and BAR doesn't
exist, then 'x' is looked up in FOO instead; if 'x' doesn't exist there
either, then "missing" is returned. If section BAR *does* exist, but 'x'
doesn't, then "missing" is also returned.

Sorry for such loaded terms -- my "forking" suggestion wasn't intended
to be Python, the Tim Version(tm), but rather, "copy the ConfigParser.py
into my project directory, make the edits I need/want, and have it
available to my project" (something similar to what I did for one of my
projects copying in the python2.6 zipfile.py to my python2.4 code-base
to supersede the standard 2.4 library's zipfile.py).

That's what I thought you meant. Forking is the right term in this case
(you are forking the ConfigParser module, not all of Python), and I still
say it is overkill. Generally speaking, when a module doesn't do what you
want, and modifying it isn't an option, there are five things you can do.
In order of decreasing desirability:

(1) Write a small wrapper function to tweak the behaviour.

(2) Sub-class the class to gain new behaviour.

(3) Monkey-patch the module at runtime.

(4) Fork the module and modify that.

(5) Re-write the whole thing from scratch.

 

[Tim Chase]

If your patch doesn't attract the interest of a Python-Dev
developer, you might need to give them a prod occasionally.
Their time for reviewing bugs and patches is always in short
supply.


Submit the patch and tests to the bug tracker at
http://bugs.python.org/

Okay -- a good first stop, and vastly more helpful than the
non-reply I got regarding my optparse patch.

Look at the tests in (e.g.) /usr/local/lib/python2.6/test/ for
an idea of what you should do.

I poked in my Debian analogue (/usr/bin/lib/python2.6/test/) but
the tests there seemed a bit...um...lacking? There was nothing
testing ConfigParser at all, just three high-level test modules.
Granted, I could create some tests that exercise my code, and
perhaps some that should have been there in the first place, but
that's a big jump from merely adding a couple tests (non)existing
ConfigParser tests.

Because this is a new feature, and 2.5 (and 2.6?) are in bug fix only
mode, you should probably aim for 2.7 or 3.2 as the ideal. Patches
written against 2.6 or 3.1 are probably good enough though.

Okay, I've submitted the patch to dev.python.org in the hopes
something comes of it, even if it's 2.7 or 3.2

That's what I thought you meant. Forking is the right term in this case
(you are forking the ConfigParser module, not all of Python), and I still
say it is overkill. Generally speaking, when a module doesn't do what you
want, and modifying it isn't an option, there are five things you can do.
In order of decreasing desirability:

(1) Write a small wrapper function to tweak the behaviour.

(2) Sub-class the class to gain new behaviour.

(3) Monkey-patch the module at runtime.

(4) Fork the module and modify that.

(5) Re-write the whole thing from scratch.

I'd say the ordering of your #3 and #4 are more ambiguous -- if
I'm distributing/deploying a project and my monkey-patching gets
broken (there's a reason monkey-patching feels dirty) by some
python lib upgrade external to my control, then I & my code look
bad. If I freeze a modified/locally-forked library module, it
continues to work regardless of the environment. Maybe that's an
indicator that I did a bad job of monkey patching, but I prefer
to prevent breakage where I can.

-tkc

 

[Paco]

I subclass ConfigParser to provide a get function with default value.

class _ConfigParser(ConfigParser, object):
    def get(self, section, option, default='_NoDefault'):
        try:
            return super(_ConfigParser, self).get(section, option)
        except (NoSectionError, NoOptionError) as e:
            if default == '_NoDefault':
                raise e
            return default