ConfigParser: writes a list but reads a string?

[Terry Carroll]

It looks like ConfigParser will accept a list to be writing to the
*.ini file; but when reading it back in, it treats it as a string.

Example:

###############################
import ConfigParser
def whatzit(thingname, thing):
print thingname, "value:", thing
print thingname, "length:", len(thing)
print thingname, type(thing)

cfgfile = "cfgtest.ini"
config1 = ConfigParser.ConfigParser()
config1.add_section("test")

t1 = range(1,11)
config1.set("test", "testlist", t1)
outfile=open(cfgfile,"w")
config1.write(outfile)
outfile.close()

config2 = ConfigParser.ConfigParser()
config2.read(cfgfile)
t2 = config2.get("test","testlist")

whatzit("t1", t1)
whatzit("t2", t2)

###############################

Output is:

t1 value: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
t1 length: 10
t1 <type 'list'>
t2 value: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
t2 length: 31
t2 <type 'str'>

That is, t1 is a list of length 10, consisting of:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
and is written out; but t2 is read back in as a string
"[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
of length 31.

It took me a while to figure this out, since they looked identical in
print statements.

Is there a pythonic way to read in a list from a .INI file with
ConfigParser? Is this expected behavior for ConfigParser? I would
not expect this conversion; rather, an exception when trying to write
the list if the list is not supported.

 

[Sybren Stuvel]

Terry Carroll enlightened us with:

It looks like ConfigParser will accept a list to be writing to the
*.ini file; but when reading it back in, it treats it as a string.

It doesn't say so explicitly in the manual, but I did find this:

"""The values in defaults must be appropriate for the "%()s" string
interpolation."""

So if the defaults go through %s, perhaps all values do.

Is there a pythonic way to read in a list from a .INI file with
ConfigParser?

I'd pickle() the list, and store that instead. Then you can unpicle()
it and regain your list.

I would not expect this conversion; rather, an exception when trying
to write the list if the list is not supported.

I agree with you.

Sybren

 

[Fuzzyman]

ConfigObj will read and write list values.

You get all sorts of other advantages as well (nested subsections to
any depth), and the resulting code will be much simpler.

from configobj import ConfigObj
cfgfile = "cfgtest.ini"

cfg = ConfigObj(cfgfile)
t1 = range(1,11)
# no *need* to create a subsection
cfg['test'] = {}
cfg['test']['testlist'] = t1
cfg.write()

To read it back in :

from configobj import ConfigObj
cfgfile = "cfgtest.ini"

cfg = ConfigObj(cfgfile)
print cfg['test']['testlist']

http://www.voidspace.org.uk/python/configobj.html

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

[Frithiof Andreas Jensen]

It looks like ConfigParser will accept a list to be writing to the
*.ini file; but when reading it back in, it treats it as a string.

ConfigParser is nasty because it does not really support type conversions
but still permits writing incompatible types to the configuration - i.e. the
conversions are generally one-way. It probably will croak on Floats in the
same way. The "string interpolation" works in mysterious ways too.

If you need ConfigParser to work then let every config item be strings and
convert manually as necessary.

Is there a pythonic way to read in a list from a .INI file with
ConfigParser? Is this expected behavior for ConfigParser?

Not in the way we normally use "expected" - but with ConfigParser it is to
be expected ;-)

I would
not expect this conversion; rather, an exception when trying to write
the list if the list is not supported.

If you want both text format config files and types that work then use
"ZConfig" instead.

PS:

ZConfig has no built-in way of writing a configuration - but this is not a
severe limitation: If the configuration is split into a "static-", which get
updated before a run and a "state-" set of files with things that are
changed by the application it is easier to keep the configuration consistent
anyway (it is not so hard to dump a dict into a ZConfig format).

 

[Fuzzyman]

Note that ConfigObj also supports type conversion via the ``validate``
module which comes with it. This validates a config file against an
(optional) ``configspec`` which you supply, and does all the
conversion. It reports any errors it encounters.

Syntax for reading and writing is a standard 'ini-like' format with an
extended syntax for nested sections. The major differences are that you
can have values in the root section, and you should use '=' as the
``keyword = value`` divider.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/configobj.html

 

[Larry Bates]

It looks like ConfigParser will accept a list to be writing to the
*.ini file; but when reading it back in, it treats it as a string.

Example:

###############################
import ConfigParser
def whatzit(thingname, thing):
print thingname, "value:", thing
print thingname, "length:", len(thing)
print thingname, type(thing)

cfgfile = "cfgtest.ini"
config1 = ConfigParser.ConfigParser()
config1.add_section("test")

t1 = range(1,11)
config1.set("test", "testlist", t1)
outfile=open(cfgfile,"w")
config1.write(outfile)
outfile.close()

config2 = ConfigParser.ConfigParser()
config2.read(cfgfile)
t2 = config2.get("test","testlist")

whatzit("t1", t1)
whatzit("t2", t2)

###############################

Output is:

t1 value: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
t1 length: 10
t1 <type 'list'>
t2 value: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
t2 length: 31
t2 <type 'str'>

That is, t1 is a list of length 10, consisting of:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
and is written out; but t2 is read back in as a string
"[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
of length 31.

It took me a while to figure this out, since they looked identical in
print statements.

Is there a pythonic way to read in a list from a .INI file with
ConfigParser? Is this expected behavior for ConfigParser? I would
not expect this conversion; rather, an exception when trying to write
the list if the list is not supported.

To read lists from .INI files I use following:

listvalues=INI.get(section, option).split(',')

where INI is an instance of ConfigParser

There is the problem of if list items contain commas.

-Larry Bates

 

[Terry Carroll]

To read lists from .INI files I use following:

listvalues=INI.get(section, option).split(',')

where INI is an instance of ConfigParser

There is the problem of if list items contain commas.

Thanks; that's basically pretty close to what I ended up with:

t3string=config2.get("test","testlist")
t3 = [int(x) for x in t3string[1:-1].split(",")]
whatzit("t3", t3)

Gives:

t3 value: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
t3 length: 10
t3 <type 'list'>

i.e., the same as the original list of integers that was written out.

I know, this just assumes the input will be a list in [] brackets,
with no verification of that. That's okay, because it's just an app
for me, not for any outside user. If anyone else was using this, I'd
extract the list with a regexp.

I just want to be able to edit the config file outside of the program
to change things. (Sybren, that's why Pickle was not appealing to me.)

Thanks also to Sybren, Fuzzyman and Frithiof for their contributions.
Nice to know I'm not doing anything wrong, anyway.

 

[funkyj]

I'm interested in the same sort of config file issues.

Unfortunately the restricted execution has fallen out of favor. My
preferred solution would be to have a configEval() function that is
just like the regular Python eval() but only accepts a subset of the
python language, e.g. creating the basic built in datatypes and
variable assignment and perhaps a short list of safe library functions:

favoriteColors = [ 'red', 'blue', 'orange']
props = {'fish': 7, 'cow': 'milk'}
# a comment
otherstuff = (props, favoritColors, 7)

While currently I just write my config file in python and eval() it,
this sucks from a security standpoint (e.g. someone could replace the
config file with a python program that deletes my hard drive). Thanks
for the configObj suggestion, I'll look into that.

making the config file XML and using xml.dom is another option,
although XML is a bit ugly to edit by hand.
--jfc

 

[Fuzzyman]

I'm interested in the same sort of config file issues.

Unfortunately the restricted execution has fallen out of favor. My
preferred solution would be to have a configEval() function that is
just like the regular Python eval() but only accepts a subset of the
python language, e.g. creating the basic built in datatypes and
variable assignment and perhaps a short list of safe library functions:

favoriteColors = [ 'red', 'blue', 'orange']
props = {'fish': 7, 'cow': 'milk'}
# a comment
otherstuff = (props, favoritColors, 7)

While currently I just write my config file in python and eval() it,
this sucks from a security standpoint (e.g. someone could replace the
config file with a python program that deletes my hard drive). Thanks
for the configObj suggestion, I'll look into that.

making the config file XML and using xml.dom is another option,
although XML is a bit ugly to edit by hand.

On the cookbook there is a serialize and de-serialize function that
works with the basic datatypes, but doesn't have the associated risks
of using eval.

ConfigObj is still a better solution IMHO.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

[Jeffrey Barish]

making the config file XML and using xml.dom is another option,
although XML is a bit ugly to edit by hand.
--jfc

I am seriously intrigued by ConfigObj. I am currently using an crude
improvisation involving tab-delimited fields to store metadata for
recordings -- not configuration data -- in text files. I had been planning
to convert to XML, but now I am wondering whether ConfigObj would be
easier. I would like for the metadata files to be editable, but editing
does not have to be easy as it needs to be done rarely. I've never used
XML, so I am wondering whether there are other tradeoffs between that
approach and ConfigObj that I should be aware of. I was thinking of XML
mainly to have a more robust format. For example, it would be nice if it
were possible to add fields without obsoleting early versions of the
reader. Crossplatform compatibility is also desirable.

 

[Fuzzyman]

I am seriously intrigued by ConfigObj. I am currently using an crude
improvisation involving tab-delimited fields to store metadata for
recordings -- not configuration data -- in text files. I had been planning
to convert to XML, but now I am wondering whether ConfigObj would be
easier. I would like for the metadata files to be editable, but editing
does not have to be easy as it needs to be done rarely. I've never used
XML, so I am wondering whether there are other tradeoffs between that
approach and ConfigObj that I should be aware of. I was thinking of XML
mainly to have a more robust format. For example, it would be nice if it
were possible to add fields without obsoleting early versions of the
reader. Crossplatform compatibility is also desirable.

ConfigObj can be used for all sorts of data persistence type uses.
Because it is pure python there are likely to be no cross platform
issues.

You would have to 'name' entries, but entries can be single values or
lists - or even subsections (effectively dictionaries).

If you are happy with text, then it is *very* easy to use. If you
wanted to store other datatypes, then you may be interested in
ConfigPersist.py :


http://www.voidspace.org.uk/python/articles/configobj_for_data_persistence.shtml

It is easy to use a validating configspec, and then extend it - without
obsoleting earlier code or data files.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml