config files in python

[sandipm]

Hi,
In my application, I have some configurable information which is used
by different processes. currently I have stored configration in a
conf.py file as name=value pairs, and I am importing conf.py file to
use this variable. it works well

import conf
print conf.SomeVariable

but if I need to change some configuration parameteres, it would need
me to restart processes.

I want to store this data in some conf file (txt) and would like to
use it same way as I am using these variables as defined in py
files.

one solution I can think of is writing data as a dictionary into conf
file. and then by reading data, apply eval on that data. and update
local dict? but this is not a good solution....

any pointers?

Sandip

 

[Michael Ströder]

In my application, I have some configurable information which is used
by different processes. currently I have stored configration in a
conf.py file as name=value pairs, and I am importing conf.py file to
use this variable. it works well

import conf
print conf.SomeVariable

but if I need to change some configuration parameteres, it would need
me to restart processes.

reload(conf)

http://docs.python.org/lib/built-in-funcs.html#l2h-61

But you should carefully consider what updating the configuration means
in your application.

Ciao, Michael.

 

[Matimus]

The 'simple but relatively dangerous way', already suggested, is to
reload() the module.

A safer way - but requiring more work - could be to build something around
the Configparser module in the standard library ...

Ciao

How is it relatively dangerous? I mean, aside from any `dangers'
associated with importing a user defined module in the first place.

Matt

 

[Matimus]

Hi,
In my application, I have some configurable information which is used
by different processes. currently I have stored configration in a
conf.py file as name=value pairs, and I am importing conf.py file to
use this variable. it works well

import conf
print conf.SomeVariable

but if I need to change some configuration parameteres, it would need
me to restart processes.

I want to store this data in some conf file (txt) and would like to
use it same way as I am using these variables as defined in py
files.

one solution I can think of is writing data as a dictionary into conf
file. and then by reading data, apply eval on that data. and update
local dict? but this is not a good solution....

any pointers?

Sandip

I would load the configuration file using `imp.load_source'. This
allows you to load the config file by filename, and gets away from the
issue of accidentally importing a file somewhere else in pythons
search path. Also, calling imp.load_source will reload the module when
called a second time.

http://docs.python.org/lib/module-imp.html

[conf.py]
a = 1
b = 2
class c:
a = "hello"
b = "world"
[/end conf.py]
'world'



There are so many ways potential solutions to your problem that,
without any more details, it is hard to suggest anything.

Here are some potential solutions:

ConfigParser - module for handling ini files
xml - several built-in modules for handling XML files
sqlite3 - a `lite' SQL database built-in in python 2.5 + (can be used
for config data)
windows registry - _winreg module
pickle - serialize python objects
marshal - similar to pickle, only works for simple objects

Those are just the built-in solutions. If you wanna look at 3rd party
solutions, prepare for overload. The number of alternative INI parsers
alone is staggering.

Also, there are many ways to organize your data and use a solution
similar to what you are already using.

I guess what I'm trying to say is... don't roll your own, it would be
a waste of time, this problem has been solved 100s of times. That is,
unless you want to do it for fun.

Matt

 

[George Sakkis]

How is it relatively dangerous? I mean, aside from any `dangers'
associated with importing a user defined module in the first place.

Read the caveats in reload()'s description. Briefly:
- It does not reload names imported with "from ... import ...".
- Any existing instances of classes defined in the module still refer
to the original class, not the reloaded one (assuming it is still
present).
- The module's dictionary (containing the module's global variables)
is retained and updated in place, so names that are deleted in the
module are still available.
- It is not recursive.
- And more...

George

 

[sandipm]

Thanks for various useful suggestions.
actually right now I am using conf files only in psp handler of
mod_python/apache
but I have other processes which might use same config files.

One way is I can put conf related data directly in database and
database handling module can directly pickup values from db.
but the problem with that approach is that it is difficult in managing
changes from version control system.as I need to explicitly
create DB scripts to put it in version control and other devs require
them run those scripts.

but if I put those conf params in files it would be easy for me to
change params and useful for other developers
to integrate those changes in their dev environments without doing any
explicit ops.

here I would like to have python file which read conf from text file
and load those params in current process space.
so only importing that python file should read up the conf file and
load the current process with configurable parameters.
I thought this would be good way to do it, rather than getting
involved in slightly complicted reload mechanisms of python modules?



Regards,
Sandip

 

[Jorge Vargas]

here I would like to have python file which read conf from text file
and load those params in current process space.
so only importing that python file should read up the conf file and
load the current process with configurable parameters.
I thought this would be good way to do it, rather than getting
involved in slightly complicted reload mechanisms of python modules?

I think you should work your way into a proper configuration system.

for ones you can use the buildin module
http://docs.python.org/lib/module-ConfigParser.html

or migrate to the more powerful configObj
http://www.voidspace.org.uk/python/configobj.html
with configobj, you will have an instance in memory that you can
modify on runtime with a gui or something and when you hit save it
will rewrite the new config file.

 

[M.-A. Lemburg]

I would load the configuration file using `imp.load_source'. This
allows you to load the config file by filename, and gets away from the
issue of accidentally importing a file somewhere else in pythons
search path. Also, calling imp.load_source will reload the module when
called a second time.
>
http://docs.python.org/lib/module-imp.html

Why not just use execfile() ?

http://www.python.org/doc/2.2.3/lib/built-in-funcs.html

[conf.py]
a = 1
b = 2
class c:
a = "hello"
b = "world"
[/end conf.py]
'world'



There are so many ways potential solutions to your problem that,
without any more details, it is hard to suggest anything.

Here are some potential solutions:

ConfigParser - module for handling ini files
xml - several built-in modules for handling XML files
sqlite3 - a `lite' SQL database built-in in python 2.5 + (can be used
for config data)
windows registry - _winreg module
pickle - serialize python objects
marshal - similar to pickle, only works for simple objects

Those are just the built-in solutions. If you wanna look at 3rd party
solutions, prepare for overload. The number of alternative INI parsers
alone is staggering.

Also, there are many ways to organize your data and use a solution
similar to what you are already using.

I guess what I'm trying to say is... don't roll your own, it would be
a waste of time, this problem has been solved 100s of times. That is,
unless you want to do it for fun.

Matt

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, May 06 2008)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

 

[Jorge Vargas]

Why not just use execfile() ?

http://www.python.org/doc/2.2.3/lib/built-in-funcs.html

that is very bad for this case, from what he is suggesting this is a
server install so you are basically giving a vector of remote code
execution (same with the first approach) but then execfile has the
issue that it goes into your current namespace possibly creating a
namespace crash which is even worst because an attacker can shallow
say your auth module with something that will just return.

[conf.py]
a = 1
b = 2
class c:
a = "hello"
b = "world"
[/end conf.py]

conf = imp.load_source("conf", "./conf.py")
conf.a


1

2

'hello'



conf.c.b

'world'



There are so many ways potential solutions to your problem that,
without any more details, it is hard to suggest anything.

Here are some potential solutions:

ConfigParser - module for handling ini files
xml - several built-in modules for handling XML files
sqlite3 - a `lite' SQL database built-in in python 2.5 + (can be used
for config data)
windows registry - _winreg module
pickle - serialize python objects
marshal - similar to pickle, only works for simple objects

Those are just the built-in solutions. If you wanna look at 3rd party
solutions, prepare for overload. The number of alternative INI parsers
alone is staggering.

Also, there are many ways to organize your data and use a solution
similar to what you are already using.

I guess what I'm trying to say is... don't roll your own, it would be
a waste of time, this problem has been solved 100s of times. That is,
unless you want to do it for fun.

Matt

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, May 06 2008)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

 

[M.-A. Lemburg]

that is very bad for this case, from what he is suggesting this is a
server install so you are basically giving a vector of remote code
execution (same with the first approach) but then execfile has the
issue that it goes into your current namespace possibly creating a
namespace crash which is even worst because an attacker can shallow
say your auth module with something that will just return.

Not really: you can pass in the globals and locals dictionary
to execfile(), just like you can with __import__(), but you can't
with imp.load_source(), so execfile() is safer than using import
directly or via the imp module.

I don't see a problem with remote code execution - there's nothing
"remote" in loading a local config

Also, you can pretty prevent all code execution that goes beyond simple
evals by restricting the globals, e.g.

globals = {'__builtins__':{}}
execfile('config.py', globals)

Doing so will prevent imports and doesn't expose the builtins
either, so there's little left for a user of the server
to manipulate - besides doing so by just inserting his own
os.py or similar common Python module would be far easier
anyway ;-)

[conf.py]
a = 1
b = 2
class c:
a = "hello"
b = "world"
[/end conf.py]


conf = imp.load_source("conf", "./conf.py")
conf.a

1

conf.b

2

conf.c.a

'hello'

conf.c.b

'world'



There are so many ways potential solutions to your problem that,
without any more details, it is hard to suggest anything.

Here are some potential solutions:

ConfigParser - module for handling ini files
xml - several built-in modules for handling XML files
sqlite3 - a `lite' SQL database built-in in python 2.5 + (can be used
for config data)
windows registry - _winreg module
pickle - serialize python objects
marshal - similar to pickle, only works for simple objects

Those are just the built-in solutions. If you wanna look at 3rd party
solutions, prepare for overload. The number of alternative INI parsers
alone is staggering.

Also, there are many ways to organize your data and use a solution
similar to what you are already using.

I guess what I'm trying to say is... don't roll your own, it would be
a waste of time, this problem has been solved 100s of times. That is,
unless you want to do it for fun.

Matt

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, May 06 2008)

Python/Zope Consulting and Support ... http://www.egenix.com/
mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, May 06 2008)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611