Newbie Alert: Help me store constants pythonically

[Brendan]

Hi all

I'm new to Python (and programming in general), and I can't decide what
is the most 'pythonic' way to approach a problem. Your advice would be
appreciated.

I have a bunch of 'scans', containing the data measured from one of
several types of 'model'. Each 'model' has different values for a list
of constants I want to create 'analyses' which interpret the data from
the scans, with reference the appropriate model. So far, I have come
up with three strategies to store this information, illustrated below
(I've simplified the constant list to just two):

1) Store the model constants as class instances:

class Model:
def __init__(self, numBumps, sizeOfBumps):
self.__dict__.update(locals()); del self.self

MODEL1 = Model(1, 2)
MODEL2 = Model(3, 4)
#etc

class Analysis:
def __init__(self, scanData, model):
#do analysis

2) Store the model constants as class variables:

class MODEL1:
numBumps = 1
sizeOfBumps = 2

class MODEL2:
numBumps = 3
sizeOfBumps = 4

class Analysis:
#as with 1

3) Store the model constants as class variables of the analysis class:

class Model1Analysis:
numBumps = 1
sizeOfBumps = 2

def __init__(self, scanData):
#do analysis

class Model2Analysis(Model1Analysis):
numBumps = 3
sizeOfBumps = 4

There may be more options, but at this point I became paralyzed with
choice. I worry about getting stuck with an unworkable structure since
I don't have the experience to decide the merits of each in advance. I
get way too frustrated about these things
Brendan

 

[Francesco Bochicchio]

Il Sun, 06 Nov 2005 08:33:17 -0800, Brendan ha scritto:

Hi all

I'm new to Python (and programming in general), and I can't decide what
is the most 'pythonic' way to approach a problem. Your advice would be
appreciated.

I have a bunch of 'scans', containing the data measured from one of
several types of 'model'. Each 'model' has different values for a list
of constants I want to create 'analyses' which interpret the data from
the scans, with reference the appropriate model. So far, I have come
up with three strategies to store this information, illustrated below
(I've simplified the constant list to just two):

1) Store the model constants as class instances:

class Model:
def __init__(self, numBumps, sizeOfBumps):
self.__dict__.update(locals()); del self.self

MODEL1 = Model(1, 2)
MODEL2 = Model(3, 4)
#etc

class Analysis:
def __init__(self, scanData, model):
#do analysis

2) Store the model constants as class variables:

class MODEL1:
numBumps = 1
sizeOfBumps = 2

class MODEL2:
numBumps = 3
sizeOfBumps = 4

class Analysis:
#as with 1

3) Store the model constants as class variables of the analysis class:

class Model1Analysis:
numBumps = 1
sizeOfBumps = 2

def __init__(self, scanData):
#do analysis

class Model2Analysis(Model1Analysis):
numBumps = 3
sizeOfBumps = 4

There may be more options, but at this point I became paralyzed with
choice. I worry about getting stuck with an unworkable structure since
I don't have the experience to decide the merits of each in advance. I
get way too frustrated about these things
Brendan

My vote would go to the first option, but I don't understand why you use
such a complicate way to set instance attributes. I would do simply

class Model:
def __init__(self, numBumps, sizeOfBumps):
self.numBumps = numBumps
self.sizeofBumps = sizeOfBumps

or, if you want to generalize the arguments of the constructor:

class Model:
def __init__(self, **model_attributes):
for attname, attvalue in model_attributes.items():
setattr(self, attname, attvalue)

As for why to choose the first option... is the one that causes you to
avoid useless code repetitions and to keep the Analysis code nicely
generic.

Ciao

 

[Brendan]

Thanks for the vote FB. The reason I'm using that method for assigning
instance attributes is that the argument list for __init__ is LOOONG.
(There are many constants, I only gave two for the examples). I wanted
to avoid typing them out twice.

 

[Steven D'Aprano]

Hi all

I'm new to Python (and programming in general), and I can't decide what
is the most 'pythonic' way to approach a problem. Your advice would be
appreciated.

I have a bunch of 'scans', containing the data measured from one of
several types of 'model'. Each 'model' has different values for a list
of constants I want to create 'analyses' which interpret the data from
the scans, with reference the appropriate model. So far, I have come
up with three strategies to store this information, illustrated below
(I've simplified the constant list to just two):

1) Store the model constants as class instances:

class Model:
def __init__(self, numBumps, sizeOfBumps):
self.__dict__.update(locals()); del self.self

MODEL1 = Model(1, 2)
MODEL2 = Model(3, 4)
#etc

My vote would go with this one, as it is the closest to
a C-style struct or a Pascal record. If you have to
change the field list, you only need to change it in
one place, the class.

2) Store the model constants as class variables:

class MODEL1:
numBumps = 1
sizeOfBumps = 2

class MODEL2:
numBumps = 3
sizeOfBumps = 4

Duplicated data... yuck. Same with method 3.

There may be more options, but at this point I became paralyzed with
choice. I worry about getting stuck with an unworkable structure since
I don't have the experience to decide the merits of each in advance. I
get way too frustrated about these things

If all the constants have the same field structure,
stick with method 1.

In a later post, you wrote:

> The reason I'm using that method for assigning
> instance attributes is that the argument list for
> __init__ is LOOONG. (There are many constants, I

only > gave two for the examples). I wanted to avoid
typing > them out twice.

You could always copy-and-paste...

How many is LOOONG? Ten? Twenty? One hundred?

If it is closer to 100 than to 10, I would suggest
putting your constants into something like an INI file:

[MODEL1] # or something more meaningful
numBumps: 1
sizeOfBumps: 99

[MODEL2]
numBumps: 57
sizeOfBumps: 245


etc.

Then sub-class the ConfigParser to create struct-like
objects from an ini file. Something vaguely like:

# warning: pseudo-code, seriously not tested!!!
class Config2Consts(ConfigParser):
"""Reads an INI file, and creates a struct-like
object from each section, storing that object
in the local scope with the name of the section.
"""

class Struct:
def __init__(self):
pass

def make_single_constant_object(self, sectionname):
obj = Struct()
for field in INI_FILE[sectionname]:
obj.__dict__[field.name] = field.value
locals().update({sectionname: obj})



If you are likely to be changing the constants (either
names, field names, or values) an hour or three
developing this code will save you a lot of heart-ache
later.

 

[Brendan]

How many is LOOONG? Ten? Twenty? One hundred?

About 50 per Model

If it is closer to 100 than to 10, I would suggest
putting your constants into something like an INI file:

[MODEL1] # or something more meaningful
numBumps: 1
sizeOfBumps: 99

[MODEL2]
numBumps: 57
sizeOfBumps: 245


etc.

Then sub-class the ConfigParser to create struct-like
objects from an ini file. Something vaguely like:

# warning: pseudo-code, seriously not tested!!!
class Config2Consts(ConfigParser):
"""Reads an INI file, and creates a struct-like
object from each section, storing that object
in the local scope with the name of the section.
"""

class Struct:
def __init__(self):
pass

def make_single_constant_object(self, sectionname):
obj = Struct()
for field in INI_FILE[sectionname]:
obj.__dict__[field.name] = field.value
locals().update({sectionname: obj})



If you are likely to be changing the constants (either
names, field names, or values) an hour or three
developing this code will save you a lot of heart-ache
later.

Thanks for your reply Steve. I like this suggestion because it
separates my config data from the code, which could mean less headaches
editing the values later. It also lets me keep my constants
language-neutral, which is good because I haven't convinced my boss yet
that Python's the way to go.

The only downside is that it doesn't do any 'name checking', so any
mistakes in my config file won't show up until the middle of the
analysis. Maybe this is a 'static language' mode of thinking, but it
seems risky. Also my config files have (a tiny bit of) nested
structure, such as:

Model1(
numBumps = 1
sizeOfBumps = 2
transversePlanes = [
Plane(type=3, z=4),
Plane(type=5, z=6),
Plane(type=3, z=8)
]
)

which I'm not sure the .ini format can easily support. I could use
(key buzzword voice) XML, but I fear that might send me down the
'overcomplicating things' path. Your suggestion has given me some new
places to search Google (configparser, python config files), so I'll
look around for better ideas.

Brendan

 

[Jorge Godoy]

seems risky. Also my config files have (a tiny bit of) nested
structure, such as:

Model1(
numBumps = 1
sizeOfBumps = 2
transversePlanes = [
Plane(type=3, z=4),
Plane(type=5, z=6),
Plane(type=3, z=8)
]
)

which I'm not sure the .ini format can easily support. I could use
(key buzzword voice) XML, but I fear that might send me down the
'overcomplicating things' path. Your suggestion has given me some new
places to search Google (configparser, python config files), so I'll
look around for better ideas.

Take a look at pickle (specially cPickle) module for pickling and unpickling
data directly from / to Python types. What you want can be mapped to
dictionaries of lists of dictionaries.

 

[Ron Adam]

How many is LOOONG? Ten? Twenty? One hundred?

About 50 per Model

If it is closer to 100 than to 10, I would suggest
putting your constants into something like an INI file:

[MODEL1] # or something more meaningful
numBumps: 1
sizeOfBumps: 99

[MODEL2]
numBumps: 57
sizeOfBumps: 245

It looks to me like you may want a data file format that can be used by
various tools if possible.

A quick search for 3d file formats finds the following list. If any of
these are close to what you need you may have an additional benefit of
ready made tools for editing.

http://astronomy.swin.edu.au/~pbourke/dataformats/

which I'm not sure the .ini format can easily support. I could use
(key buzzword voice) XML, but I fear that might send me down the
'overcomplicating things' path. Your suggestion has given me some new
places to search Google (configparser, python config files), so I'll
look around for better ideas.

Brendan

One approach is to just store them as Python dictionaries. Then just
import it and use it where it's needed.

# models.py

M1 = dict(
numBumps=1,
sizeOfBumps=2,
transversePlanes=[
dict(type=3, z=4),
dict(type=5, z=6),
dict(type=3, z=8) ]
)

M2 = ...


Then in your application...

# makemodels.py

import models

class Model(object):
def __init__( self, numBumps=None, sizOfBumps=None,
transversePlanes=None ):
self.numBumps = numBumps
self.sizeOfBumps = sizeOfBumps
self.transversePlanes = []
for p in tranversePlanes:
self.transversePlanes.append(Plane(**p))

mod1 = Model(**models.M1)


This may be good to use until you decide how else to do it. You can
easily write the dictionaries to a text file in the chosen format later
and that will tell you what you need to do to read the file back into
the dictionaries as it will just be reversed.

Cheers,
Ron

 

[Steven D'Aprano]

Thanks for your reply Steve. I like this suggestion because it
separates my config data from the code, which could mean less headaches
editing the values later. It also lets me keep my constants
language-neutral, which is good because I haven't convinced my boss yet
that Python's the way to go.

Cool

The only downside is that it doesn't do any 'name checking', so any
mistakes in my config file won't show up until the middle of the
analysis.

That's easy: in your ConfigParser subclass, create a
method:

def validate_name(self, name):
if not name in self.__class__.ALLOWED_NAME_LIST:
raise ValueError("Name '%s' not recognised." \
% name)

then call that method before adding the attribute to
the object __dict__. ALLOWED_NAME_LIST should be a
class attribute, or perhaps use a global instead.

Maybe this is a 'static language' mode of thinking, but it
seems risky. Also my config files have (a tiny bit of) nested
structure, such as:

Model1(
numBumps = 1
sizeOfBumps = 2
transversePlanes = [
Plane(type=3, z=4),
Plane(type=5, z=6),
Plane(type=3, z=8)
]
)

That's a little trickier, but not much. You effectively
are just creating a struct where some attributes are
themselves structs.

which I'm not sure the .ini format can easily support.

Perhaps not easily, but maybe you can modify the code
or subclass it.


Good luck!

 

[Nicola Larosa]

Also my config files have (a tiny bit of) nested

structure, such as:

Model1(
numBumps = 1
sizeOfBumps = 2
transversePlanes = [
Plane(type=3, z=4),
Plane(type=5, z=6),
Plane(type=3, z=8)
]
)

which I'm not sure the .ini format can easily support. I could use
(key buzzword voice) XML, but I fear that might send me down the
'overcomplicating things' path. Your suggestion has given me some new
places to search Google (configparser, python config files), so I'll
look around for better ideas.

You may want to look into ConfigObj:

http://www.voidspace.org.uk/python/configobj.html#config-files

--
Nicola Larosa - (e-mail address removed)

No inventions have really significantly eased the cognitive difficulty
of writing scalable concurrent applications and it is unlikely that any
will in the near term. [...] Most of all, threads do not help, in fact,
they make the problem worse in many cases. -- G. Lefkowitz, August 2005

 

[Brendan]

Thanks for all the suggestions everyone.

After a bit of googling on the c.l.p. group, I see that building config
files is one of those 'Everyone has a favourite way of doing it' types
of problems, with lots of reimplementations. I should change the
thread topic to "Yet Another Config File Question"!

Based on my requirements (thanks again for helping me identify these),
my config file should:

a) store numbers, strings, bools, keyed dictionaries and un-keyed lists
b) store nested structures of the above
c) be language neutral, preferably well supported in other languages,
and editors
d) have a simple api in Python, preferably translating values to native
types
e) be validated when read

I've checked out ConfigParser, ConfigObj, Pickle, PyYaml and
gnossis.xml.serialize, and none meet all the above criteria (though
they're all neat).

So I've decide to use ...drumroll please.... plistlib (
http://svn.python.org/projects/python/trunk/Lib/plat-mac/plistlib.py ).
Why plists?

- I've got a simple api (readPlist(pathOrFile), writePlist(rootObject,
pathOrFile) ) already installed with macPython
- I have a dirt simple plist editor for knocking out the hundreds of
config values I need (
http://homepage.mac.com/bwebster/plisteditpro.html )
- The file format is xml, which is well supported in all languages.
- Since plists are the standard approach on the mac (though they aren't
OS specific by definition), the XML schema is well documented
(http://developer.apple.com/documentation/Darwin/Reference/ManPages/man5/plist.5.html),
and shared with thousands of other apps, so I won't have to worry about
obsolescence.

So away I go. If anyone can suggest reasons I should avoid this
approach, please let me know before I get too invested. Otherwise,
this might be a reasonable avenue for standardizing Python. (I hope
that doesn't draw to many flames

Brendan.

 

[Stefan Rank]

on 08.11.2005 17:40 Brendan said the following:
[snip config/properties file needs]

I've checked out ConfigParser, ConfigObj, Pickle, PyYaml and
gnossis.xml.serialize, and none meet all the above criteria (though
they're all neat).

So I've decide to use ...drumroll please.... plistlib (
http://svn.python.org/projects/python/trunk/Lib/plat-mac/plistlib.py ).

like myriads of others, i went through the same dance as you did
(several times, several programming languages), and sadly, i did not
come across plistlib.py in the python case.

as a first glance inspires liking and the lib does not seem to require
anything mac-specific, i would like to ask why it is not in the standard
python distribution?

So away I go. If anyone can suggest reasons I should avoid this
approach, please let me know before I get too invested. Otherwise,
this might be a reasonable avenue for standardizing Python.

I second that.

stefan