Thoughts on language-level configuration support?

[jfager]

I've written a short post on including support for configuration down
at the language level, including a small preliminary half-functional
example of what this might look like in Python, available at
http://jasonfager.com/?p=440.

The basic idea is that a language could offer syntactic support for
declaring configurable points in the program. The language system
would then offer an api to allow the end user to discover a programs
configuration service, as well as a general api for providing
configuration values.

The included example implements the first bit and hints at the third,
defining a function that looks up what variable its output will be
assigned to and tries to find a corresponding value from a
configuration source. It's very preliminary, but I hope it gives a
flavor of the general idea.

Any thoughts or feedback would be greatly appreciated.

 

[Steven D'Aprano]

I've written a short post on including support for configuration down at
the language level, including a small preliminary half-functional
example of what this might look like in Python, available at
http://jasonfager.com/?p=440.

The basic idea is that a language could offer syntactic support for
declaring configurable points in the program.

What's a configuration point? Where do I find them in my programs?

The language system would
then offer an api to allow the end user to discover a programs
configuration service, as well as a general api for providing
configuration values.

Why is an end user changing variables in my program?

The included example implements the first bit and hints at the third,
defining a function that looks up what variable its output will be
assigned to and tries to find a corresponding value from a configuration
source. It's very preliminary, but I hope it gives a flavor of the
general idea.

Not really. It would help if you explained what problem you are trying to
solve, what people do now, and how what you suggest would make that
easier.

Having read your blog post, I'm going to take a wild stab at guessing
what you mean:

* Programs often have variables which relate to user-configurable
options. For example, a basic list files command might include something
like this:

if sys.argv[1] == '-l':
show_long_listing = True
else:
show_long_listing = False
# ...
# much later...
for filename in directory():
if show_long_listing:
print "%s %s %s" % (permissions(), filename, date())
else:
print filename


* The current solution is for the program author to define the interface
and the implementation separately. For example, in the above code
snippet, the command line option '-l' is the interface available to the
end user, while the variable show_long_listing is the implementation used
to implement configuration options in the program.


* You propose to expose the variable show_long_listing and it's ilk
directly to the user, presumable via some service which can interrogate
the program, discover what "configuration variables" are available, and
allow the user to stuff random values into them in the hope of getting
new and exciting bugs^W behaviour.


* This is better than current solutions to the question of getting
configuration options from the user, like getopt and similar, because... ?


* This would be better with syntactic support rather than a library or
module because... ?



Am I close?

 

[jfager]

What's a configuration point? Where do I find them in my programs?

A configuration point is what you create when you use this proposed
feature. You find them in your programs wherever you (or whoever
wrote them) defined them.

Why is an end user changing variables in my program?

Because you used this construct to tell them they could. It's not a
hook for every variable, just the ones you want to expose.

Not really. It would help if you explained what problem you are trying to
solve, what people do now, and how what you suggest would make that
easier.

It's the configuration problem. Right now you would use something
like ConfigParser or optparse to populate some configuration object,
which you would then pass around and extract values from. This would
provide two advantages over these approaches: it would separate "what
can be configured" from "the mechanism by which it is configured" -
i.e., I, programmer, don't have to make a decision about what the end
user has to do to give me those values. And it would allow the
configurable surface of the program to be discoverable; I wouldn't
have to poke through code or documentation that was maintained
separate from the source code.

Having read your blog post, I'm going to take a wild stab at guessing
what you mean:

* Programs often have variables which relate to user-configurable
options. For example, a basic list files command might include something
like this:

if sys.argv[1] == '-l':
    show_long_listing = True
else:
    show_long_listing = False
# ...
# much later...
for filename in directory():
    if show_long_listing:
        print "%s %s %s" % (permissions(), filename, date())
    else:
        print filename

* The current solution is for the program author to define the interface
and the implementation separately. For example, in the above code
snippet, the command line option '-l' is the interface available to the
end user, while the variable show_long_listing is the implementation used
to implement configuration options in the program.

* You propose to expose the variable show_long_listing and it's ilk
directly to the user, presumable via some service which can interrogate
the program, discover what "configuration variables" are available, and
allow the user to stuff random values into them in the hope of getting
new and exciting bugs^W behaviour.

What, exactly, is preventing them from stuffing random values into
them now? The boilerplate of manually reading the param from the
command line and assigning it to a variable makes things magically
safe? If you'll notice in the example code, there's an optional
parameter to process whatever argument is passed in; that's where any
safety checks can be performed.

* This is better than current solutions to the question of getting
configuration options from the user, like getopt and similar, because... ?

Because it means the developer doesn't have to know or care how the
end-user ultimately inserts configured values. Because it provides a
unified mechanism for all configuration instead of a hodge-podge of
different techniques for what is ultimately the same basic task.
Because it lets you quickly make a parameter of a program configurable
(or not) without having to rearchitect the application.

* This would be better with syntactic support rather than a library or
module because... ?

Because a big part of this is the discovery of these endpoints, which
seems like it would best be handled at a lower level than a library
can currently provide. Because finding what variable you're assigning
to currently requires querying the bytecode. Because libraries
inevitably introduce boilerplate that can simply be avoided if its a
language feature. Because the requirement is broad enough and similar
enough across concerns to be provided as a basic service.

- Jason

 

[jfager]

This would be a interesting idea, but ultimately no more than a veneer
over the current set of configuration possibilities.  Quite how such a
system would tell whether to get configuration data from command line
parameters, a config file somewhere, or just pull bytes off the Z-wave
 from Mars, I'm not at all clear.

Not a veneer over; a foundation under. As I acknowledged in my blog
post, this would require a small bit of bootstrapping, to configure
which end-user configuration system was used. But this would simply
point to an adapter, that would map from the desired configuration
interface into the canonical standard api provided by the language
system.

The problem with the current set of configuration possibilities is
that there's nothing really constant between them, unless the
programmer explicitly codes it, even though they're basically
accomplishing the same thing. There's nothing amazingly special about
this proposal, it's just saying: that basic thing that we do in a lot
of different ways, let's make as much of that as possible standard. I
think that cutoff point is below the end-user interface, since there
are a lot of valid ways to want to actually inject config data
(command line, local files, from a db, etc.), but there's no reason
that the end-user interfaces can't rest on top of a lower-level common
format that the programmer can rely on.

Except that you still do.  

No, you don't. If you do, then this proposal hasn't been implemented
(and the little code snippet I provided in my post, please don't
consider that a full implementation, it was just a sketch I threw
together in a really short time). I'm not saying discoverability
magically arises; I'm saying that it's one the main points that would
be deliberately put into place.

You've just specified a different way in
which you have to do this, one that's a good deal less visible in
the code

Why would it be less visible? If it's syntax, you would know exactly
where it was just by looking. I would argue that it's more clear - no
more boilerplate code spent shuffling around config objects that you
peel values out of, just write the code that does the work you're
actually trying to accomplish, with a small bit of syntax to mark that
the end user can provide an alternate value.

and a lot less self-documenting because it's spread out all
over the place.  

Actually, you get the best of both worlds. You get to see clearly in
the code how the configured values are actually used, and you get the
gathered summary of configurable values from the "discoverability"
implementation.

It also has the major disadvantage from my point of
view of requiring Python to do magic in the background to figure out
just what is being configured.

If it's built into the language, it stops being magic, and turns into
"how it works".


- Jason

 

[David Stanek]

I've written a short post on including support for configuration down
at the language level, including a small preliminary half-functional
example of what this might look like in Python, available at
http://jasonfager.com/?p=440.

The basic idea is that a language could offer syntactic support for
declaring configurable points in the program.  The language system
would then offer an api to allow the end user to discover a programs
configuration service, as well as a general api for providing
configuration values.

What value does this have over simply having a configuration file. In
your load testing application you could have easily checked for the
settings in a config object. I think that the discover-ability of
configuration can be handled with example configs and documentation.

 

[rustom]

I am not sure I understand your solution. I certainly think that the
problem is big, very much bigger than is appreciated.
Think of the hoopla in the RoR world about convention-over-
configuration.

On the other hand I feel that emacs is becoming messier and messier
because it has taken up something like your idea. Originally there
was only setq (lisp for assignment). Now there is the whole customize-
mess. Then again I guess its not the idea that is wrong but its
current state of implementation. To elaborate on this mess would be
too OT for this list. Nevertheless its a good starting point for the
kind of thing you are talking of.

 

[jfager]

What value does this have over simply having a configuration file.

"Simply having a configuration file" - okay. What format? What if
the end user wants to keep their configuration info in LDAP? Did the
library I'm including make the same decisions, or do I have to do some
contortions to adapt? Didn't I write basically this exact same code
for the last umpteen projects I worked on, just schlepping around
config objects?

In your load testing application you could have easily checked for the
settings in a config object.

Not really easily, no. It would have been repeated boilerplate across
many different test cases (actually, that's what we started with and
refactored away), instead of a simple declaration that delegated the
checking to the test runner.

I think that the discover-ability of
configuration can be handled with example configs and documentation.

Who's keeping that up to date? Who's making sure it stays in sync
with the code? Why even bother, if you could get it automatically
from the code?

 

[CTO]

On the one hand, I can 110% see why you want to reduce boilerplate
code and provide a discoverable, common mechanism for automating the
two and three-quarters parsers that a lot of applications have to
write to handle a config file, CLI, and/or registry values, but why
introduce a syntax for it? A module would do just fine in terms of
function. Are you worried about the look of it, or do you want to make
a change to make it seem more "mainstream"? I don't see the
rationale.

 

[Steven D'Aprano]

Not a veneer over; a foundation under. As I acknowledged in my blog
post, this would require a small bit of bootstrapping, to configure
which end-user configuration system was used. But this would simply
point to an adapter, that would map from the desired configuration
interface into the canonical standard api provided by the language
system.

Let's talk about a practical example. The ls command has the API that the
"-l" switch means "show me long options". You're suggesting that users
should not interact with the user-interface, but directly with the
implementation. You are, essentially, assuming that there is a one-to-one
correspondence between data that the user inputs and variables in the
implementation, and that users can understand the implementation.

But that's not necessarily the case. The switch -l might affect a dozen
different variables. So instead of the user needing to learn *one*
command line option (or click on one checkbox in a GUI, or whatever), you
expect him to reason "I want to see a long display of my files, so I need
to set the value of line_width to 47, date_style to 3, show_perms to
True, and format_into_columns to -1".

I don't think that's going to fly. Separation of interface and
implementation is a Good Thing.


Or consider another scenario:

def ls:
if '-l' in sys.argv:
file_iterator = SimpleFilenameWriter()
else:
file_iterator = DetailedFilenameWriter()
#...


Under your proposal, the user would somehow have to create the
appropriate instance and feed it to your program. That's simply not
practical! So you still need some sort of proxy variable, virtually
identically as you do now:


conf make_long_list = True

def ls(optionlist):
if make_long_list:
file_iterator = SimpleFilenameWriter()
else:
file_iterator = DetailedFilenameWriter()
#...

The problem with the current set of configuration possibilities is that
there's nothing really constant between them, unless the programmer
explicitly codes it, even though they're basically accomplishing the
same thing. There's nothing amazingly special about this proposal, it's
just saying: that basic thing that we do in a lot of different ways,
let's make as much of that as possible standard.

Over-generalization actually makes things more complicated. I think
you're over-generalizing.


....

Why would it be less visible? If it's syntax, you would know exactly
where it was just by looking.

It could be *anywhere* in your project. It could be in some random
library that you imported, and the user discovers that they can modify
variables you didn't even know existed.

Actually, you get the best of both worlds. You get to see clearly in
the code how the configured values are actually used, and you get the
gathered summary of configurable values from the "discoverability"
implementation.

I've learned to distrust "discovery", ever since I learned that my
doctests, which were supposedly all running without error, in fact hadn't
been discovered at all, and not one single test was running. So even if
you could get this working, I'd be luke-warm on the idea.

 

[jfager]

I am not sure I understand your solution.

Any questions, please ask.

I certainly think that the
problem is big, very much bigger than is appreciated.
Think of the hoopla in the RoR world about convention-over-
configuration.

Certainly, it's a big problem. I'm not saying this will solve it
completely, or anything like that. I just want to identify the most
common, basic needs that can be solved once and provided as a service
to the programmer and the end user, and to get rid of some of the
repetitive work around dealing with configuration.

On the other hand I feel that emacs is becoming messier and messier
because it has taken up something like your idea.  Originally there
was only setq (lisp for assignment).  Now there is the whole customize-
mess.  Then again I guess its not the idea that is wrong but its
current state of implementation.  To elaborate on this mess would be
too OT for this list.  Nevertheless its a good starting point for the
kind of thing you are talking of.

I don't think emacs is a great parallel, for a couple of reasons.
First, the customize system seems weird and out of place in a world
where the entrenched configuration mechanism is 'program it directly
in your .emacs file' - by the time you know enough emacs to be able to
improve the customize interface, you don't want to use it anymore.
Also, the lack of namespacing in elisp means there's not a great way
to automatically name and organize these points, so again it falls to
the individual programmers to decide, and they inevitably decide on
something slightly different from each other.

 

[jfager]

On the one hand, I can 110% see why you want to reduce boilerplate
code and provide a discoverable, common mechanism for automating the
two and three-quarters parsers that a lot of applications have to
write to handle a config file, CLI, and/or registry values, but why
introduce a syntax for it? A module would do just fine in terms of
function. Are you worried about the look of it, or do you want to make
a change to make it seem more "mainstream"? I don't see the
rationale.

Syntax is kind of a rubbery term. I just mean that there should be a
clear and easy way to do it, that it should be considered a basic
service, and that if the best way to satisfy all the goals is to
integrate it directly into the language, that shouldn't be shied away
from.

The example that I have on my blog post, I consider that 'syntax',
even though it's implemented as a function, mainly just because it
digs into the bytecode and modifies the normal way a function is
evaluated (the function's value is determined by where the output
would go).

 

[jfager]

Let's talk about a practical example. The ls command has the API that the
"-l" switch means "show me long options". You're suggesting that users
should not interact with the user-interface, but directly with the
implementation. You are, essentially, assuming that there is a one-to-one
correspondence between data that the user inputs and variables in the
implementation, and that users can understand the implementation.

No, not at all. I'm saying that the programmer shouldn't have to care
what the end-user's interface is, not there's no end-user interface at
all. For the program 'ls', why should I, the programmer, care at all
how the end user actually specifies that they want 'long options'? I
might think or know that a command line argument is the 'best' way,
but why should I even worry about it? I should just tell them that
option exists, and then they can choose how to give me a value for it
in whatever way they please.

But that's not necessarily the case. The switch -l might affect a dozen
different variables. So instead of the user needing to learn *one*
command line option (or click on one checkbox in a GUI, or whatever), you
expect him to reason "I want to see a long display of my files, so I need
to set the value of line_width to 47, date_style to 3, show_perms to
True, and format_into_columns to -1".

Not at all, not even a little bit. Why wouldn't you just say some
variable 'long-lines' is configurable, and then use that to do all the
work you would have done by manually parsing the command-line
arguments for the -l flag? You, the programmer, still have complete
control over what is or isn't visible to the end user, it's not like
I'm advocating that every variable in the system automagically become
end-user configurable without programmer input.

I don't think that's going to fly. Separation of interface and
implementation is a Good Thing.

Agreed, that's the whole point of this. Use the interface provided by
the language, then let the end user provide their own implementation
(of course, there will be basic ones provided out of the box) of how
to specify their configuration values.

Or consider another scenario:

def ls:
    if '-l' in sys.argv:
        file_iterator = SimpleFilenameWriter()
    else:
        file_iterator = DetailedFilenameWriter()
    #...

Under your proposal, the user would somehow have to create the
appropriate instance and feed it to your program. That's simply not
practical! So you still need some sort of proxy variable, virtually
identically as you do now:

conf make_long_list = True

def ls(optionlist):
    if make_long_list:
        file_iterator = SimpleFilenameWriter()
    else:
        file_iterator = DetailedFilenameWriter()
    #...

This second one is what I intended (no optionlist param needed,
though). I'm curious - why do you think this is a bad thing? One toy
example looks pretty similar to how you would do things now, and
you're ready to throw out the whole concept? Notice that you did
actually gain something that I think is significant - you no longer
have any hardcoded reference to the fact that make_long_list is
defined as a command line parameter.

Over-generalization actually makes things more complicated. I think
you're over-generalizing.

Keep throwing out examples of where this makes things more
complicated, it's good to work through all the concerns.

It could be *anywhere* in your project. It could be in some random
library that you imported, and the user discovers that they can modify
variables you didn't even know existed.

If that's true, it's because the developer consciously said at some
point, "This needs to be configurable". And I, as an occasional end
user of software products, would sometimes love to find I could modify
variables I didn't even know existed. Discoverability is a good
thing, much better than having a potentially useful knob hidden away
behind stale documentation or buried deep in source code somewhere.

I've learned to distrust "discovery", ever since I learned that my
doctests, which were supposedly all running without error, in fact hadn't
been discovered at all, and not one single test was running. So even if
you could get this working, I'd be luke-warm on the idea.

I don't know what to say to this, except that it seems like the
penalty for missing something in the discovery phase of what I'm
proposing would be either less dangerous or more obvious, less
dangerous because the program would just fall back to a default and
continue on (you didn't come across some variable you didn't even know
existed), more obvious because you might be expecting to be able to
configure something specific, or because possibly no value would be
provided and an error would be thrown.


- Jason

 

[Kay Schluehr]

I've written a short post on including support for configuration down
at the language level, including a small preliminary half-functional
example of what this might look like in Python, available athttp://jasonfager.com/?p=440.

The basic idea is that a language could offer syntactic support for
declaring configurable points in the program.  The language system
would then offer an api to allow the end user to discover a programs
configuration service, as well as a general api for providing
configuration values.

The included example implements the first bit and hints at the third,
defining a function that looks up what variable its output will be
assigned to and tries to find a corresponding value from a
configuration source.  It's very preliminary, but I hope it gives a
flavor of the general idea.

Any thoughts or feedback would be greatly appreciated.

The problem with your idea is that those declared declaration points
can be overlooked no matter how much syntactical support is added.
Lets say a resource file is loaded and there are a few of the config-
properties declared in modules you have written. Now an object wants
to access a resource defined in the file and fails because the
resource providing property could not be found since the property
defining module wasn't loaded yet and the property couldn't register
itself. That's why things are centralized as in optparse and the
workflow is designed upfront or things are implemented locally and
individual units have to take care of their own.

 

[jfager]

The problem with your idea is that those declared declaration points
can be overlooked no matter how much syntactical support is added.
Lets say a resource file is loaded and there are a few of the config-
properties declared in modules you have written. Now an object wants
to access a resource defined in the file and fails because the
resource providing property could not be found since the property
defining module wasn't loaded yet and the property couldn't register
itself. That's why things are centralized as in optparse and the
workflow is  designed upfront or things are implemented locally and
individual units have to take care of their own.

What is a resource file? How am I accessing the "resource-providing
property" of an unloaded module? That is, if a module isn't loaded,
how do I know about its properties? Or conversely, if I know about
its properties, why isn't the module loaded? In other words, why
would accessing a configuration point be any different than accessing
any other name in a module?

 

[jfager]

"Simply having a configuration file" - okay.  What format?  What if
the end user wants to keep their configuration info in LDAP?  Did the
library I'm including make the same decisions, or do I have to do some
contortions to adapt?  Didn't I write basically this  exact same code
for the last umpteen projects I worked on, just schlepping around
config objects?

Ah I see your point here. During PyCon I was trying to add the ability
to inject configuration into objects that are constructed by the
snake-guice framework. The code is not yet in the Subversion
repository, but I did brain dump a little documentation[0]. It is
still very much a work in progress.

0.http://code.google.com/p/snake-guice/wiki/InjectingConfiguration

This is getting close I think it would be nice if you didn't have
to come up with your own names (so that projects across different
authors would share more or less the same naming structure), and if
those names didn't encode their expectation of a particular end-user
configuration scheme.

 

[David Stanek]

"Simply having a configuration file" - okay.  What format?  What if
the end user wants to keep their configuration info in LDAP?  Did the
library I'm including make the same decisions, or do I have to do some
contortions to adapt?  Didn't I write basically this  exact same code
for the last umpteen projects I worked on, just schlepping around
config objects?

Ah I see your point here. During PyCon I was trying to add the ability
to inject configuration into objects that are constructed by the
snake-guice framework. The code is not yet in the Subversion
repository, but I did brain dump a little documentation[0]. It is
still very much a work in progress.

0.http://code.google.com/p/snake-guice/wiki/InjectingConfiguration

This is getting close  I think it would be nice if you didn't have
to come up with your own names (so that projects across different
authors would share more or less the same naming structure), and if
those names didn't encode their expectation of a particular end-user
configuration scheme.

For my purpose I am writing the glue infrastructure that allows
components to be put together within an application. What I am missing
is a schema-like way to define configuration files. I have debated
starting a project to do that, but at this time I'm already
overextended

 

[Lorenzo Gatti]

A configuration "service"? An "end user" that bothers to discover it?
API for "providing" configuration "values"? This suggestion, and the
companion blog post, seem very distant from the real world for a
number of reasons.

1) Users want to supply applications with the least amount of useful
configuration information as rarely and easily as possible, not to use
advanced tools to satisfy an application's crudely expressed
configuration demands.

Reducing inconvenience for the user entails sophisticated and mostly
ad hoc techniques: deciding without asking (e.g. autoconf looking into
C compiler headers and trying shell commands or countless applications
with "user profiles" querying the OS for the current user's home
directory), asking when the software is installed (e.g. what 8 bit
character encoding should be used in a new database), designing
sensible and safe defaults.

2) Practical complex configuration files (or their equivalent in a DB,
a LDAP directory, etc.) are more important and more permanent than the
applications that use them; their syntax and semantics should be
defined by external specifications (such as manuals and examples), not
in the code of a particular implementation.

User documentation is necessary, and having a configuration mechanism
that isn't subject to accidents when the application is modified is
equally important.

3) Configuration consisting of values associated with individual
variables is an unusually simple case. The normal case is translating
between nontrivial sequential, hierarchical or reticular data
structures in the configuration input and quite different ones in the
implementation.

4) Your actual use case seems to be providing a lot of tests with a
replacement for the "real" configuration of the actual application.
Branding variables as "configuration" all over the program isn't an
useful way to help the tests and the actual application build the same
data structures in different ways.

"Simply having a configuration file" - okay. What format? What if
the end user wants to keep their configuration info in LDAP?

Wait a minute. Reading the "configuration" from a live LDAP directory
is a major feature, with involved application specific aspects (e.g.
error handling) and a solid justification in the application's
requirements (e.g. ensuring up to date authentication and
authorization data), not an interchangeable configuration provider and
certainly not something that the user can replace.

Deciding where the configuration comes from is an integral part of the
design, not something that can or should be left to the user: there
can be value in defining common object models for various sources of
configuration data and rules to combine them, like e.g. in the Spring
framework for Java, but it's only a starting point for the actual
design of the application's configuration.

Not really easily, no. It would have been repeated boilerplate across
many different test cases (actually, that's what we started with and
refactored away), instead of a simple declaration that delegated the
checking to the test runner.

A test runner has no business configuring tests beyond calling generic
setup and teardown methods; tests can be designed smartly and factored
properly to take care of their own configuration without repeating
"boilerplate".

Who's keeping that up to date? Who's making sure it stays in sync
with the code? Why even bother, if you could get it automatically
from the code?

It's the code that must remain in sync with the documentation, the
tests, and the actual usage of the application. For example, when did
you last see incompatible changes in Apache's httpd.conf?

You seem to think code is central and actual use and design is a
second class citizen. You say in your blog post: "Users shouldn’t have
to pore through the code to find all the little bits they can tweak".
They shouldn't because a well designed application has adequate
documentation of what should be configured in the form of manuals,
wizards, etc. and they shouldn't because they don't want to tweak
little bits, not even if they have to.

Regards,

Lorenzo Gatti

 

[jfager]

A configuration "service"? An "end user" that bothers to discover it?
API for "providing" configuration "values"? This suggestion, and the
companion blog post, seem very distant from the real world for a
number of reasons.

1) Users want to supply applications with the least amount of useful
configuration information as rarely and easily as possible, not to use
advanced tools to satisfy an application's crudely expressed
configuration demands.

It depends on the user, doesn't it? In the vast majority of cases,
you're right, the end user is more than happy (or at least, willing)
to use whatever the software developer tells them to. Command line
interfaces, .ini files, whatever, there's no reason the language
couldn't ship with a set of standard adapters (and that the
bootstrapping question could be bypassed by the developer offering a
'hint' about which they think would be best, so the user wouldn't have
to care unless they wanted to).

I spend a lot of time in the Java world (yes, I know), where XML
reigns supreme. I would *love* to be able to seamlessly replace that
with something human readable.

Reducing inconvenience for the user entails sophisticated and mostly
ad hoc techniques: deciding without asking (e.g. autoconf looking into
C compiler headers and trying shell commands or countless applications
with "user profiles" querying the OS for the current user's home
directory), asking when the software is installed (e.g. what 8 bit
character encoding should be used in a new database), designing
sensible and safe defaults.

It seems that something like autoconf would be the perfect consumer
for something like this, both for querying what could be set in and
for injecting values back into the application.

I don't understand your point about "countless applications with "user
profiles" querying the OS for the current user's home directory", so
I'm going to skip it.

Asking for configuration when the software is installed, for something
that can't be changed be later - there are still multiple ways to want
to interact with the code that's doing that setup, aren't there? A
GUI wizard, interactively on the command line, an automated
installation script. And, for the main program - you know that's a
one-off thing, why would you mark it as configurable every time the
program runs?

As for deciding sensible and safe defaults, I don't see how what I
proposed at all ignores that necessity, at least any more than any
other configuration technique.

2) Practical complex configuration files (or their equivalent in a DB,
a LDAP directory, etc.) are more important and more permanent than the
applications that use them; their syntax and semantics should be
defined by external specifications (such as manuals and examples), not
in the code of a particular implementation.

User documentation is necessary, and having a configuration mechanism
that isn't subject to accidents when the application is modified is
equally important.

Developers have to take care to maintain backwards compatibility with
entrenched config formats today, why is it a strike against this idea
that that need would continue? It's not like an implementation of
this would seek out valuable existing configurations and destroy them
whenever the code changed.

Right now, when you have a rich configuration format, and you need to
make code changes, how do you deal with the problem? You have a layer
of indirection, one that's responsible for keeping the end-user
configuration more or less constant while mapping that configuration
to the changing code underneath. There's no reason that would have to
change - your project gets complicated enough, just start maintaining
your own consumer module. Indirection is valuable from both sides.

3) Configuration consisting of values associated with individual
variables is an unusually simple case. The normal case is translating
between nontrivial sequential, hierarchical or reticular data
structures in the configuration input and quite different ones in the
implementation.

I don't think it's an "unusually" simple case. It's a question of
what you're writing. Most of what I use python is fairly small
projects that started out as one-off scripts that I gradually build
up.

And, yes, how this would handle more complicated config structures is
something that needs to be given more thought.

4) Your actual use case seems to be providing a lot of tests with a
replacement for the "real" configuration of the actual application.
Branding variables as "configuration" all over the program isn't an
useful way to help the tests and the actual application build the same
data structures in different ways.

Sorry, I'm not understanding you here. Are you referring to the
testing app I mentioned in my post? It was a webservice load testing
application. Each test was more like a full little script defining a
scenario that could be tweaked to produce different load profiles; the
configuration requirement was to expose to the end user who was
running the load test what, exactly, could be tweaked. Doing this in
a declarative way in the scenario files was *very* useful, there
really wasn't a better way to solve the problem.

Wait a minute. Reading the "configuration" from a live LDAP directory
is a major feature, with involved application specific aspects (e.g.
error handling) and a solid justification in the application's
requirements (e.g. ensuring up to date authentication and
authorization data), not an interchangeable configuration provider and
certainly not something that the user can replace.

I'm not disputing this. I never intended to suggest I thought that
the configuration providers would be trivial pieces of software.

Deciding where the configuration comes from is an integral part of the
design, not something that can or should be left to the user: there
can be value in defining common object models for various sources of
configuration data and rules to combine them, like e.g. in the Spring
framework for Java, but it's only a starting point for the actual
design of the application's configuration.

This is just a starting point, too; I didn't mean to suggest
otherwise, apologies if I gave that impression.

A test runner has no business configuring tests beyond calling generic
setup and teardown methods; tests can be designed smartly and factored
properly to take care of their own configuration without repeating
"boilerplate".

Yes, they can, which is why we refactored them to do so as the system
grew from 'one-off script' to 'load-testing framework'. And the test
runner acting as a middleman between the scripts and the end-user,
that was invaluable, really. I promise. It let us run the same tests
easily from the command line, from a web application, from a nightly
batch process, etc, to add new tests quickly and to manage the
configuration without going crazy.

It's the code that must remain in sync with the documentation, the
tests, and the actual usage of the application. For example, when did
you last see incompatible changes in Apache's httpd.conf?

You seem to think code is central and actual use and design is a
second class citizen. You say in your blog post: "Users shouldn’t have
to pore through the code to find all the little bits they can tweak".
They shouldn't because a well designed application has adequate
documentation of what should be configured in the form of manuals,
wizards, etc. and they shouldn't because they don't want to tweak
little bits, not even if they have to.

The amount of software with thorough and current hand-maintained
documentation and manuals, I would guess that it's vastly outnumbered
by that without, and anyways, nothing in this proposal precludes any
of that. This isn't "the software developer suddenly has no
responsibility for how their app is maintained", this is "the software
developer has a standard set of tools for handling common
configuration tasks, and the end user doesn't necessarily have their
hands tied by developer decisions."


Thanks for your feedback, I really do appreciate the engagement on the
topic.

- Jason

 

[Lawrence D'Oliveiro]

In message <36148830-22c0-4f19-ab23-

I've written a short post on including support for configuration down
at the language level ...

If you're advocating anything resembling php.ini, you deserve to die.

 

[jfager]

In message <36148830-22c0-4f19-ab23-



If you're advocating anything resembling php.ini, you deserve to die.

I'm not. I don't.