Storing configuration data

Render a configuration file template

Since configuration management systems like Chef or Puppet becomes more and
more popular, configuration files are no longer edited by hand. One core
feature of a CM system is to render configuration files from templates.
But we need to tell the system what should be inserted into the template for
the placeholders. So we need to store some data that is used when a configuration
file is generated.

Key-Value store

So my first attempt to store this kind of data was to use a Key-Value store
like Chef’s data bags. You can store a hash structure under a certain key
name. For example:

|_admins
    |_charlie
    |_bob
    |_tom

Now you can iterate over the ‘admins’ key and generate some configurations
with an entry for each admin.
This works quit well for simple scenarios, but I found more and more use cases
where I need something different.

Hierarchical store

My main problem was to deal with exceptions. A good example might be the proxy
setting for a system or an application. Lets say we have a website that consists
of three separated applications. We can make a general conclusion that the
outgoing proxy for this setup is found under the domain name proxy.setup.com.
But now there is this one application that needs another proxy, e.g.
exceptionalproxy.setup.com. So we need the chance to configure a general
setting and then refine this setting for a special use case.
If we store our configuration data in a directory structure and merge
every level of this structure for a particula path, we get this problem
solved for free.

|_environment
    |_application-1
    |_application-2
    |_application-3

In every directory we put a YAML or JSON file that contains the configuration
data only valid for this setup level. If we access this data all YAML files
get merged through the way down to the requested property. If we access a
property for application-1 our request looks something like this:

/environment/application-1/property

Now the YAML file for environment is loaded and merged with the YAML file in
the application-1 folder.
With this construction, we are able to configure a proxy server for the whole
setup in environment and refine this setting for our special application
in one of the application directorys. So the exceptionalproxy.setup.com
is only used if we render a configuration file for this particular application.
Every other configuration file is rendered with the default proxy.
On the other hand we get a mechanism for changing a setup wide property only
at one place. So if we move our proxy server to another DNS name, we need to touch
only the environment configuration. The redundancy of configuration properties
can decrease with such a system.

I try to build this in a RESTful webservice under github.com/mariussturm/pita
You can also look at ThoughtWorks Escape

05 February 2012