Reading · perezdecastro.org

This page cannot be shown here. You can still read it on the original site — the toolbar below keeps your place in the directory.

Sometimes one has to parse programmatically some file containing key=value pairs. In the world of systems administration this means configuration files most of the time. Also, one thing I like is functional programming , but in the real world one ends up making almost all of the code in imperative style. Python allows some functional constructs, and sometimes I like to use them to make code…

Sometimes one has to parse programmatically some file containing key=value</code> pairs. In the world of systems administration this means configuration files most of the time. Also, one thing I like is functional programming</a>, but in the real world one ends up making almost all of the code in imperative</a> style. Python allows some functional constructs, and sometimes I like to use them to make code most concise, because it express better what the code is trying to do or just because I wanted to melt my mind doing some functional tricks.</p>

First, let me introduce the code:</p>

from</span> itertools import</span> imap, ifilter
config_items = lambda</span> iterable:
    imap(lambda</span> (k, v): (k.strip(), v.strip()),
        imap(lambda</span> s: s.split("="</span>, 1</span>),
            ifilter(lambda</span> s: s and</span> not</span> s.startswith("#"</span>),
                imap(lambda</span> s: s.strip(), iterable))))
</code></pre>
Neat, huh? As promised in the title, this is functional. And yes, I am
aware of ConfigParser</a>, but I do not need its full power, and also I
have found some problems with files containing Unicode</a> strings.</p>
I think this is one of the most beautiful snippets of code I have ever
written in Python: it makes just one thing well, and it is terse and
concise. Moreover, it is quite easy to explain.</p>
How does it work</h2>
I have just written that it is easy to explain how this works. Okay, I
will dissect this beast one line at a time, starting at the innermost.
But first, a quick introduction to imap()</code> and ifilter</code>:</p>

imap()</code></strong>: Works like map()</code>, which returns a list whose
contents are the results of applying a function (first argument) to
each of the elements of another list (the second argument). The
difference is that imap()</code> uses generators</a> instead.</li>
ifilter()</code></strong>: This one works like filter()</code> and will also return
a list, whose contents are the items of another list (second
argument) for which the result of calling the given function (first
argument) is True</code>. This one also works with generators.</li>
</ul>
Now, let us start with the first one:</p>
imap(lambda</span> s: s.strip(), iterable)
</code></pre>
This picks each line, and removes whitespace sitting at the left and and
the right of the string.</p>
ifilter(lambda</span> s: s and</span> not</span> s.startswith("#"</span>),
</code></pre>
We want to keep interesting</em> lines: empty lines and comment-lines
starting with a hash mark (#</code>) must be thrashed away. We check for
lines which both</em> are not empty and that do not start with a hash-mark.</p>
imap(lambda</span> s: s.split("="</span>, 1</span>),
</code></pre>
That one picks each string and splits it at the first =</code> character,
thus separating the key from the value. This is what converts each
string into a (key, value)</code> tuple.</p>
imap(lambda</span> (k, v): (k.strip(), v.strip()),
</code></pre>
This is the last remaining detail: Removes extra leading and trailing
whitespace from the keys and values of the generated tuples. This is
needed for removing the spaces around the =</code> character.</p>
How to use it</h2>
Fire in the interpreter, type in (or copy-and-paste) the above code and
guess by yourself:</p>
>>> text = """a = 1
... b = this is b"""
>>> tuple(config_items(text.splitlines()))
(('a', '1'), ('b', 'this is b'))
>>> dict(config_items(text.splitlines()))
{'a': '1', 'b': 'this is b'}
>>>
</code></pre>
So you pass it a list</del> an iterable which yields lines, and it will
return another iterable, which yields (key, value)</code> tuples. Thanks to
how dict()</code> is defined, we can directly pass the result to it and get a
dictionary.</p>
But it would be useful as well to use it on files, so here we go:</p>
>>> file("test.conf", "w").write(text)
>>> dict(config_items(file("test.conf")))
{'a': '1', 'b': 'this is B'}
>>>
</code></pre>
For your convenience, you may want to define a helper function if it
makes you feel more comfortable:</p>
>>> def config_file_items(path):
...    with file(path, "rU") as f:
...        return config_items(f)
...
>>> dict(config_file_items("test.conf"))
{'a': '1', 'b': 'this is B'}
>>>
</code></pre>
Extra niceties</h2>
I have already mentioned that this code uses generators</a> in its
entirety. What is passed from one function to another in the chain of
imap()</code> and ifilter()</code> calls are always</em> generators. This means that
if config_items()</code> is used to read a big file (e.g. some hundreds of
megabytes) only one line is in memory at a given time</em>. This is why I
did not use map()</code> and filter()</code> but their “incremental” counterparts
from the itertools</code> module. So the bottom line is that this may not be
the most efficient implementation out there, but it is good and is
capable of working over arbitrarily long sequences of data while the
function remains small and understandable.</p>
Error Handling</h2>
Whenever the input is not well formed, then this function will raise
ValueError</code>, or when a =</code> character is not found in some line. This
means that you can do something like this:</p>
import</span> sys
try</span>:
    items = dict</span>(config_items(sys.stdin))
except</span> ValueError:
    raise</span> SystemExit("Malformed 'key=value' input in standard input"</span>)
</code></pre>
Of course more elaborate error checking could be done i.e. to be able of
showing to the user the exact offending line number, but the goal is to
keep things as simple as possible. Also the syntax of those simple
configuration files is so simple that it should be fairy simple for the
user to spot typos.</p>
Final words and advice</h2>
My advice is that if you have the possibility, make your Python code in
such a way that it uses generators, unless you are sure that it will
always</em> handle reasonably small amounts of data.</p>
I hope that things are explained well enough, and (who knows!) maybe
this can help someone to better understand why generators are a good
idea. I will also be happy if you came here looking for some code to
parse simple configuration files and this did the trick for you.</p>

Python recipe: Functional config file parsing

Comments