Reading · menno.io · RSS Amplifier

This page cannot be shown here. You can still read it on the original site — the toolbar below keeps your place in the directory.

A relatively unknown part of the Python standard library that I find myself using fairly regularly at work these days is the groupby function in the itertools module. In a nutshell, groupby takes an iterator and breaks it up into sub-iterators based on changes in the "key" of the main iterator. This is of course done without reading the entire source iterator into memory. The "key" is almost…

A relatively unknown part of the Python standard library that I find myself using fairly regularly at work these days is the groupby</a> function in the itertools</a> module. In a nutshell, groupby takes an iterator and breaks it up into sub-iterators based on changes in the "key" of the main iterator. This is of course done without reading the entire source iterator into memory.</p>

The "key" is almost always based on some part of the items returned by the iterator. It is defined by a "key function", much like the sorted</a> builtin function. groupby probably works best when the data is grouped by the key but this isn't strictly necessary. It depends on the use case.</p>

I've successfully used groupby for splitting up the results of large database queries or the contents of large data files. The resulting code ends up being clean and small.</p>

Here's an example:</p>

from</span> itertools</span> import</span> groupby</span></span>
from</span> operator</span> import</span> itemgetter</span></span>
</span>
things</span> =</span> [(</span>'2009-09-02'</span>,</span> 11</span>),</span></span>
          (</span>'2009-09-02'</span>,</span> 3</span>),</span></span>
          (</span>'2009-09-03'</span>,</span> 10</span>),</span></span>
          (</span>'2009-09-03'</span>,</span> 4</span>),</span></span>
          (</span>'2009-09-03'</span>,</span> 22</span>),</span></span>
          (</span>'2009-09-06'</span>,</span> 33</span>)]</span></span>
</span>
for</span> key, items</span> in</span> groupby(things, itemgetter(</span>0</span>)):</span></span>
    print</span> key</span></span>
    for</span> subitem</span> in</span> items:</span></span>
        print</span> subitem</span></span>
    print</span> '-'</span> *</span> 20</span></span>
</span></code></pre>
Here the dummy data in the "things" list is grouped by the first item of
each element (that is, the key is the first element). For each key, the
key is printed followed by the items returned by each sub-iterator.</p>
The output looks like:</p>
2009-09-02</span></span>
('2009-09-02', 11)</span></span>
('2009-09-02', 3)</span></span>
--------------------</span></span>
2009-09-03</span></span>
('2009-09-03', 10)</span></span>
('2009-09-03', 4)</span></span>
('2009-09-03', 22)</span></span>
--------------------</span></span>
2009-09-06</span></span>
('2009-09-06', 33)</span></span>
-------------------</span></span></code></pre>
The "things" list is a contrived example. In a real world situation this
could be a database cursor object or a CSV reader object. Any iterable
object can be used.</p>
Here's a closer look at what groupby is doing using the Python
interactive shell:</p>
>>> iterator = groupby(things, itemgetter(0))</span></span>
>>> iterator</span></span>
<itertools.groupby object at 0x95d3acc></span></span>
>>> iterator.next()</span></span>
('2009-09-02', <itertools._grouper object at 0x95e0d0c>)</span></span>
>>> iterator.next()</span></span>
('2009-09-03', <itertools._grouper object at 0x95e0aec>)</span></span></code></pre>
You can see how a key and sub-iterator are returned for each pass
through the groupby iterator.</p>
groupby is a handy tool to have under your belt. Think of it whenever
you need to split up a dataset by some criteria.</p>

itertools.groupby

Comments