Criterion: robust, reliable performance measurement

criterion is a library that makes accurate microbenchmarking in Haskell easy.

Features

• The simple API hides a lot of automation and details that you shouldn't need to worry about.

• Sophisticated, high-resolution analysis which can accurately measure operations that run in as little as a few hundred picoseconds.

• Output to active HTML (with JavaScript charts), CSV, and JSON. Write your own report templates to customize exactly how your results are presented.

• Linear regression model that allows measuring the effects of garbage collection and other factors.

• Measurements are cross-validated to ensure that sources of significant noise (usually other activity on the system) can be identified.

To get started, read the tutorial below, and take a look at the programs in the examples directory.

Credits and contacts

This library is written by Bryan O'Sullivan (bos@serpentine.com) and maintained by Ryan Scott (ryan.gl.scott@gmail.com). Please report bugs via the GitHub issue tracker.

Tutorial

Getting started

Here's Fibber.hs: a simple and complete benchmark, measuring the performance of the ever-ridiculous fib function.

{- cabal:
build-depends: base, criterion
-}
import Criterion.Main
-- The function we're benchmarking.
fib :: Int -> Int
fib m | m < 0     = error "negative!"
      | otherwise = go m
  where
    go 0 = 0
    go 1 = 1
    go n = go (n - 1) + go (n - 2)
-- Our benchmark harness.
main = defaultMain [
  bgroup "fib" [ bench "1"  $ whnf fib 1
               , bench "5"  $ whnf fib 5
               , bench "9"  $ whnf fib 9
               , bench "11" $ whnf fib 11
               ]
  ]

(examples/Fibber.hs)

The defaultMain function takes a list of Benchmark values, each of which describes a function to benchmark. (We'll come back to bench and whnf shortly, don't worry.)

To maximise our convenience, defaultMain will parse command line arguments and then run any benchmarks we ask. Let's run our benchmark program (it might take some time if you never used Criterion before, since the library has to be downloaded and compiled).

$ cabal run Fibber.hs
benchmarking fib/1
time                 13.77 ns   (13.49 ns .. 14.07 ns)
                     0.998 R²   (0.997 R² .. 1.000 R²)
mean                 13.56 ns   (13.49 ns .. 13.70 ns)
std dev              305.1 ps   (64.14 ps .. 532.5 ps)
variance introduced by outliers: 36% (moderately inflated)
benchmarking fib/5
time                 173.9 ns   (172.8 ns .. 175.6 ns)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 173.8 ns   (173.1 ns .. 175.4 ns)
std dev              3.149 ns   (1.842 ns .. 5.954 ns)
variance introduced by outliers: 23% (moderately inflated)
benchmarking fib/9
time                 1.219 μs   (1.214 μs .. 1.228 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.219 μs   (1.216 μs .. 1.223 μs)
std dev              12.43 ns   (9.907 ns .. 17.29 ns)
benchmarking fib/11
time                 3.253 μs   (3.246 μs .. 3.260 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 3.248 μs   (3.243 μs .. 3.254 μs)
std dev              18.94 ns   (16.57 ns .. 21.95 ns)

Even better, the --output option directs our program to write a report to the file fibber.html.

$ cabal run Fibber.hs -- --output fibber.html
...similar output as before...

Click on the image to see a complete report. If you mouse over the data points in the charts, you'll see that they are live, giving additional information about what's being displayed.

Understanding charts

A report begins with a summary of all the numbers measured. Underneath is a breakdown of every benchmark, each with two charts and some explanation.

The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.

The chart on the right contains the raw measurements from which the kernel density estimate was built. The