An optimized wc implementation in Zig, 5-20x faster than GNU and BSD wc. zwc achieves this by using a finite-state machine to process UTF-8 text branchlessly, and by implementing data parallelism to distribute the work across multiple CPU cores.

This is not intended as a complete replacement for the GNU or BSD wc programs. It's missing some of the features, like the --max-line-length flag. Additionally, it only supports ASCII and UTF-8 encodings.

This program is based heavily on Robert Graham's wc2 program.

Here is what the core loop looks like:

const table: [State.STATE_MAX][256]u8 = gen_table();
var counts = [_]usize{0} ** State.STATE_MAX;
var state: usize = State.WASSPACE; // initial state
while (true) {
    const b = file.takeByte() catch break;
    state = table[state][b];
    counts[state] += 1;
}

Development

For the simple version of the state machine, see simple.zig.

For the UTF-8 version of the state machine, see dfa.zig.

For the parallel version, see parallel.zig.

Building

This project currently only compiles with the 0.15.2 version of Zig.

# Build the binary into ./zig-out/bin/zwc
zig build -Doptimize=ReleaseFast
# Alternatively, you can use `just`:
just build-exe ~/.local/bin/zwc # build into local path

Benchmarks

Here are some benchmarks from the english and chinese wikipedia title dumps, which are 165MB text files with lots of words and lines. Programs (in order): zwc, BSD wc, GNU wc, Rust uutils/coreutils wc.

Same-size file with mostly chinese characters. The -lwm flags tell the programs to count lines, words, and multibyte characters.

See justfile bench_all and bench_all_multibyte.