You need to serialize a thing. Maybe it's protobuf, because the team next door decided proto is what events look like. Maybe it's Avro, because Iceberg is involved. Maybe it's CBOR, because you need to do something with COSE. The exact format doesn't matter. The next ten minutes are roughly the same regardless.

You open Hackage. There are usually a couple of packages. Now you must audit the code:

Is it fast? The README doesn't say. The benchmarks live in a bench/ directory last touched in 2019, when GHC was on a different number. Inconclusive.

Does it pass the upstream conformance suite? The README doesn't mention a conformance suite. Every serious format has one. You find yourself wondering whether the library you're looking at actually behaves as expected.

Has anyone touched it lately? Define "lately." There's a commit from six months ago that says bump bounds, and before that one from two years ago that says wip. The author has, in the intervening period, started writing Rust full-time. Understandable, but inconvenient.

Will it pull half of Hackage in? Let's check the cabal file. bytestring, fine. text, fine. lens, because the author wanted view _Just somewhere. lens-aeson, naturally. Before too long, the transitive footprint means you'll spend 20 minutes just compiling the dependencies. You start to wonder if maybe you should just write this yourself.

Will the API match the rest of your stack? It will not. The same concepts end up with slightly different names in each library:

aeson says eitherDecodeStrict.

The CBOR library calls it deserialiseFromBytes.

The MsgPack library calls it unpack.

And so on.

You end up doing this song and dance semi-regularly. After enough times you notice that determining whether to trust libraries from random authors takes as long as writing the library yourself.

You repeatedly end up writing the same custom code over and over:

Benchmarks to see if the library is actually fast enough, property tests to see if it actually behaves as expected, and so on. Template Haskell deriver boilerplate or Generics, an orphan instance to bridge to/from JSON for the format that didn't think it would need one.

Surely someone has already solved this, you think. Rust has serde, for example.

Enter wireform.

Wireform provides an ecosystem of roughly thirty format, networking, and infrastructure packages where every one shares the extremely performant core utilities (wireform-core), the same annotation-driven Template Haskell deriver (wireform-derive) where relevant, aggressively complete test suites, and, where an upstream conformance suite exists, an opt-in test runner that wires it up.

For example:

• Protobuf runs against the official protocolbuffers/protobuf harness.
• TOML runs against toml-test.
• YAML runs against yaml-test-suite.
• Iceberg, Delta Lake, Hudi, and Lance round-trip through their respective Python or Rust readers.
• Fory tests against pyfory.
• Kafka clients test against a live broker.

Our goal, with this project, is to have any package published on Hackage under the wireform monicker to be a trustworthy promise that the library is ergonomic, performant, correct, and avoids incurring more dependencies than necessary.

Every format draws on the same handful of heavily optimized primitives. SIMD-accelerated parsing, zero-copy encoding/decoding where possible, and specialized C kernels for the hottest paths. All generated code in the repo is held to the standard that it is has to be as fast as, or faster than, hand-written codecs, and within spitting distance of Rust/C/Zig, if not faster.

wireform as a project is unapologetically maximalist. If you need to parse, render, encode, decode, frame, or otherwise shuffles bytes between two systems, we want to support it.

The current package set is a starting point for the project, but new format packages are welcome and actively wanted, provided they clear the same bar the existing ones aspire to:

• fast enough to rival C/Rust/Zig, minimal garbage collection overhead, tested hard enough to prove it fully conforms with the format's official conformance suite (or, where no such suite exists, with an explicit interop test against another language's implementation)
• wired into the shared annotation deriver so users don't learn a new API per format
• and dependency-light enough to not raise eyebrows. Acceptance criteria in full are under Adding a new format.

Packages

The repo is a workspace of wireform-* packages. Each is self-contained and depends only on wireform-core, wireform-derive, and whatever third-party libraries the format genuinely needs. If you only need CBOR, you only build CBOR.

Infrastructure

Serialization formats

Text and config formats

Analytics and table formats

Networking

Workflow, testing, and simulation

Internal tooling

Module conventions

Every per-format package follows the same layout:

<Format>.Encode / .Decode      -- wire codec primitives
<Format>.Class                 -- typeclass (ToCBOR, FromThrift, etc.)
<Format>.Derive                -- TH deriver consuming Modifier annotations
<Format>.Value                 -- dynamic value ADT (where applicable)
<Format>.JSON                  -- JSON bridge (where applicable)

Formats with an IDL also have <Format>.Parser and <Format>.CodeGen. See each package's README for specifics.

The deriver

One annotated Haskell record drives instance generation for every format. Annotations live in Wireform.Derive.Modifier:

data Person = Person
  { personFullName :: !Text
  , personAge      :: !Word32
  , personBalance  :: !Int64
  , personSecret   :: !Text
  } deriving stock (Show, Eq, Generic)
{-# ANN type Person ("Person" :: String) #-}
{-# ANN personFullName (tag 1) #-}
{-# ANN personAge      (tag 2) #-}
{-# ANN personBalance  (tag 3) #-}
{-# ANN personSecret   (tag 4) #-}
{-# ANN personFullName (renameStyle SnakeCase) #-}
{-# ANN personAge      (renameStyle SnakeCase) #-}
{-# ANN personBalance  (renameStyle SnakeCase) #-}
{-# ANN personSecret   (renameStyle SnakeCase) #-}
{-# ANN personFullName (forBackend backendJSON (rename "fullName")) #-}
{-# ANN personSecret   (forBackend backendJSON skip) #-}
DProto.deriveProto    ''Person
DCBOR.deriveCBOR      ''Person
DMP.deriveMsgPack     ''Person
DAeson.deriveJSON     ''Person

personFullName becomes full_name on every binary wire but fullName in JSON. personSecret is omitted from JSON entirely.

The annotation vocabulary:

• rename, renameStyle SnakeCase, renameIdiomatic for wire keys
• tag N for field numbers (proto, Bond, Thrift, Iceberg)
• skip, defaults, required, optional, flatten, wireOverride WireZigZag
• forBackend backendJSON (rename "x") for per-format overrides
• extension XmlFieldOpt for typed per-backend configuration

Formats that need knobs beyond the core vocabulary use BackendModifier extensions (XmlFieldOpt, HtmlFieldOpt, Asn1Tag). See wireform-derive for the full API.

Building

cabal update
cabal build all

A Nix flake is provided:

nix develop          # default (GHC 9.8)
nix develop .#ghc96  # GHC 9.6
nix develop .#ghc910 # GHC 9.10

LLVM is off by default (-fasm in cabal.project). Production builds benefit from -fllvm (up to 27% faster on tight loops).

Testing

cabal test all

Format packages have opt-in interop tests against upstream conformance suites (silent skip when the runner isn't installed):

• wireform-proto: protobuf conformance suite (2675/2675)
• wireform-toml: toml-test
• wireform-yaml: yaml-test-suite
• wireform-iceberg: pyiceberg + fastavro
• wireform-delta: delta-rs
• wireform-hudi: hudi-rs
• wireform-lance: pylance
• wireform-fory: pyfory
• wireform-kafka: live broker via WIREFORM_KAFKA_BROKER=host:port

Examples

Run from the workspace root with cabal run <name>:

See examples/ for the full list.

License

BSD-3-Clause. See LICENSE.