The Shape of Duplicate Detection

Duplicate detection looks like a solved problem: use a hash set. A benchmark suite of 4050 measurements across finite batches and streaming workloads shows the fastest strategy can be 148x faster than a hash set, or 90,000x slower, depending on what you are deduplicating and what guarantees you need.