Compact transformers can't learn all sequences.

TL;DR This paper studies a subset of possible transformer models called “compact transformers” (CT). Those are transformers that have compact input embeddings and compact positional encodings in every layer. The authors prove that such transformers cannot learn to predict every possible sequence with confidence – instead the sequence space is separated into equivalence classes and a CT can only…