Begone '##'! From continuing subword markers to word-initial markers

Under a very specific but nevertheless very common reading of the practice of “tokenization”, tokenizers are things that turn a string into sequences of atomic identifiers, i.e., integers. These identifiers are then sent to a downstream model, which uses them in some way. 1 Modern tokenizers are subword tokenizers . As the name implies, these tokenizers can segment text into pieces which are…