Better Greedy Tokenizers: Handling WordPiece's [UNK] Problem

In a previous post , I showed that making a tokenizer greedy, that is, always picking the longest matching subword like WordPiece does, can improve results without retraining. But WordPiece can unfortunately silently break your tokenization. Consider this example: from tokenizers import Tokenizer tokenizer = Tokenizer . from_pretrained ( "bert-base-uncased" ) result = tokenizer . encode ( "talk" *…