NLP Person
Under a very specific but nevertheless very common reading of the practice of “tokenization”, tokenizers are things that turn a string into sequences of atomic identifiers, i.e., integers. These identifiers are then sent to a downstream model, which uses them in some way. 1 Modern tokenizers are subword tokenizers . As the name implies, these tokenizers can segment text into pieces which are…
skeletoken has a new version! For those not in the know: skeletoken is a set of Pydantic datamodels that fully describe the Hugging Face tokenizer.json format, so you can validate, edit, and transform tokenizers as typed objects instead of editing JSON files. So: version 0.4.0 is out, and it’s a pretty big jump from 0.3.3, which is why I decided to write a blog post about it. Highlights…
The English Writer and Christian apologist G. K. Chesterton is, perhaps, most well known to programmers through a paragraph in which he introduces what is now known as “Chesterton’s fence”. It’s a very simple idea: You walk through a field and see a fence which, seemingly, has no purpose. Instead of tearing it down because it seemingly has no use, try to understand or ask why somebody put it…
I recently listened to the album Music for Existing by producer Martyn . I wasn’t really familiar with him, and in fact it got algorithmically recommended by me because the album features Duval Timothy , whose album Meeting with a Judas Tree I adore. The track Musa at Erbil , which features the voice and words of Musa Okwonga features a beautiful reflection on modern life, which really resonated…
If you’ve ever used code from scikit-learn , you will have seen the following pattern: import numpy as np from sklearn.preprocessing import StandardScaler X = np . random . randn (( 100 , 32 )) scaler = StandardScaler () scaler . fit ( X ) X_transformed = scaler . transform ( X ) # Or equivalently X_transformed = scaler . fit_transform ( X ) For all scikit-learn transformers ( 1 ), the fit call…
The group of researchers associated with the Massive Text Embedding Benchmark (MTEB) has released a new benchmark: the Retrieval Text Embedding Benchmark . As you may know, MTEB ranks models on their ability to perform well at a variety of tasks in a zero-shot setting, and is meant to reflect how well your model transfers to new tasks. Ranking high on MTEB can make or break your model, so it has…
Without reducing dimensionality, static models can be hundreds of MB large. Choosing the right dimensionality-reduction technique can shrink them without sacrificing retrieval quality. I was always a huge fan of Principal Component Analysis (PCA) for making static models smaller. For example, PCA is used in model2vec and was used in an older version of tokenlearn to post-process models, and is…
Late interaction is an interesting paradigm for computing the similarity between two documents, and can be seen as a hybrid of sparse and dense retrieval. In this post, I will show how static models in a late interaction setting actually reduce to sparse models. I will also argue that, in absence of empirical evidence to the contrary, there’s no good reason to assume that static late interaction…
In a previous post , I showed that making a tokenizer greedy, that is, always picking the longest matching subword like WordPiece does, can improve results without retraining. But WordPiece can unfortunately silently break your tokenization. Consider this example: from tokenizers import Tokenizer tokenizer = Tokenizer . from_pretrained ( "bert-base-uncased" ) result = tokenizer . encode ( "talk" *…
In a previous note , I discussed an alternative for setting split to true in a ByteLevel pretokenizer. I suggested using a ByteLevel normalizer first, and then splitting using a complicated regex in “byte space”. However, this turned out to not work very well: there are certain character classes in an original Regex, such as \s , that are very difficult to convert to a pattern in byte space. I was…
This note is wrong! This was revealed to me by Sasuke___420 . As it turns out, the regex does not work the same as the original one, specifically for non-ascii spaces. Upon further reflection, I don’t think you should really use this. This is a short note to dissuade you from using a ByteLevel pretokenizer in your tokenizers. The ByteLevel pretokenizer, as implemented in Hugging Face tokenizers…