Laura's personal website and blog
Authors . Laura Ruis , Maximilian Mozes , Juhan Bae , Siddhartha Rao Kamalakara , Dwarak Talupuru , Acyr Locatelli , Robert Kirk , Tim Rocktäschel , Edward Grefenstette , Max Bartolo TL;DR . LLMs strategy for mathematical reasoning looks unlike retrieval from the parametric knowledge formed during pretraining. Instead, the models learn to apply procedural knowledge extracted from documents…
Authors . Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, Edward Grefenstette. TL;DR . Understanding of pragmatics is an essential and ubiquitous part of human communication. We show large language models (LLMs) mostly don’t capture this aspect of language zero-shot, hindering their applicability in the real world. Our analysis indicates where the largest room for…
In this post I’ll attempt to shed some light on the conclusion that is drawn (in part 1 ) from the above image: We shouldn’t use interpolation/extrapolation in the way the terms are defined below when talking about generalization, because for high dimensional data deep learning models always have to extrapolate, regardless of the dimension of the underlying data manifold. The image is taken from…
In this final part of the series on structured prediction with linear-chain CRFs we will use our implementation from part two to train a model on real data. To learn such a model, we need a dataset with examples consisting of input sentences annotated with POS tags. We will choose the Universal Dependencies dataset ( Silveira et al., 2014 ). Then all the things we need to implement are: A…
In this part of the series of posts on structured prediction with conditional random fields (CRFs) we are going to implement all the ingredients that were discussed in part 1 . Recall that we discussed how to model the dependencies among labels in sequence prediction tasks with a linear-chain CRF. Now, we will put a CRF on top of a neural network feature extractor and use it for part-of-speech…
You’re looking at part one of a series of posts about structured prediction with conditional random fields . In this post, we’ll talk about linear-chain CRFs applied to part-of-speech (POS) tagging. In POS tagging, we label all words with a particular class, like verb or noun. This can be useful for things like word-sense disambiguation, dependency parsing, machine translation, and other NLP…