Morgan McGuire's machine learning journey through blogs and code
This post was originally pubished on the fast.ai forums here The Challenge Way back in October 2020 the Papers With Code ML Reproducibility Challenge 2020 was launched and shared in the fast.ai forums . A few of us jumped at the chance to test our ML knowledge and push our skills. Fast forward 110 days since that initial post and we delivered our Reformer Reproducibility submission via OpenReview!…
This post has been updated to show how to use HuggingFace's normalizer s functions for your text pre-processing In the following post, I'll cover the following using the HuggingFace Datasets libray : Loading data, single or multiple files, csv, txt or dataframes, train/test splits Processing data with 11 text processing functions Tokenizing data for use with MobileBERT Saving processed data to…
I recently hosted a presentation with the authors of the AdaHessian optimizer paper , Zhewei Yao and Amir Gholami and, to the fastai community. AdaHessian is a promising new optimizer and one of the first second-order optimizers that is becoming practical to use. I've been working on a port of it to fastai which I hope to release shortly, I;ll post up here once I have it released. Until then, I…
tl;dr Fastai's Textdataloader is well optimised and appears to be faster than nlp Datasets in the context of setting up your dataloaders (pre-processing, tokenizing, sorting) for a dataset of 1.6M tweets. However nlp Datasets caching means that it will be faster when repeating the same setup. Speed I started playing around with HuggingFace's nlp Datasets library recently and was blown away by the…
tl;dr If there is one thing I would like you to take away from this article it is the ability to use the Bokeh library to dynamically visualise, select and extract text embeddings of interest directly from a plot into a list for further processing with pandas, numpy etc: In order for Machine Translation to be useful in the real world, we should should strive to train it on high quality translation…
tl;dr Looking through papers to track down the Irish-English parallel corpora they used was a real pain, so I built nlp.irish to document where to find them and how to process them easily What? The intention behind nlp.irish is to make NLP for folks new to working with Irish a little easier by documenting the datasets that are available out there, where to find them and how to load them to a…
I was lucky enough to volunteer and attend (virtual) ICLR 2020. It delivered a huge amount of learning for me and I was fortunate to join some really great discussions. Efficient Deep Learning was big focus of many of the papers and in this second ICLR2020 article* I will focus on techniques presented that either enable more efficient training and/or inference from the papers I managed to see.…
I was lucky enough to volunteer and attend (virtual) ICLR 2020. It delivered a huge amount of learning for me and I was fortunate to join some really great discussions. Efficient NLP was big focus of many of the papers and here I will focus on a few of the more well known transformer architectures proposed over the past year or so; Reformer, ELECTRA, Lite Transformer and ALBERT. Towards the end of…
This aims to be an end-to-end description with code of how to train a transformer language model using fastai (v2) and HuggingFace, enjoy! TL;DR Main interesting bits in this notebook: Provides full code to train a transformer (RoBERTa) using a Masked Language Model task Utilise's many of HuggingFace's tokenizer features within fastai Make predictions of masked tokens like this: Before we get…
All FastHugs code can be found in my FastHugs GitHub Things You Might Like ( ?) FastHugsTokenizer: A tokenizer wrapper than can be used with fastai-v2's tokenizer. FastHugsModel: A model wrapper over the HF models, more or less the same to the wrapper's from HF fastai-v1 articles mentioned below Padding: Padding settings for the padding token index and on whether the transformer prefers left or…