Tea, statistics and t-statistics: a data science blog by David Selby.
Why are entrance kerbs in Germany so absurdly high? An entrance kerb or dropped kerb ( abgesenkter Bordstein ) is meant to be flush, so that people can smoothly cross the road. However, badly designed or poorly installed kerbs have an upstand, a raised lip, that inevitably trips pedestrians and violently jolts bikes, prams, wheelchairs and trolleys, strewing your shopping and Pfandflaschen across…
At the railway station, a lost-looking US soldier asked me if I spoke English. Do I? At times it feels like it, but the Germans keep me guessing. Since moving to Germany, I have been continually tested on the true meanings of English words. Here are a few examples. Denglisch It turns out I’d been using these words wrong all along. Denglisch German meaning to Anglophones der Body a babygrow…
Overleaf , formerly known as Share$\LaTeX$, is the go-to collaborative document editor for many researchers, who have taken advantage of its free tier. It’s a web-based editor that compiles $\LaTeX$ documents in real time, with several word-processor-like features like commenting, tracked changes and a GUI. However, the company has made collaboration a paid feature, meaning you either need…
There is a widespread critique that too many pundits fail to make measurable predictions. For example, Philip Tetlock takes aim at what he calls vague verbiage , the use of vaguely probabilistic phrases such as “it’s a real possibility” or “there’s a fair chance”. We’d like to think that with respect to the World Cup we escaped the crowd of…
Sports prediction has exploded in the last couple of decades with entire journals , conferences and books devoted to it. Much of this focuses on utilising ever-greater amounts of data, with soccer, for example, now providing sub-second ball and player tracking data. But sometimes it is nice to try to do more with less. Here we ( Ian Hamilton , Stefan Stein and David Selby ) describe a method that…
Everybody knows that R is an inferior programming language, because vector indices start from 1, whereas in real programming languages like C and Python, array indexing begins from 0 . Sometimes this can be quite annoying if a problem—be it a mathematical algorithm or a coding challenge —calls for zero-based indexing. You find yourself having to add + 1 to all your indices and…
It’s that time of year again. And not just for Secret Santa —it’s time for the Advent of Code , a series of programming puzzles in the lead-up to Christmas. I’m doing the 2021 challenge in R—in the form of an open-source R package , to demonstrate a test-driven workflow. Each puzzle description typically comes with a few simple examples of inputs and outputs. We can use these to define…
R 4.1.0 has been released and has a couple of handy new features . One is a shorthand syntax for defining functions. Now, instead of writing, for example triple <- function(x) x * 3 you can use the more concise syntax triple <- \(x) x * 3 which may come in handy when using anonymous functions in apply -type calls. More interesting, though, is the addition of a native pipe operator to R, given by…
A common task in data analysis is to merge or join two tables according to shared keys or values. The operation is perhaps most commonly associated with relational databases and structured query language ( SQL ) but it’s just as useful in R with data frames. Most joins are equi-joins , matching rows according to two columns having exactly equal values. These are easy to perfom in R using the base…
Advent of Code is a series of programming puzzles you can tackle to hone your coding skills each day in the run-up to Christmas. This year I am attempting it using R, which can make some challenges easier or harder depending on whether they are more ‘computer sciencey’ or more ‘data sciencey’. Generally it makes parsing datasets easier but low-level string manipulation more fiddly. Here are my…
My brother wrote to me this week with an interesting task: I’ve got a list of films I’ve noted down whenever I’ve seen a good review. But it’s hundreds long and I can never remember what a film is from its title and can’t be bothered to look them up each time. Any chance you can knock up a script that cross-references a Rotten Tomatoes score and quick synopsis so I…
How many different ways are there to create an identity matrix in R? This was an interesting little challenge set by Guillaume Nicoulaud on Twitter. In code golf , programmers try to write an algorithm using the shortest programme possible, often exploiting lesser-known eccentricies of each programming language. This R challenge is less about minimising your golf handicap, however, and more about…
Buying milk in France is not like at home. In the supermarket, it is always found in the farthest possible point from the shop entrance, but the similarities end there. The colour scheme used to be the same as in Old Blighty but Napoléon changed it in the early 1800s, around the same time he switched everybody to driving on the wrong side of the road. Blue milk bottles are semi-skimmed, red is…
I have just moved to Paris ahead of a six-month internship. It is my first time even visiting the city, so here are some initial impressions. As many a Français(e) will keenly remind you: ‘Paris is not France’, so I also hope to venture outside la Périphérique soon. La circulation In Paris, traffic signals are suggestions rather than orders. A red man at a pedestrian crossing means…
I have just published an interactive graphic showing the effect of ranking scientific communities with pairwise comparison models. The visualisation is an interactive version of my (award-winning) useR!2017 poster, Ranking influential communities in networks . You can see how academic journals have been grouped into communities based on their citation behaviour, and notice the relative ranking…
Neural networks can seem like a bit of a black box. But in some ways, a neural network is little more than several logistic regression models chained together. In this post I will show you how to derive a neural network from scratch with just a few lines in R. If you don’t like mathematics, feel free to skip to the code chunks towards the end. This blog post is partly inspired by Denny Britz’s…
I just discovered a way to get R running on my smartphone, with full support for packages, graphics and R Markdown , and no need to connect to an external server. This is really handy for quickly checking R code, trying out ideas and writing blog posts on the go. It works quite well! Here I will show you how to do the same on your Android device. This post was inspired by this answer on…
Next week I will be presenting a poster at the useR!2017 conference in Brussels. My topic is Ranking influential communities in networks . Using a large dataset of citations from the Web of Science, we grouped academic journals into communities based on their citation behaviour. These communities closely correspond to recognisable research fields, so I was able to label them. We then modelled the…
I am pleased to introduce my first R package for general consumption, ggChernoff . Inspiration stems from Leland Wilkinson’s book, The Grammar of Graphics , whose principles were later implemented as the layered grammar of graphics in Hadley Wickham’s popular R package, ggplot2 . Wilkinson’s grammar generalises data visualisation. Rather than choosing a types of graph from a list—for example bar…
Yihui Xie’s new blogdown package is a brilliantly elegant and simple tool for creating R-themed blogs and web sites. Starting your own blog is as simple as one line of R code: blogdown::new_site() By default, blogdown uses Hugo , a static web site generator written in Google’s Go programming language. This has a number of advantages over Jekyll (the site generator that powers GitHub…
Last summer I took part in an internship with the fledgling Alan Turing Institute at the British Library in London. My group helped analyse data from Cloudy with a Chance of Pain , a citizen science project that uses smartphone app data to model the relationship between weather and joint pain. A robust statistical model relies on a steady stream of self-reported pain data from each participant.…
When knitting an R Markdown document to HTML output, R chunks can produce warnings, errors or messages. Normally these messages look like any other console output: Pretty ugly, and usually something I find myself trying to hide at the earliest opportunity. But if you’re using R Markdown’s default template, which uses Twitter Bootstrap , you can promote warnings, errors and messages to…
GitHub Pages' built-in Jekyll support makes it very easy for techie types to deploy static web sites. Simply push your plain text Markdown content to a repository and the server-side Jekyll engine will render it for the web. Markdown is good, but R Markdown is even better, assuming we ever want to write anything involving plots or data analysis. How can we write and edit blog posts in R Markdown…
Our office just exchanged presents for Secret Santa , a tradition where each person is randomly assigned someone else to give an anonyous gift. One of the challenges of Secret Santa is keeping the pairs of gift-givers and receivers both random and secret. How can you do this while also taking part yourself? Using R, of course! Firstly, recruit people! Write their names down, one per line, in a…
Here is my poster that I presented at the 2nd IMA Conference on the Mathematical Challenges of Big Data in London on 1st–2nd December 2016. It is entitled “PageRank and the Bradley–Terry model: Measuring influence with the Scroogefactor”.
I have just moved to the University of Warwick to start my PhD in Statistics. So far most of the people I have met would regard Coventry as a bit of a downgrade from Edinburgh, but the Warwick campus seems to make up for it. The Mathematics and Statistics building comes with resident geese.
Despite living in the capital city of Scotland, my home broadband speeds (especially upload speeds) are painfully slow. There is no fibre-optic offered on my road, while my friend two streets over enjoys super-fast internet. It turns out, through the power of data visualisation, that EH16 is a bit of a broadband speed “notspot” along with a few other slow neighbourhoods throughout…
Today I took part in the Lego Calculator Challenge, an event that was run as part of Edinburgh University’s Innovative Learning Week. The day included revising the finer points of adding in base 6 and learning about how people managed to do complicated calculations on mechanical computers before Alan Turing invented the Casio fx -83. With the aid of this therapeutic video , Lego expert Alex…
This evening saw the Netherlands face Spain for their first match in the group stage of the 2014 Fifa World Cup. 604 score predictions were made on Twitter in the hour before kick-off. When the final whistle blew, 100% of them had missed the mark. The Twittersphere overwhelmingly tipped a 2–1 victory to Spain, the reigning world champions. A bit of a shame, then, that the Netherlands…
As part of my first foray into data science, I decided to have a go at opinion mining on Twitter. It’s common knowledge that everything stops for tea, but how much does the Twittersphere agree? And what else are microbloggers saying about the drink? Using the free R statistical software it is very easy to start data mining on Twitter 1 . A short R script quickly retrieved 699 recent tweets…
The set of all t makes the perfect tea set for mathematicians and other numerate geeks. A natural accompaniment to your Student’s t-distribution teapot , the t-set lets you collect mugs in a mathematical way. Get yours now from Zazzle .
Where is a student’s tea distributed? Probably in a teapot. Thus the Student’s t-distribution teapot is born. Warmly emblazoned across the outside is the probability density function of Student’s t-distribution , in case you need to estimate the mean of a normally distributed population and pour a cup of tea at the same time. Lovingly designed for students and academics who take…
Contours magazine is a student publication produced by the University of Edinburgh School of Mathematics, featuring interviews with maths lecturers. I was editor and graphic designer for the 2013–14 edition. Click here to read it on Issuu , or pick up a hard copy from the King’s Buildings campus. What does a mathematician do when they aren’t lecturing students? What kind of…
Hello, my name is David Selby and I am a researcher at the German Research Center for Artificial Intelligence (DFKI) and University of Kaiserslautern. I previously worked as research associate in data science at the Centre for Epidemiology Versus Arthritis, University of Manchester, and completed my PhD in statistics at the University of Warwick under the supervision of Prof David Firth.…