Recent content in Welcome on Ellis Valentiner
It’s early and the coffee machine is still broken, but thankfully my wife ordered us coffee and pastries to kick-start the day. I’m tackling updates to our data warehouse this morning. We take a selective approach to BigQuery replication, choosing specific tables rather than mirroring our entire database. Today’s focus is on tables needed for an internal report. Our data…
It’s 2025 and Large Language Models (LLMs) are still all the rage. They’ve been shown to be useful at many tasks, notably classification information extraction, although most people seem interested in their generative applications. When it comes to LLMs my professional interest is primarily in their feature embeddings. Embeddings are a vector representation produced by the LLM. These…
Developers like to use JSON and when they store data, often don’t normalize it. Thus, it’s not unusual to encounter JSON(b) fields in the database. For data analysts, data scientists, and machine learning engineers unnesting, or flattening, the data is often a prerequisite to subsequent analysis. Wrangling JSON columns in the database can be challenging. Often the goal is to expand JSON arrays…
Hardware and OS I’ve continued to use macOS as my operating system of choice. After being issued a ThinkPad P53 through work, I briefly used Ubuntu. Unfortunately that machine would kernel panic after a day or so of use. Given that I’m already embedded in the Apple ecosystem, using a Mac is convenient and makes a lot of sense. In November, I migrated to a new 16" MacBook Pro with the…
I usually shy away from JavaScript unless I’m writing front-end code – I typically write my RESTful APIs in Python using Flask. I’ve written APIs in other languages too – R using Plumber, Julia using Genie.jl, and C# using RestSharp – but this was my first go at writing one in JavaScript so I thought I’d summarize my thoughts about it. The real motivator to…
Sometimes you want to match a value to a range – for instance suppose you want to map an age to one of these age categories: 0-14, 15-24, 24-54, 55-64, 65+. If you’re working in the database you might write this using a CASE statement like this: SELECT CASE WHEN age BETWEEN 0 AND 14 THEN 0 WHEN age BETWEEN 15 AND 24 THEN 1 WHEN age BETWEEN 25 AND 54 THEN 2 WHEN age BETWEEN 55 AND 64…
Sometimes people ask me for advice about how to start a career in data science. I’ve written before about how I became a data scientist , which gives a bit of my personal journey. I’ve told my story because people often can only see where someone is now and not always where they’ve been or the path taken to get there. Recently an aspiring data scientist reached out to me with…
Back in 2016 I wrote about how numpy.float64 is JSON serializable but numpy.float32 is not . Since then I’ve learned a much better way to seralize numpy.float32 (and other types). In my other post I noted that you can convert numpy.float32 back to numpy.float64 (which is JSON serializable) simply by multiplying it. But who wants to do this every time ? What if your array sometimes contains…
A former coworker recently learned I was a psychology major in college and asked how I became a data scientist. Here’s a cleaned up version of what I sent him: I started college in 2007 without having declared a major although I thought I might study film. There wasn’t an official film major but there were a few film courses and the university had a process for “designing your own…
This was my 3rd year running the Big House 5K, an annual run in Ann Arbor, Michigan. Less than a week later I my first 10K in Richmond, Virginia. One thing about running is that it gives you lots of time to think and one thing that runners tend to think during a race about is their time/pace. It is easy to look up race results. The results include your time (and pace) and standing overall and by…
This is my second post in the speed test report series ( read the January 2018 report here ). This month I conducted 1,298 speed tests. Of these, 698 tests were against the nearest site in Chicago. The map below is a close-up of North America. Each labelled city hosts a M-Lab site and is colorized by median download speed. Download speeds are faster from sites in the United States than in Canada,…
A few months ago I setup a Raspberrypi to run speed tests at regular intervals. The speed tests run hourly, first to the nearest server and then to a random server. Through out the month of January 2018 this resulted in 1,431 speed tests. Each record consists of the timestamp when the speed test ended, information about the server (city, IP address, fully qualified domain name, site), and upload…
Raspberry Pis are great hobbyist computers. They’re fairly cheap and provide a friendly introduction to Linux. People use Pis for teaching computer science, developing robotics, as routers, cameras, media servers, and many other applications. In this tutorial I’m going to cover how to setup a Raspbery Pi when you don’t have access to a display. I’ll cover preparing your…
Sequential data such as text from a book, recorded speech, or financial time series is one of the most common types of structured data. Fitting models on sequence data typically involves constructing small, overlapping sequences by moving a window across the entire dataset. Each sequence is a small example of the larger picture. However because the data is continuous, it is often stored as a long…
In Postgres 8.1 and later, users and groups are combined as roles. So roles can own database objects (schemas, tables, functions, etc.). Roles can be granted on other roles, conferring their privileges. Suppose you want to share data with a collaborator and you’ve decided to use Postgres running in Amazon RDS. Good choice! You create a new role that can login and has a password: CREATE ROLE…
Many data scientists I know come from an academic background — they went to college and majored in statistics, math, or research heavy field. Some have masters in statistics or PhDs in psychology or economics. Most of them don’t work in software development but a lot of them care about reproducibility. Reproducibility refers to the ability to reproduce an analysis. It’s a big topic in…
This is the second post in my Two Sigma Connect: Rental Listing Inquiries Kaggle competition series. The first post covered data import/processing and this post contains some of my exploratory data analysis. Univariate Summaries The dataset contains 49,352 RentHop listings. I began by taking a look at univariate summaries of continuous variables. I excluded ID variables such as listing_id and…
Kaggle is a great source to find data. Last month (Feb. 2017) a competition was posted that caught my eye: Two Sigma Connect: Rental Listing Inquiries . The purpose of the competition was to predict the popularity of apartment rental listings. I’m not particularly familiar with either Two Sigma or RentHop but I saw the data included lat/lon coordinates and I enjoy working with spatial data. While…
Recently I was faced with the problem of reshaping a single long vector into an array of overlapping sequences. There are a lot of different ways to tackle this problem but I wanted a fast and elegant solution that could generalize to different window lengths and step sizes. For a concrete example, suppose we have a vector with digits 1 to 100: We want to cut it into overlapping sequences with…
Application programming interfaces (APIs) are a general term to describe methods for communicating between different software components, such as facilitating the user of a low-level library by a higher-level language. Web APIs provide methods for communicating with a web service by making requests through designated endpoints. Since web applications already make use of APIs, they’re a great…
Flat files are great, except they’re not. There is a RFC standard specification for CSVs, but pretty much no one follows it. So every CSV is it’s own unique snowflake and different applications will sometimes parse the files differently. In practice it’s common (and much better) to leverage databases to store and access data. There are several good packages for accessing databases from R.
Most data scientists first learn to import data from flat files, such as comma or tab delimited files. Afterall, we’re familiar with seeing our data organized with observations across rows and variables in columns. readr The best starting place for most flat files is the readr package by R Studio.
In part one I wrote a little about my motivation for switching to Hugo. At that point my (limited) experience with Hugo had been mostly positive and I didn’t think I needed to say much about the actual process of the migration. The full title of this post could be “Moving to Hugo pt. 2: Why I should never have done this or least nuked everything and started from scratch” .
For the past two years I’ve generated my static site using Jekyll . Jekyll is a great tool (despite being written in Ruby) and still a good option for anyone looking to create a static site. So why did I move to Hugo? Recently Yihui Xie released blogdown , a R package for creating blogs and static sites. Blogdown is built on Hugo and rmarkdown. It’s pretty new but a great way to render…
The battery in my iPhone 6s had been really terrible lately, shutting down sometimes when the battery shows more than 50% remaining. I decided to go to the Apple Store and as it turns out they have a whole program for iPhone 6s unexpected shutdown issues (yay). When I’ve had my iPhone battery replaced in the past, it usually takes the Genius in the backroom about 20 minutes. It’s about…
Recently I’ve been playing around with Keras for fitting neural networks. Whenever I’m learning a new tool I find it’s easier to start by applying it to something familiar. This time I decided to try using an unfamiliar dataset for learning about Keras. The dataset I chose was the Credit Card Fraud Detection dataset hosted on Kaggle, submitted by Andrea Dal Pozzolo , who…
I’ll be honest I didn’t get the Docker hype for a long time. I’m even sure I really get it now but let’s dive in. Package management in R is abysmal although getting better as more packages are made available on GitHub. Revolution Analytics created checkpoint and RStudio developed packrat in attempts to solve the problem but adoption has been relatively poor. Neither do a…
Earlier this week Bitbucket announced new features , including Bitbucket Pipelines. Pipelines is a continuous integration service similar to AppVeyor, Travis CI, CircleCI, and others. If your code depends on environment variables, for example to store database connection strings or API keys, these could be exposed during the build process. Interestingly Bitbucket Pipelines doesn’t show the…
Package management in R is terrible. Don’t get me wrong, R has a lot of other great features but when it comes to package management it is far behind other languages. In R, packages are installed in libraries. Libraries are just directories in the file system with subdirectories for each package installed there. Packages are installed with install.packages("pkg") and loaded with the…
The Fitbit Charge HR captures heart rate using LED lights on the tracker to detect blood volume changes. Measurements are taken every 5 seconds and averaged into 5 minute bins. The graph below shows the distribution of my heart rate (bpm) for several days in April 2016. The histogram reveals a bimodal distribution with centers around 58 and 80 bpm. The centers were identified using a Gaussian…
Sometimes useful to split an integer into its digits and often its desirable to revert the process (digits back to an integer). The first part is trivial using the base function digits . Unfortunately the second part has no base function. Integer to digits The digits function in base takes an integer (and optionally a base and padding) and returns an array of the digits of the integer. julia>…
As an excuse to write more code in Julia, I’ve been working on exercises from Project Euler. These are a collection of numerical coding problems of varying difficulty. Since I’m still learning my way around Julia it is likely that I’ll learn tricks while solving later problems will also apply to earlier problems. Although Project Euler doesn’t evaluate the code or how…
Update (2018-08-10) I’ve written a newer, better post about how to serialize numpy.float32 to JSON . It also covers how to serialize other data types. I recommend reading that post instead. Earlier this week I made a small modification which promptly broke the code I was working on. The change was so minor that it was very difficult to see why the changes would cause it to fail. As it turned…
Apple has used the name AirPort since 1999 on wireless products including routers and Wi-Fi cards. Additionally every Mac comes with an obscure utility called airport. The airport utility can do several things such as get/set preferences, create logs, and even sniff frames. Gettings Around Set up a symlink The airport utility is buried in the system’s library folder. I added a symlink so…
I had several gigabytes of video files in an ancient video format (AVI) that I wanted converted. QuickTime does not play well with AVI files so I immediately fired up VLC. After a few minutes of poking around I figured out that there wasn’t a good way of batch transcoding. There was the option to run VLC from the commandline but then I realized VLC was just a wrapper around other libraries…
GRidded Binary (GRIB) is a file format used to store meteorological and oceanographic data such as precipitation and temperature. Forecast and analysis datasets distributed by government meteorological agencies, such as NOAA, are often in the GRIB2 format. The format is used for distribution because it has high compression; one example shows that a NetCDF-3 file for the same data has a 6.4 times…
Back in September, Bob Carpenter posted a Stan puzzle to Andrew Gelman’s blog. I didn’t notice until Bob posted a second puzzle in November. I’m still learning to use Stan and thought this puzzles would be great exercises. So I decided to take a look and try to solve the puzzle myself (although the solution has already been posted). Free Throws Here is the puzzle as posted on…
R batch processing is a way to run R scripts non-interactively from the commandline. This is especially useful when you want to generate a log file but don’t need the (minimal) overhead of R Markdown – or the major overhead of Sweave/knitr. Running a script in batch mode is easy but doesn’t have very good error handling behavior. By this I mean that if R encounters an error, it…
Stan is a probabilistic programming language used to fit full Bayesian models. It is relatively new (initial release was August 2012) but the language has a lot of support. It is the successor to WinBUGS and OpenBUGS and JAGS. A noteable feature of Stan is that it uses the No U-Turn Sampler (NUTS), an adaptive variant of Hamiltonian Monte Carlo (HMC) sampling. While familiar with WinBUGS and INLA…
In programming there are many ways to accomplish the same task. Take the case of calculating the average (mean) value of a raster stored in a PostGIS database. The task seems simple enough – surely there is a built-in function to do exactly this. Such a function, ST_Avg , would be used like this: SELECT ST_Avg(my_rast) FROM my_table WHERE rast_id = 7514; however the function ST_Avg does not…
I bought my Fitbit Flex from Costco sometime around 2:00 PM on January 4th, 2015. I had been holding out for the Charge HR model - which ironically was released the next day - but ultimately decided that having continuous heart rate measurements wasn’t worth the extra $50. After all I am certainly capable of taking my own heart rate. Its been about 7 months so I thought I would take a look…
Today I decided to write a function to read a PostgreSQL connection string from an environment variable and to setup the connection. This is a pretty simple function but I couldn’t find a built in way to do this using DBI.jl and PostgreSQL.jl . The main assumption is that the database connection string is an environment variable, which allows you to make database connections without exposing…
While working with a DB of about 135,000 rasters my co-worker Frank and I discovered that despite coming from a reference grid, some of the rasters weren’t properly aligned. Although we had a reference shapefile containing the correct tile positions, PostGIS doesn’t have an easy way to snap to a reference geometry. So instead Frank used ST_SnapToGrid and the upper left corner of the…
rs762551 is a single nucleopeptide (SNP) encoding the CYP1A2 1F allele of the CYP1As gene. The CYP1A2 enzyme is encoded as the CYP1A2 gene in humans, and belongs to the cytochrome P450 superfamily. Along with other enzymes in the CYP superfamily, the rs762551 SNP is cited as being associated with xenobiotic (xeno-forgien, bio-life) metabolic functions. Specifically the SNP has been reported to be…
One of the useful features of random forests is the ability to assess the importance of predictor variables. This can be the mean decrease in accuracy or mean decrease in node impurity when excluding this variable. While many statisticians, data scientists, and machine learnists are familiar with variable importance measures for random forests, I don’t often see many analyses that include…
Every time I check-in, it seems that Julia has taken another small step forward. When I first learned of Julia I encountered a lot of issues that prevented me from performing even simple data tasks. Now at version 0.3.6 I’ve decided to make another attempt at using Julia for a mini-analysis. The purpose of this is to see how Julia might grow as a contender against other languages (e.g. R,…
Make something that people want. Everyday I see the Y Combinator motto hanging on the wall at the office. The motto is only five words, but the words urge us to create something. But we can’t simply create anything . Y Combinator reminds us that what we are creating is not just for ourselves but is something that should be desired by others. The process of making something that people want…