Recent content on Connor Skennerton
SQL is lingua franca of the data world. Every tool supports it or has some version of it. I’ve been exposed to Postres, Trino, Spark-SQL, Flink-SQL, MySql, MS-SQL and they all work the same way for basic things. Of course the real challenge is when things get more complicated, certain functions have different names, or don’t exist, and some query patterns that work well in one engine…
Writing these down because I always forget… Replace all occurances of something create a selection. For example, using x for lines, v for a flexible selection, or % for the whole file search in the selection with s followed by whatever you want to search for, then enter (this will create a whole bunch of cursors) use c to change the text and enter insert-mode to replace
Experience Senior Software Engineer Roblox 2022 - Present Part of the core Data Engineering team to build growth accounting and engagement datasets. Developed tooling and frameworks to abstract boilerplate of building data pipelines. Senior Staff Data Engineer Pendulum Therapeutics 2019 — 2022 Worked cross functionally with relevant stakeholders at Pendulum to architect the full stack of software…
Helix is a modal editor in the same vein as Vim which means that there are different “modes” where the same key strokes mean different things. The main benefit for a modal editor is that many complex actions can be handled with just the keyboard rather than using the mouse or menus. I’ve been a Vim user for over a decade so I was interested to see what a modal editor that…
One of the things that I’ve discovered about parenting is that babies are tracked religously. How much are they sleeping? Are they eating enough? Did they pee/poop the correct number of times in the last 24 hours? This isn’t just frettful new parent stuff (although it totally is that too) but things that pediatricians want to know and could be the early signs of problems when there…
I am a Data Engineer currently working at Roblox . I received my PhD at UQ in the Australian Centre for Ecogenomics . Throughout my PhD I was at the interface of microbial ecology, chemical engineering, and computational biology and have gained many skills ranging from microscopy, reactor operation and computer programming. During my PhD, I developed many data analysis skills, including…
Yesterday I went to a data engineering meetup hosted by Roblox . The talk in the meetup was by Yan Shen and William Ng on how they cut down on processing costs in their data processing pipelines by making use of datelist tables. A datelist table acts as an intermediate incremental accumulating aggregate of a quantity from a fact table. Their key feature is that they have a column that contains an…
Recently, one of my colleagues came to me with a mysterious problem they were having with one of our production tasks running on AWS Batch. This particular task would be launched based on user interaction in a internal web application and run a fairly hefty machine learning model in docker container. Recently, this job started failing, not all the time, but sometimes on larger input files the job…
I’ve been working on an expanded AWS SDK for Pharo. Currently, 234 AWS services are available. See the code on GitHub . Amazon Web Services (AWS) is a huge set (approximately 300 at time of writing) of services for doing just about anything related to computing infrastructure and tools, often with multiple ways to achieve the same or similar thing. The awesome thing about AWS is that there…
I’m constantly learning new things about the Python language. I consider myself a pretty good python programmer but often you never need to use all of the language features when writing your own code. For example I’ve not used is the class factory pattern using the type built in function. I’ve been aware of class factories, and read a few blog posts but never grokked it, until…
Filing taxes in America sucks. Your options are to do it by hand, pay someone like Intuit, or if you are below a certain income threshold get some tax software for free. The kicker is that free tax software is from Intuit who will try very hard to make sure that you either don’t find it in the first place, or try to get you to pay for it and upsell you on something that should be free. Check…
Smalltalk syntax can be a little confusing coming from other languages. Here I’ll show some comparisions between Python string operations and Smalltalk. Substrings / Slicing Python strings use the slice notation where you can place up to three colon-separated values for the start, stop, and step. Python strings are 0-indexed and the stop argument is one past the final element that you want.…
Starting out with Smalltalk can be a little jarring as it doesn’t have the similar syntax as launguages that are more heavily inspired by C. Dictionaries are one kind of data structure where I noticed this the most so I put together my notes on using them in Pharo with some comparisons to Python. In many other languages there is a subscript operator that allows you to access a value in a…
One of the key differences between SQL databases and graph databases is the concept of joining information from different nodes. In a tinkerpop-enabled graph database nodes have labels that define their type and properties that are part of that type. It’s natural to draw the comparison to a label being an SQL table and the properties of the nodes being the columns of that table. But trying…
The first time I heard about Smalltalk was reading through the wikipedia page for Ruby , which mentioned it as an influence. At the time I was just a few months into my transition from a wet-lab biologist into a bioinformatician and trying to decide between Perl, Python, and Ruby as a scripting language to learn. Python became my language of choice after a long battle with Perl (this was some…
Make has been a one of the key tools in my arsenal for gettings things done. Although it was developed for compiling code, its functionality can be generalized to any process that requires files to be generated based on dependancies. I recommend you look at these slides by Vince Buffalo as a good introduction to using make for scientific workflows. Make works by creating a dependancy graph of…
I use KEGG a lot to understand microbial metabolism. KEGG is one of the largest resources of enzymes, biochemical reactions, genes, and molecules, all cross-linked and organized into what’s called metabolic maps. These maps are well-constructed images of enzymes that functions together for the same overall purpose like amino acid synthesis, or the metabolism of glucose. One of the great…
As part of my PhD thesis I studied an emerging field of bacterial adaptive immunity, known as CRISPR . At the time I was interested in tracking this type of immune system in bacterial communities to track co-evolution between bacteria and their viruses. For anyone interested and brave enough, here is a link to my thesis. After I finished up writing, submitting, and ultimately obtaining my PhD I…
I’ve been trying out Amazon Neptune’s full text search feature. Overall it’s been a great experience although there are a few caveats when searching that means that you’ll have to craft your queries carefully to make full use of the feature. The tinkerpop standard has some text searching features however it lacks any advanced features such as searching using regular…
Bioawk is a great project started by Heng Li some years ago. The aim was to take the awk source code and modify it slightly for use with common biological formats and adding in some new functions. Heng’s original doesn’t accept too many pull requests so to add in some features, I maintain my own fork that has a few improvements. Long ago I added in a translate function and recently I…
NCBI’s assembly database is a great one-stop-shop for genomic data and annotations but it’s actually kind of difficult to download data if you only know the accession number of an assembly. The documentation says that the assembly database is integrated with entrez-direct , a great set of command line utilities for accessing NCBI data from the command line. Most of the databases have…
I love looking at KEGG maps and using them to understand an organisms metabolism but they have their limitations. For starters, you’re obviously stuck with how they are drawn, which in most cases includes many variations on a particular pathway. Secondly, the tools for mapping on your own genes to a pathway are limited to one organism at a time. What I really wanted was a way of quickly…
Here is a function that will take a character string in R and return an expression for fancy formatting in plots that properly italicize scientific names. The syntax for doing this is truly quite horrible, but this is how R does it. scientific_name_formatter <- function ( raw_name ) { # strsplit returns a list but we are passing in only # one name so we just take the first element of the list…
Unix has so many great ways to perform text manipulation but one niche which hasn’t been filled is splitting a tabular file into pieces based on the contents of certain columns. There are two commands, split and csplit , that do a similar role. split can split a file into a certain number of bytes or lines; csplit uses a regular expression to determine where to split the file. Often for my…
The Entrez Direct toolkit is great for programmatic access to all of NCBI’s resources. This little snippet below finds all of the refseq representative genomes for a given NCBI taxonomy ID, makes a little summary of the genomes downloaded and uses wget to download the genbank files from the Assembly FTP. Change the inital query in the first call to esearch to change what genomes are…
I feel like I’m on a life-long quest to make all of my phylogenetic tree figures completely programmatically. The best tool I’ve found for making them is the ete library for the python programing language. I’ve already figured out how to get trees drawn in the style that I like but there was still one thing left to do: making organism names italicize correctly. I work with…
So. You’ve got yourself a nice new genome sequence and you want to know what kind of metabolism it has. There is a good chance that you have some idea already — you think it’s a nitrogen fixer or a sulfate reducer etc. — based on other analyses and now it’s time to strengthen your paper with a bit of genomic evidence. Getting an initial annotation The vast majority of the genes…
ete3 has support for phyloxml which I use with archaeopteryx tree viewer for a lot of my day-to-day phylogenetics visualisation. My main reason for using phyloxml is one of convenience as I have a script that will easily add in the proper organism name onto the tree and I think that archaeopteryx is a really good basic tree viewer. I wanted to draw a tree from phyloxml in ete using my own style…
Bioinformatics… Or ‘advanced file copying’ as I like to call it. — Nick Loman (@pathogenomenick) January 29, 2014 Get ready for some advanced file copying! I recently had to clean up some data from the supplementary material from Pereira et. al 2011 , which is a very nice table of manually annotated genes in sulfate reducing bacteria. The only problem is that the table is…
I do a lot of work in phylogenetics, which means that for just about every paper I’ve written I’ve had at least one figure that is a phylogenetic tree. Making pretty looking trees for a publication is tedious and my previous workflow involved using ARB for actually drawing the tree and producing an initial file in postscript, and then loading that into Adobe Illustrator to make…
Genome bins comming off automated pipelines can be contaminated with parts of other genomes. As part of my workflow I use CheckM (I’m biased since I’m a coauthor) to assess the contamination of genome bins using single-copy marker genes. If you’re lucky then the genome bins that you’re interested in will be relatively complete without much contamination. Unfortunately that…
With the pace of science what seemed top stuff three years ago is now an order of magnitude less than what just got published. When I was in my PhD I was working on EBPR communities using metagenomics to characterize the the microbial and phage populations. From all of my microbial data I could get about 70 draft bacterial genomes. At the time (2012) that was pretty huge considering the first big…
The question that most people ask when looking at a metagenomic draft genome bin is: should this gene really be there? The answer is that sometimes it’s not easy to know I spend a lot of my time looking at genome assemblies. They are almost always from metagenomic data and usually are from some novel phylogenetic lineage with few (if any) “close” relatives. Unfortunately the…
Every PhD student can contribute to open science by uploading their thesis literature review onto Wikipedia! My thesis was entitled “Phage-host evolution in a model ecosystem”, where I tracked the evolution of phage genome evolution and the evolution of bacterial defense mechanisms using metagenomics. When I was writing my thesis I spent a lot of time writing up the section on CRISPRs…
Can you remember the first useful thing that you ever coded? I sure can, and I’m thankful for it every day. I’ve recently finished writing a little program called fxtract (which I’ve blogged about before ) that acts like grep but returns whole fasta or fastq records from a file. It’s taken me a very long time to write this thing, primarily cause I’m writing it in C++…
Got a Metagenome? want to know what the community looks like? rRNA operons are typically poorly assembled in metagenomic datasets due to highly conserved sequences. More targeted assembly approaches may be necessary to obtain accurate reconstructions from short read datasets. There are a few ways in which we can extract reads originating from either 16S or 18S reads and there are a number of…
Playing around with the grep source code to make it output fasta/fastq records. Check out the code here . I’m quite interested in string searching algorithms as I’ve written a program, crass , which uses a few of them to search for CRISPR elements. Crass is pretty fast, but I want it to be faster, specifically there is one point in crass where it searches for exact matches to many…
Experiences using a variety of contig scaffolding tools. It was not a good experience. Recently in our lab we’ve been getting some Illumina mate-pair data to improve some metagenomic assemblies. The sequencing has been going well and we’ve been generating a good number of mate-pairs without too much duplication, but we’ve had quite a bit of trouble with the bioinformatic part of…
I recently discovered Seqan , a header-only C++ library for bioinformatics. I’ve been playing around with the toolkit to make some small programs just to see whether I want to use it in a larger project. So far I’ve written prepmate , an adaptor trimming program for Illumina’s Nextera mate-pair libraries; and fxtract , a grep-like program for extracting fasta/fastq records from…