Recent content on Vince Buffalo
Disclosure: camdl is an open-source project I am developing at the Institute for Disease Modeling (Gates Foundation). I'm cross-posting here from internal channels since the project is public; the views expressed are my own and don't represent IDM or the Foundation. Engineering scientific software is a different problem than general software engineering. The most consequential bugs in scientific…
The Great Obsession of population geneticists, to borrow John Gillespie’s words, is genetic variation. As an evolutionary biologist, it’s rather hard to not be obsessed with genetic variation, for it’s the ultimate source of the two most striking features of life on earth: the mind-boggling diversity of species, and adaptations so utterly clever they look as though they were…
The last chapter of my dissertation with Graham Coop was recently published in PNAS ( pdf , bioRxiv ) last week. In an effort to communicate my research to a broader audience, I have written two blog posts on our work. The first post , is meant to introduce the historical context and concepts like linked selection and polygenic adaptation to a non-scientist, and the second post, below, describes…
The last chapter of my dissertation with Graham Coop was recently published in PNAS ( pdf , bioRxiv ) last week. In an effort to communicate my research to a broader audience, I have written two blog posts on our work. The first post, below, is meant to introduce the historical context and concepts like linked selection and polygenic adaptation to a non-scientist, and the second post describes our…
Heraldic snake from Flickr (CC Licensed). Each day, data scientists, computational biologists, astronomers, and other folks that spend far too much time in front of a computer screen spend hours doing somewhat horrible, monotonous tasks. Scientific programming, when done right, is supposed to prevent us from doing these monotonous tasks, and this is certainly true when we compare what we do today…
An example of a present-day female's X material being broken up across her X ancestors in her X genealogy back through the generations. My article with Steve Mount and Graham Coop, A Genealogical Look at Shared Ancestry on the X Chromosome has been published in Genetics . In the spirit of both outreach and continuing Graham’s terrific series of blog posts 1 on genetic genealogy, I’m…
I had a matrix (88662 loci x 2060 genotypes) of maize chromosome 1 genotypes, encoded as 0, 1, 2 (e.g. the number of alternate alleles). I needed genotype counts per row, which at first glance is quite easy to solve: just use apply and table : counts <- apply ( chr1g , 1 , table ) What’s the problem with this approach? First, technical: if a row only has genotypes 0 and 2, we don’t get…
I needed to work with the MD tag in BAM/SAM files for a recent project. There’s not too much discussion online about this, so I took some notes as I went through examples. The MD tag is for SNP/indel calling without looking at the reference. It does this by carrying information about the reference that the read does not carry, for a particular alignment. A SNP’s alternate base is…
It’s hard not to fall in love with Unix as a bioinformatician. In a past post I mentioned how Unix pipes are an extremely elegant way to interface bioinformatics programs (and do inter-process communication in general). In exploring other ways of interfacing programs in Unix, I’ve discovered two great but overlooked ways of interfacing programs: the named pipe and process substitution.…
Day to day bioinformatics involves interfacing and executing many programs to process data. We end up with some refinement of the data from which we extract biological meaning through data analysis. Given how much interfacing bioinformatics involves, this process undergoes very little thought or design optimization. Much more attention is needed on the design of interfaces in bioinformatics, to…
Note: I’ve been writing this essay in my head for a few months, but I felt it needed to be completed and released after the sad loss of Open Access advocate Aaron Swartz , a hacker and activist I admired. 21st Century Science and the Need for Open Data and Open Tools Open Science rests upon three core principles: open access, open data, and open tools. However, “open science”…
Scientists are learning programming at an unprecedented rate. I’ve expressed concern over the fast-paced growth of computing across the sciences and what this could mean for reproducibility and incorrect findings in the sciences. Perhaps the best example that illustrates the severity of this issue is Coombes and Baggerly’s Duke Saga . I think a lot about how scientists learn…
Simple Parallel Bioinformatics Pipelines with find, basename, and xargs Big-Ass Servers and Data Parallelism A routine operation in bioinformatics is to process a lot of files on a so-called “Big-Ass Server” . In most cases, these have to be processed using the same tools, in the same way, making this a prime example of data-parallism. The unit of data divided across multiple cores is…
Bioconductor is one of the open source projects of which I am most fond. The documentation is excellent, the community wonderful, the development fast-paced, and the software very well written. There’s a new package in the development branch (due to be released as 2.10 very soon) called gwascat . gwascat is a package that serves as an interface to the NHGRI’s database of genome-wide…
Git Notes These are updated by me periodically. I have tried my best to illustrate common use cases, and the motivation for doing things the “Git” way. Example Set Up I’ll use this setup scenario frequently. In a suitable scatch repository (i.e. git-sandbox ), make a fake remote: mkdir fake-remote cd fake-remote git init --bare cd .. Now, clone it, pretending you are two…
In talking with bioinformaticians, biologists, and other researchers, I’ve seen some worrying trends in computation in the sciences. I plan on writing about these extensively in the future, as I believe computation in the sciences will not scale well to face the huge wealth of data coming experiments will provide. This is not due to algorithmic or hardware limitations, but rather to the fact…
Hello, Julia Julia is an exciting new technical computing language. It’s still in its infancy, but it’s fast (see below), and already does a lot. Comparison of Julia to other languages There’s been some excitement on Twitter about Julia. Excitement combined with open source often yields development, which then leads to further excitement, until a mature open source project…
In between refactoring some qrqc code this morning and looking at RNA-seq data, I grabbed some cold brew coffee and caught up on some missed tweets. Admittedly, my brain glosses over most tweets, but this tweet from Drew Conway had the right mix of keywords to actually make me click and read the link: The data science debate: domain expertise or machine learning? by @medriscoll…
Recently a coworker showed me a FASTQ file from an Illumina HiSeq run (which will be packaged in the new release of my Bioconductor package qrqc ) that was severely contaminated. Below is the file in less with a string highlighted: A severely contaminated file in less, with many contaminants highlighted Holy contamination, Batman! There are a few approaches to handling this level of contamination.…
Please developers, don’t be dicks. As the author of a few open source tools, I’ve had my fair share of users seeking help. Emails range from the very useful (bug reports, patches, etc) to the annoying (“can you help guide me through this process”). But never once (that I can remember) have I been a dick (and yes, I’ve wanted to be). It will be tricky to write this…