A blog on software development by Andrew Berls
Overseer is a library for building and running data pipelines in Clojure. Workflows are designed as a graph (DAG) of dependent tasks, and it handles scheduling, execution, and failure handling among a pool of workers. It ran the vast majority of the Framed predictive analytics platform for over a year up to the acquisition by Square, and today we’re happy to release the project as open source to…
In the last post, we introduced the Overseer workflow engine that ran Framed, and saw how to use plain Clojure data structures and functions to wire up an example pipeline. In this post, we’ll talk about simplicity and how we were able to leverage Overseer and a variety of techniques to do away with Hadoop and other heavyweight processing frameworks.
Note: This is an unmodified re-publication of a post that originally appeared on Medium.com in October 2015. At Framed, machine learning is our bread and butter. Every day, we run a lot of models over a lot of data, and recently ran up against some limits in the disk serialization format we were using to store our training datasets, which models use to discover patterns between input attributes…
Now that the dust has settled from Square’s acquisition of Framed, I wanted to take some time to talk about the building blocks of simple software, and the things I think we got right at Framed. The Framed platform was written entirely in Clojure, and at its peak it was processing hundreds of millions of analytics events per day and running machine learning models over tens of millions of input…
I have always been an anxious person. It’s a pervasive undercurrent, the constant voice of ‘no’ in my head; No, I don’t want to go to that party where I only know one person. No, I’ll talk to that girl tomorrow. It’s all too easy to succumb to this voice, and I’ve been spending a lot of time thinking of what to do about it.
When I moved into a short-term apartment in January, my then-roommate had an espresso machine, a gorgeous Breville BES900XL. He taught me how to use it, and I immediately fell in love with making cappuccinos in the morning. My roommate knew all about the different types of art, and taught me how to froth the milk and make a simple tree shape. It’s trickier than it looks, and most of the time I…
For a brief period of time after college, I ran a small consulting firm with a partner. During that time we had a short but insightful conversation with the CEO of a client company, asking for his advice and talking about general motivations. Several things he said really stuck with me - this post is about one of those things. ~ My partner and I incorporated an LLC for our firm. We had not…
If you’ve ever spent any time reading about functional programming, you may have heard about “partial function application” or “currying”. For the longest time these were just big scary words to me, but it turns out they’re relatively simple concepts. I’ve read plenty of posts describing what they are , but it wasn’t until recently that I really came to grasp why one would ever need them. In this…
This is part 3 of a series taking an in-depth look at how the internals of Rails handle requests and produce responses - be sure to catch up on previous parts! Part 1 - Introduction + Unicorn Part 2 - Routing Last time we focused on how requests are routed to controller actions through Journey and the ActionDispatch stack. This time, we’ll look at ActionController and how controller actions turn…
This is part 2 of a series taking an in-depth look at how the internals of Rails handle requests and produce responses - you can read part 1 here. Last time we took a brief look at how Unicorn workers accept clients from a shared socket and call our Rails application, and how requests bubble down through the middleware stack before arriving at Blog::Application.routes. We’re now at the routing…