Recent content on Spootnik.org
I have long parted with my initial, lacking approach to component handling in Clojure. I now rely on Stuart Sierra’s component library for this. In this short post, I want to showcase how this library helps structure code around clear functional boundaries and allows testing without having to depend on mocking. This might induce building components for seemingly innocuous code. I will also…
Atoms provide a way to hold onto a value in clojure and perform thread-safe transitions on that value. In a world of immutability, they are the closest equivalent to other languages' notion of variables you will encounter in your daily clojure programming. Storing with an atom One of the frequent uses of atoms is to hold onto maps used as some sort of cache. Let’s say our program stores a…
I recently wrote about kafka log compaction and the use cases it allows. The article focused on simple key-value storage and did not address going beyond this. In practice, values associated with keys often need more than just bare values. To see how log compaction can still be leveraged with more complex types, we will see how to approach maintaining the state of a list in kafka through the lens…
A hands-on dive into Apache Kafka to build a scalable and fault-tolerant persistence layer. With its most recent release, Apache Kafka introduced a couple of interesting changes, not least of which is Log Compaction, in this article we will walk through a simplistic use case which takes advantage of it. Log compaction: the five minute introduction. I won’t extensively detail what log…
Wherever your init daemon preference may go to, if you’re involved in writing daemons you’re currently faced with the following options: Start your software in the foreground and let something handle it Write a huge kludge of shell script which tries to keep track of the daemons PID Work hand in hand with an init system which can be hinted at the service’s PID Why is tracking…
One of Alan Perlis' famous quote is It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures. In a recent talk about reducers (available here), Rich Hickey playfully asserted that it might be even better to have 100 functions operate on anything. This article aims to show the effective way clojure offers to emulate common datastructures to use…
Clojure 1.7 is around the corner, we’re already at version 1.7.0-alpha3. Fortunately, the iterative approach of clojure taken since 1.3 means that upgrading from one version to the next usually only needs a change in your project.clj file (as well as working interop accross versions, which is always nice). There are some neat changes in 1.7, the most notable being the addition of…
A while back we did a small introductory talk on angular.js with @brutasse. Our talk was aimed at backend developers looking for a solution to build simple interfaces for REST services. The app used exposed a simplistic job board with no persistence and a mere three views. Our main user-facing app at exoscale is an angular one and we also have a large internal one. Most of our backend work is done…
*TL;DR*: I love clojure.tools.logging, but setting JVM logging up can be a bit frustrating, I wrote logconfig to help. When I started clojure development (about 5 years ago now), I was new to the JVM - having no real Java background. My first clojure projects where long running, data consuming tasks and thus logging was a consideration from the start. The least I could say is that navigating the…
pickle is python’s serialization format, able to freeze data, as long as all leaves in class hierarchies are storeable. pickle falls into the category of formats that I’m not a huge fan of. Like all serialization formats heavily tied to a language, it makes interop harder and pushes platform and language concerns all the way to the storage layer. You could do worse than look into…
A recent vulnerability in iOS and Mac OS can boils down to a double goto resulting in making critical ssl verification code unreachable. hashOut.data = hashes + SSL_MD5_DIGEST_LEN; hashOut.length = SSL_SHA1_DIGEST_LEN; if ((err = SSLFreeBuffer(&hashCtx)) != 0) goto fail; if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)…
In a previous article, I tried to make the case for using a private certificate authority to authenticate access to internal tools with SSL client certificates. This approach is perfect to secure access to the likes of kibana, riemann-dash, graphite or similar tools. If you start depending more and more on client-side certificates, you’re bound to reach the point when you need to tackle…
When writing daemons in clojure which need configuration, you often find yourself in a situation where you want to provide users with a way of overriding or extending some parts of the application. All popular daemons provide this flexibility, usually through modules or plugins. The extension mechanism is a varying beast though, lets look at how popular daemons work with it. nginx.org , written in…
I recently stumbled upon this article http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/ by Michael Noll which explains a strategy for computing twitter trends with Storm. I love Storm, but not everyone has a cluster already, and I think computing tops is a problem that lends itself well to single node computing since datasets often are very simple (in the…
Nginx is well-known for only logging to files and being unable to log to syslog out of the box. There are a few ways around this, one that is often proposed is creating named pipes (or FIFOs) before starting up nginx. Pipes have the same properties than regular files in UNIX (to adhere to the important notion that everything is a file in UNIX), but they expect data written to them to be consumed…
I thought I’d whip together a short blog post as a reaction to Mark Burgess' (@markburgess_osl) keynote at the Cloudstack Collaboration Conference. I thoroughly enjoyed the keynote but was a bit taken aback by one of the points that was made. It can be summed up in a quote that has since been floating around: I think a few things are misguided in Mark’s analysis of what is meant by…
It’s a shame that so many organisations rely on HTTP basic-auth and self-signed certs to secure access to internal tools. Sure enough, it’s quick and easy to deploy, but you get stuck in a world where: Credentials are scattered and difficult to manage The usability of some tools gets broken Each person coming in or out of the company means either a sweep of the your password databases…
A quick tip which helped me out in a few situations. I’d be inclined to point people to core.match for any matching needs, but the fact that it doesn’t play well with clojure’s ahead-of-time (AOT) compilation requires playing dirty dynamic namespace loading tricks to use it. A common case I stumbled upon is having a list of homogenous records - say, coming from a database, or an…
Another quick blog post today to show-case usage of riemann for in-app reporting. This small blurb will push out a metric for each wrap route with the metric name you provide. When handlers raise exceptions, a metric is sent out as well with the exception’s message as description. import socket import time import bernhard # [...] def wrap_riemann(metric, client=bernhard.Client(),…
Many thanks to the #devops crew for hosting devops days Paris ! I loved the diversity of the talks, the ignite format is really interesting. What struck me most is how devops is now reaching its intended audience, i.e: everyone involved in delivering quality software and services. It was really nice to see people from all types of companies and all types of positions. Here are the slides for the…
I did a bit of frontend work for internal tools recently and chose to go with clojurescript, for obvious reasons. Like clojure, clojurescript supports macros which let you express common lengthy idioms. I use the jayq library to interact with the browser since JQuery is the de-facto standard and battle tested. A standard call to a JSON get in JQuery looks like this: $.getJSON('ajax/test.json',…
There’s a fair amount of things that are pretty much set on current architectures. Configuration management is handled by chef, puppet (or pallet, for the brave). Monitoring and graphing is getter better by the day thanks to products such as collectd, graphite and riemann. But one area which - at least to me - still has no obvious go-to solution is command and control. There are a few…
I was very happy to give a talk on the experience we had with cassandra over at paper.li for the first Paris-Cassandra meetup (if you’re interested, go follow @pariscassa). While not diving too deep, I tried to give a battlefield overview of how we came to the decision to go for cassandra, what we do with it and a few tips should you choose it, very much in the same vein than my previous…
I use jenkins extensively to test apps and clojure is my go-to language for plenty of use cases now. leiningen is a very nice build tool that lets you get up and running very quickly, even for java only projects. It relies on maven under the covers. Its only drawback used to be the lack of support in jenkins which left you with two choices: Add a pre-build step that would get leiningen Produce…
Clojure at paper.li I’ve been involved with clojure almost exclusively for a year as smallriver’s lead architect, working on the paper.li product and wanted to share my experience of clojure in the real world. I had a previous experience with clojure where I put it to work where ruby on rails wasn’t a natural fit, and although smallrivers is a close neighbor of typesafe in…
What does a good nerd do when feeling the urge to write on a topic ? Why, update his blogging tool, of course. I’ve actually loved static site generators since the days where I had one based on make and m4, I also want a lot of simplicity from the tool I use to blog. I started out with posterous because I didn’t want to bother with hosting details from a blog, being a technology…
Taking on a new platform design recently I thought it was interesting to see how things evolved in the past years and how we design and think about platform architecture. So what do we do ? As system developers, system administrators and system engineers, what do we do ? We develop software We design architectures We configure systems But it isn’t the purpose of our jobs, for most of us, our…
Protocols and mixins I recently had to implement something in clojure I’ve done many times in ruby, which involved using protocols. I thought it would be a nice example of comparing class re-opening in ruby and protocol extension in clojure. The problem I use cassandra, and in many places, cassandra needs to work with UUID types a lot. When exposing results over JSON, this is often a problem…
A bit of sugar The wrap-with function described in my last post1 is useful, but you still end-up having to write closures which might be confusing to people who just want to write simple wrappers. Fortunately, clojure provides the ability to enhance the language with syntactic sugar for use cases such as this one. A word of warning I’m obviously going to talk about macros in this article. I…
Functions in the wild Functional programming rears its head in the most unusual places in software design. Take the web stack interfaces (rack1, plack2, WSGI3), they all implement a very functional view of what a web application is: a function that takes a map as input and returns a map as output. Middleware then build on that abstraction composing themselves down to the actual handler call. The…
I just uploaded a small library to github (and clojars), It’s a generalisation of what I use for recurrent tasks in my clojure programs, the fact that the recent pragprog article had a handrolled version of it too convinced me it was worth putting together in a lib. The library provides an easy mechanism to register process for later execution, either recurrent or ponctual. It is called…
Lately, monitoring has been a trending topic from the devops crowd. ripienaar and Jason Dixon amongst other have voiced what many are thinking. They’ve done a good job describing what’s wrong and what sort of tool the industry needs. They also express clearly the need to part from a monolithic supervision solution and monolithic graphing solution. I’ll take my shot at expressing…
Found myself typing this in a shell: gem list --local |\ egrep '.*(.*,.*)' |\ sed 's/^\([^ ]*\) ([^,]*,\(.*\))/\1\2/' |\ sed 's/,//g' |\ awk '{for (i = 2; i <=NF ;i++) {printf "sudo gem uninstall %s --version=%s\n", $1, $i}}' Sometimes perl or ruby with -e is just faster
When running pf firewalls and loadbalancers with relayd, some metrics are critical. One thing that might not be obvious when looking at pf is that the maximum number of sessions a firewall can handle is fixed, as evidenced from looking at the output of pfctl -si Status: Enabled for 10 days 00:39:47 Debug: err State Table Total Rate current entries 6 searches 7970148 9.2/s inserts 60227 0.1/s…
When you dive in a complex project, even though you try to be as good as possible with documentation, sometime you just miss things. I’ve had a good opportunity to realize this just recently while working on puppet. A common head scratcher in puppet is finding a way to keep system, technology and platform specifics separate. I will probably write at length on this particular subject a bit…