Software, data, and cloud infrastructure engineer
I generally avoid referring to anything as technical debt. While not an entirely meaningless term, I suspect that in almost all contexts there are better things to say. In this post I argue why I think the term shouldn't be used, and what should be said instead. It means too many things I've heard the term technical debt being used for: Bugs that exist right now Properties that risk introducing…
It can be extremely tempting to announce that something is done , essentially because it can make you and others smile and feel good—we've made progress! But if there are still steps to be taken it's almost definitely not going to be seen as done from the point of view of the person or people you're talking to. Saying something is done when it's not at best is not helpful, but at worst makes you…
I so frequently see extremely capable software engineers waste their skills with over-engineering, and it makes me feel sad. Here are some tips to help avoid the over-engineering trap, and so also to help keep me happy 😀. Find out what is seen as valuable We're not here to write code, but to solve problems. Find exactly what problems you are expected to solve and focus on those. Remember that…
There are virtually infinite options on how to split up all but the most trivial pieces of software engineering work, but rarely is emphasis placed on the skill of making that choice. But this choice is extremely important: the order in which work is done affects how frequently feedback is received (even from just running the code yourself) and so can seriously affect how successful projects are…
I've made a few data dashboards recently, and I've realised it can be hard to know where to begin sometimes. Hopefully these rules can help focus your thoughts. 1. Know the use cases Just like designing anything, if you don't know how it's going to be used, it's not likely you're going to do a good job. Often there aren't actually that many use cases. But if there are - you can construct different…
I've been fortunate to work on projects where there is often a direct (or at least short) route between myself and the users of products I've worked on. These are the sorts of questions I ask myself when communicating with them. Am I answering their question? I might suspect that they're not asking a question where the answer will ultimately help them (an XY problem), but I can't be sure. In most…
To offer HTTP file downloads via your own code [rather than redirecting elsewhere], it's often easy to rustle something up. However, the default behaviour in a lot of cases may not give users as good an experience as possible. With a bit of effort, you can polish that right up, and here are 4 sets of HTTP headers that help you do just that. content-length If you are able to, set the content-length…
Something that occasionally catches developers out is the fact that S3 is not a filesystem, but a key-value store. Specifically the keys can be any UTF-8 encoded string, between 1 and 1024 bytes long; the values can be any binary string, beteen 0 bytes and 5 terabytes long. Yes, you can emulate certain features of a filesystem using slashes in the keys. In fact, the AWS console does this: it…
I have a confession: I assumed things about Django's transaction.atomic() that are not true, at least not true by default in PostgreSQL. I assumed that in a transaction.atomic() context as below, database statements are protected from any race conditions, and everything will Just Work™. with transaction . atomic ( ) : # Database statements But that's really not true. Enter the world of transaction…
Here's Queue , a small utility function to queue and run tasks with limited concurrency. const Queue = ( concurrency ) => { var running = 0 ; const tasks = [ ] ; return async ( task ) => { tasks . push ( task ) ; if ( running >= concurrency ) return ; ++ running ; while ( tasks . length ) { try { await tasks . shift ( ) ( ) ; } catch ( err ) { console . error ( err ) ; } } -- running ; } } You…
Python's asyncio gets a fair bit of bad press. Some of it I agree with, but there is one aspect of asyncio I like: the API needed for a lot of common tasks is actually fairly small and clear. Here's a small but fairly realistic program. It creates a pool of HTTP connections, and uses this to make two concurrent chains of requests. import asyncio import httpx async def async_main ( ) : async with…
I've noticed a bit of a skill gap: I think a lot of developers are not able to code up "streaming" solutions to problems. However, streaming can often be useful, even needed, in what are now run-of-the-mill web applications; and wonderfully, we often don't need anything fancier than the tools already being used: we just need to know how to use them. What is streaming? Any situation when you…
In these days of medium data [data that is too big to fit in memory on a single machine, but could otherwise be processed by one], it's important to know what features your programming language offers to help you process data using streaming. Generator functions in Python are one such feature. [For brevity, this post will refer to generator functions as generators ]. Generators vs functions…
Frameworks often hide/abstract parts of HTTP away. I think this is often a bit of a shame: it hides what's possible with HTTP, and so can lead to effects on engineering decisions. This short guide aims to rectify that. It details a few of the most common and useful parts of HTTP, and is aimed for developers with some experience making or receiving HTTP requests. [In this post, the term HTTP is…
One of the beautiful, and maybe even genius, things about the core S3 API: it's just an HTTP PUT to store an object, and an HTTP GET to fetch it. You need a few headers , but that's it. Structure of an S3 PUT request PUT /key/of/the/object HTTP/1.1 r n host: my-example-bucket.s3-eu-west-1.amazonaws.com r n authorization: ... r n content-length: ... r n x-amz-content-sha256: ... r n x-amz-date: ...…
What is the underlying data transformation I need? If it's trivial, or almost trivial, consider writing it yourself. What does it do in my specific case? Consider the actual data transformation you need done, and go through it in the dependency's codebase. Often libraries have extremely generic components to support lots of use cases. You however, may just have the one, and so this genericness is…
When writing Python, sometimes you need to store/manipulate state. There are two typical options: Plain data structures [dictionaries, lists, sets etc.], passed to functions which perform mutation. Instances of classes, i.e. objects, with the state as members, mutated using instance methods. However, there is a third way: A class-like function [I didn't come up with this term], with the state…
Lots of people hate Python's ternary operator. If laid out as: selected_value = value_1 if condition_1 else value_2 if condition_2 else value_3 I can see why: it's hard to see what's going on. However, laid out differently, it's a different story: selected_value = value_1 if condition_1 else value_2 if condition_2 else value_3 You can much more clearly see what the possible values are, and under…
Ideally, changes of behaviour are released incrementally to base your next steps on feedback. But sometimes you think this just isn't possible. Maybe it had been decided that large public behaviour changes must all go live at once; or sometimes planned changes are so large, you might initially think that there is no alternative other than for it all to go live at once, a few days, weeks, months…
AWS S3 is a key/value store, with operations that only operate on a single key at a time. There is no native concept of a folder: the closest thing is a group of keys with the same prefix. These facts mean that in applications that treat S3 as a filesystem, operations on such pseudo -folders, such as a renames or copies, are not atomic: if performed by different users at the same time, corruption…
In this post, I present an implementation of an asyncio read/write lock [also known as a shared/exclusive lock]. Since this post was written, the lower-level class, FifoLock, has been released separately . What is a lock? When you have concurrent tasks, there may be parts of the code that have to be protected from being run by multiple tasks concurrently. A typical example is non-atomic reading…
It is not a requirement to Boto 3 in order to communicate with AWS from Python: you can make requests using any HTTP client, as long as you can work out the correct headers. Here's a function that does just that [with some caveats ]. For example to PUT some data to S3 using aiohttp: Why use this, and not Boto 3? Your application is event-loop based This was my original reason for writing this:…
Can I think of any combination of input where the exception can be thrown? If there is no combination of input that you can determine that can cause the exception, then if somehow it does get thrown, your assumptions on how the program behaves have been violated. Consider letting the error bubbling up, and defering to a general error handler. In a web context, returning a 500 would be a typical…
I recently wrote a small Python application that had to be configurable so it can run differently in different environments. As is often done, I used environment variables for this. However, for various reasons beyond the scope of this post, it was helpful to have structured data stored in the environment: lists, dictionaries, and even lists of dictionaries. So when the application runs, it should…
In an Agile environment, you are likely keeping in mind the following 3 Agile principles when working. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Working software is the primary measure of progress. So if…
There is a "feature" of type-safe language: it is often an effort to use union types . This has the consequence that the developer is given a little push to avoid such types, and consider alternatives. In this post I argue that the simpler, non-union, types and corresponding code are often easier to reason about, and less likely to have bugs. Therefore, even in a in a type-unsafe environment, you…
Am I confident the feature I wrote today works because of the tests and not because I loaded up the application? If no, consider instead writing higher level tests that each test more of the code. I've tested a number of cases manually. Are these cases covered by tests? If no, write tests covering these cases. This may be some combination of higher and lower level tests. Don't worry if some code…
The word monad can be seen as scary, but you don't particularly need to worry about what it means in the general case in order to use the implementations of >>= (bind). For example, the list monad, where >>= is defined as below. ( >>= ) :: [ a ] -> ( a -> [ a ] ) -> [ a ] xs >>= k = join ( fmap k xs ) When evaluating >>= , each element of xs is passed to k , which returns a list, and then the…
Trying to understand functors, I came accross the initially strange fact that functions are functors. Which means, roughly speaking, you can "map", another function over it. So given a function g :: r -> a and another function f :: a -> b you can "map" the second function over the first to get another function, which is defined as the composition of the two. fmap f g :: r -> b fmap f g = f . g…
This post assumes some understanding of and familiarity with Haskell monads. The beauty There are a number of patterns that come up frequently in programming. Some of these are: Combining IO actions, making the result of each available to later ones: IO monad Building up, or reading from, a peusdo-mutable state or configuration: Reader, Writer, and State monads Running a sequence of functions…
Starting in Haskell, I wanted to lean away from do notation, to make sure I knew what was going on under the hood before taking syntactic shortcuts. However, I have found a small stumping block in my quest for de-sugaring. Function application $ and nested do notation results in quite clear imperative-style code. For example, E2E tests using Test.Hspec.Webdriver . main :: IO ( ) main = hspec $…
I have seen multiple developers leap to the conclusion that long files of source code are automatically bad, leading them to separate the code, virtually as-is, into separate files. This post suggests that this view is too simplistic, and other things should be considered before moving code about. So why do developers think long files are bad? My suspicion is that upon seeing a long file, there is…
Coming from an imperative programming background, or working in an imperative code base, it can be tricky to write purer code. However, there is one guideline I try to follow that gently prods me in the "right" direction. Avoid stateful variables Following this guideline leads to a number of things that I think makes the code "better". Stateful variables What I mean by stateful variable is a…
When designing a deployment strategy for your web application or site, keep in mind that browsers do not request a site atomically. They ask for the HTML, and then, some time later, ask for the resources such as scripts and stylesheets. If a deployment is fully atomic, as in only one version of a site is accessible at any given time, this time period can introduce a race condition: A…
A common pattern I have seen is having some default behaviour in a base class, that is sometimes overwritten in subclasses, and sometimes not. class Base ( object ) : def colour ( self ) : return "red" class ClassA ( Base ) : pass class ClassB ( Base ) : def colour ( self ) : return "green" a = ClassA ( ) print a . colour ( ) # "red" b = ClassB ( ) print b . colour ( ) # "green" The usual reason…
This post contains a few recommendations on how to write certain aspects Protractor E2E tests. Specifically how to depend less on the internals of the page. You can't entirely not depend on internals: they all depend some amount of HTML, and currently you have to choose and interact with the elements of the page somehow . However, some internals are better than others. The point of E2E tests First…
Page objects seem to have gained a bit of traction in AngularJS E2E testing, especially since they seem to be officially recommended . In this post I offer a few reasons not to use them. Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should. Ian Malcom, Jurassic Park Reason 1: They obscure behaviour Consider the official example below. var…
Often you want to pass in an extra variable to an existing function def my_func ( a ) : . . . so it supports a new use. You usually want the old use to remain the same, so you add a optional argument with a default value. Often, in Python at least, this default is None . def my_func ( a , b = None ) : . . . So after adding this, you would have two (or more) call sites. # Original call site my_func…
The code below follows a common pattern in impure code. if ( condition ) { doFooImpurely ( ) ; } else { doBarImpurely ( ) ; } If you can extract out common logic from doFooImpurely and doBarImpurely into a meaningful intermediate variable that can be calculated purely, so the code is of the following form const intermediateValue = condition ? getFooValuePurely ( ) : getBarValuePurely ( ) ;…
This post contains the high level steps for the site architecture and deployment strategy for blue-green deployment for a static site hosted on S3 . There are a few things that are required for a reasonable blue-green strategy. No downtime between releases. Atomic deployment. A visitor sees a whole working version from before or after the deployment. This can be tricky because a user doesn't…
There might be times when you have a chain of piped streams where you want to delay a part of the chain until a promise is resolved. You can do this with a function that returns a simple Transform stream. function waitFor ( promise ) { return stream . Transform ( { objectMode : true , transform : function ( file , enc , cb ) { var self = this ; promise . then ( function ( ) { self . push ( file )…
Animating elements between parts of an application can be tricky. The beauty of ng-repeat means we can declare that a list, such as a ul , should represent a data model, such as an Array of data, and the list keeps up to date with whatever changes we make to the underlying model. The tricky bit comes when we want to view not just current state of the model, but transitions between states, such as…
Usually shadowing introduced by prototypical inheritance of $scope is something to be avoided, often the source of bugs cause by not having a dot in models . However, there is a way it can be used to throttle variable changes in templates. Thottling variable changes is something you might want to do to avoid flickering of fast changing variables, so the user has a chance to see each state of the…
It's possible for Angular apps to communicate, where one is running in an iframe of a parent, using standard scope events. This means you can treat an iframe much like a custom directive, responding to $broadcast -ed events, or $emit -ting its own. This technique relies on the child app being able to access the $scope of the iframe element in the parent app. This is possible by using…
Transclusion allows a directive to move clone(s) of the contents of a directive to an arbitrary place in the DOM. Usually this is at the original location of the directive, wrapped in some extra elements, but it doesn't have to be. So why move the element? You might want components to still have access to the original scope, with any models and methods, but for page-layout or CSS reasons be…
AngularJS promises are an extremely powerful tool. They allow you to make a multi-layered and complex system based on asynchronous functions, with error and in-progress notification handling, all without getting into callback hell. This post attempts to explain both creating and using AngularJS promises. It assumes some familiariy with AngularJS, specifically defining and injecting services and…
The Problem Running a basic node.js + express + socket.io setup, I would occasionally get ECONNRESET errors. After a bit of trial and error, I found that this seems to have been caused by running socket.io and express on the same port, port 80. Solution I opened up port 81 on the server, and ran socket.io through port 81. My code to set things up now looks like: var httpPort = 80 ; var…
The problem With an ever-increased user and code-base, with ever increasing amounts of data, we are approaching limits of what any single relational database can handle over at intelligentgolf . Caching output of database queries and/or generated HTML must be part of any solution, as it's clearly innefficient to be re-calculating the same things again and again on every page load. While caching…
A few technical details about this site. Front End A couple of fonts are included from Typekit . and a few Fontawesome icons are used as well. Disqus for comments, as below. Repsonsive by way of a single @media query to change the position of the content relative to its title. At the moment, no HTML5 tags other than the DOCTYPE. It's on the todo list. HTML is also a bit too complicated as in a…
This is a known bug bugs.jqueryui.com/ticket/8873 , but I couldn't find it on a search, so it might be helpful to post here. I was creating a datepicker for a date of birth field, with no default value, and with a yearRange of "c-100:c-1". However, the year dropdown kept on jumping to 100 years before the actually selected year. I changed the yearRange option to "c-100:c" and it then worked fine.