Recent content from Dan Palmer
There’s a disconnect between the experiences different people are having with LLMs. Some find them transformational, others find them incapable. Partly this is down to skill, partly this is down to a dislike of the technology and it’s social implications, but I think there’s a further issue. This post is not trying to be pro-AI or anti-AI, I think the social issues are real, I…
It’s uncontroversial to say that the code we write should be simple. In a discipline that is all about grappling with complexity, keeping things simple is critical to software development efforts, team productivity, scalability, and maintainability. Despite well known satirical counter-examples such as Enterprise FizzBuzz , simplicity is a goal for most software engineers. But what is…
Two new “AI” devices have just launched, the Humane AI Pin , and the Rabbit R1 . Both have received broadly negative reviews , criticising a lack of features, alongside the classic issues with modern AI systems – hallucination and confidently incorrect answers. The main distinguishing factor between the two devices has been price. Where the AI Pin sells for $700 with a $24 per month…
There’s a lie at the core of the SaaS ecosystem. In such a wide ranging category of services there are few commonalities, but arguably the main differentiator between SaaS and more traditional businesses is the notion of self-service. A customer can sign up, provide an email address and credit card, and be successful with the product. This is increasingly proving to be a lie, and customers…
A classic part of the NoSQL sales pitch is that SQL JOIN s are too expensive and don’t scale, and a classic response is to point to big websites running smoothly on SQL databases. The reality, as always, is a bit more complicated than that. Types of scale When engineers talk about scale, they’re almost always referring to some sort of usage scale, but even this is not always clear.…
Code Ownership is the practice of assigning explicit owners to areas of codebases. Before Google I worked at small companies where it’s easy to know who should review each code change, but that doesn’t scale far. Even in a team of 10 it wasn’t always obvious who knew an area of code the best, and it was certainly less clear for new starters. Various tools have been developed to…
At Thread we went through several iterations of Search, evolving the technology as we evolved the business and our understanding of what our customers wanted. Later stages went beyond my naive understanding of search at the time, and may prove useful inspiration to others. Before we dive in, some clarification of terms. For us, search meant free text entry that generated product results, whereas…
In an attempt to self-host a low-cost fediverse node, I started with GoToSocial , but later decided to switch to Mastodon for better compatibility. This transition presented some challenges and got me thinking about whether existing web frameworks are well designed for linked data services. Activity Pub, the underlying protocol for the fediverse, necessitates storing URIs to resources on other…
I’ve just started using Raycast, an application launcher for macOS. Like every other launcher before it, it does a lot more than just launch applications, and most of that functionality comes from extensions. Also like several other launchers before it, I decided to have a go at writing an extension and see what the process is like. Back in the mid-2000s I was an avid user of Quicksilver. It…
This is not a tutorial on how to write your own task queue, but rather an attempt to convince you that you should write your own. What’s a “task queue” in this context? For the purposes of this post, a task queue is a system for performing work out of band from a user interaction, often at some later time. Typically this is a core component of many web apps, and is used for…
At Thread I’m involved in hiring engineers for frontend, backend and iOS roles. One of the things I have become more aware of as I have gained experience in hiring and interviewing is how my biases affect the outcomes of interviews. This is something I’m always trying to improve – to understand what biases I have, to mitigate their effects – and in the process I have found a mental…
A mental framework for library design For those with plenty of experience managing complexity in large complex codebases, this post will likely be nothing new. However many open-source libraries, frameworks, and tools make mistakes in how they handle cross-cutting concerns and end up being difficult to use as a result. I’m no stranger to this, and have several times found myself unsatisfied…
There’s a common theme in software engineering communities of software that’s too complex. Slack and other Electron apps are frequent targets – why do we need yet another “web browser” using 2GB of RAM when IRC worked perfectly well? While I can empathise with the performance issues, the question often betrays a misunderstanding of the problem being solved or the target…
Information Exposure Vulnerability with Django and Memcached On Wednesday April 29th, Thread started experiencing a partial outage of our main backend service. We traced the issue down to the existence of malformed Memcached keys and corrected the issue on thread.com . Along the way we suspected that this could be exploited on some Django sites using Memcached to cause private data exposure –…
Last year I bought a copy of Scythe from publisher Stonemaier Games , based in large part on the art. I was very happy with the art and enjoy playing the game, but what I found even more satisfying was the design of the rulebook, the iconography, and the use of physical tokens to re-inforce processes used throughout the game. This week I bought Wingspan from the same publisher, again based in…
During GitHub’s annual product announcement on Wednesday, new features to edit code online were demoed. At one point a code snippet was shown from a toy web-app, written in Javascript using the Express server library. Here’s the code sample… After the announcement, David Heinemeier Hansson (DHH), the creator of Ruby-on-Rails gave his thoughts on Twitter. Is this really what…
I’m an armchair space enthusiast – I like to watch new launches but I know very little about rockets. Recently there’s been a lot of renewed interest in landing on the moon which is very exciting, and also a lot of press coverage of NASA’s Commercial Crew programme returning manned spaceflight capability to the United States. Between these two advances, there have been many…
I’ve been reading this extensive breakdown by Bethany McLean and Peter Elkind of Enron’s collapse after a colleague’s recommendation (based on my enjoyment reading Bad Blood ). I found it fascinating how much of the classic image I have of corporate greed stems from the relatively recent collapse of Enron in 2001. Since I just missed the Enron collapse, being about ten years old…
Not long after a recent one to one with my manager, discussing how we could improve our incident response process in engineering at Thread, I returned to my desk to find a copy of The Checklist Manifesto that he had kindly got for me. This is less of a book review and more of some highlights that I wanted to pull out from the book. Going into it, I had already read about the effectiveness of…
Last month at their annual Worldwide Developers Conference (WWDC), most interesting announcements was Sign in with Apple . Built to compete with Facebook and Google’s single-sign-on (or social sign-on, SSO) offerings, Apple’s SSO will eschew control over the data and analytics that its competitors seek in favour of a privacy preserving design intended to advance Apple’s…
GraphQL’s type system allows us to make many invalid states impossible to represent, which improves the usability and reliability of our APIs. Two features of the type system that contribute significantly to this are Interfaces and Unions, however they can be used to address similar design considerations so it’s not always obvious which is the right option. In this post we’ll look at several…
Four of us from the Thread engineering team went to PyCon UK again in September for the third year running, and I was lucky enough to have my talk selected. At Thread we use Django for the backend of the main site which has grown to over 350 “apps” , and various members of the team have used the framework since not long after the initial public release. I’ve learnt many tips, tricks, and best…
At Thread one of our core beliefs is that technology allows for great change. This is important to our product, but it’s also important to how we work internally. Because of this way of working, we try to represent everything in data—products, measurements, styles, suppliers, locations in our warehouse, support ticket resolutions, and many more things that you’d never even think about. All of…
Following on from my previous post about Haskell web frameworks, I’ve dived into making a non-trivial web application, with type-safe database access.
A quick overview of 5 of the popular Haskell web frameworks, and what they can do to improve the state of web development.
Qualys have become well known in the recent crop of SSL and TLS vulnerabilities as a first-responder with automated testing and validation, but scoring top marks on their SSL Labs test can be difficult. I explored what was required to score full marks.
RESTful APIs are a popular thing, but is anyone really doing it properly? This post highlights some common flaws in RESTful APIs, and explains why it’s important that we improve them beyond the current standard.
MongoDB, the company behind MongoDB published a new whitepaper this month, about ‘quanityfing business avantage’. As I’ve recently completed a research project at university where I critically analysed the design decisions taken in MongoDB, I thought it would be interesting to see how the company sells it. I’ll write about my research sometime, but for now, I’m going…
Last Wednesday, Stripe started their 3rd Capture the Flag competition. As a provider of online payment services, security has been critical to them, so over the last few years they have run two CTFs based around hacking and securing systems. This year they chose a different subject: distributed systems. The CTF happened over the course of the last week, and consisted of 5 levels of supposedly…
If I mentioned that I like C, C++ or Python to other students on my course, or colleagues, there would be no reaction. There are things you can criticise about each one, but they are all very safe bets. When I tell people that I enjoy writing Objective-C however, they are confused and often quite hostile towards the language. I am by no means an Objective-C expert, but I’ve been thinking…
I’m writing this on the train home from Rewired State ’s latest event: National Hack the Government Day 2013 ( event summary page ). It was another great event with the same friendly atmosphere that goes along with so many (especially Rewired State’s) developer events. My friend Elliot and I won in one of the categories, and so this post is mostly about what we did, how we did…
The security of GitHub’s website and systems has been the focus of a fair amount of news in the industry over recent months, this is an account of my experience finding a vulnerability, getting it fixed, and also my opinions on the recent ‘mass assignment’ exploit that was publicly demonstrated on GitHub. This was the first security issue I noticed in the wild, a problem with how…