A blog by Danny Hermes; musing on tech, mathematics, etc.
This is cross-posted from the WorkWhile engineering blog . This post describes a data pipeline we built to convert an analytical SQL query into an aggressively cached JSON API with zero load on our database or application. Contents Motivation Core problem Cloud primitives for the win Conclusion Motivation WorkWhile is the …
This is cross-posted from the WorkWhile engineering blog . This post describes a zero-downtime schema change to a high contention table — guided (and greatly accelerated) by ChatGPT. Contents We made the wrong assumption Online migrations and table contention I get by with a little help from my (AI) friend Add new …
It turns out it's quite straightforward to construct an " almost unique" constraint in PostgreSQL via EXCLUDE . I was recently stuck in a jam in our PostgreSQL database during a feature migration. We needed a " partial UNIQUE " constraint and I was surprised to find out that PostgreSQL doesn't support them 1 …
This post was included in Golang Weekly issue 499 . This is cross-posted from the Hardfin engineering blog . The Go standard library uses a single overloaded type as a stand-in for both full datetimes 1 and dates. This mostly "just works", but slowly starts to degrade correctness in codebases where both …
Below, we'll show two facts which help understand the positive integer solutions to the following diophantine equation: 1 + ( n 2 ) + ( n 4 ) = 2 m . 1 + \binom{n}{2} + \binom{n}{4} = 2^m. 1 + ( 2 n ) + ( 4 n ) = 2 m . We'll show that there is a lone positive root and …
This is cross-posted from the Hardfin engineering blog . As a small and growing team, we need to focus our operational efforts as much as possible. Using Tailscale as our VPN is very much aligned with this need. It's simple, with no central control plane to manage or other significant operational …
This post was included in Golang Weekly issue 414 . This is cross-posted from the Hardfin engineering blog . As programming language ecosystems evolve, they learn from each other. When this happens we all benefit from more useful features, patterns, libraries, and tools. A core part of a programming language ecosystem — packaging …
This is cross-posted from the Hardfin engineering blog . At Hardfin, we use AWS RDS for managed PostgreSQL instances to simplify our infrastructure and focus on product. For the most part, RDS is "just PostgreSQL" and acts like the PostgreSQL instances that we run on our development machines. Unfortunately, it differs …
Goodbye I recently made the very hard decision to leave Blend . Today (March 4, 2022) is my last day. I am overwhelmed with emotion thinking about all of the amazing people I worked with and projects I worked on at Blend. I want to thank you all from the bottom …
This post was included in Golang Weekly issue 399 . I recently 1 had the privilege of co-authoring a blog post on the Blend engineering blog with my esteemed colleague Thomas Taylor . I copied this content over mostly as-is from the original post but wanted to preserve it here as well …
For my most recent side project tailsk8s , I've been hacking on a bare metal Kubernetes cluster that uses Tailscale for networking. I had a lot of fun and learned a lot, but the TL;DR is that each Kubernetes node is a Tailscale subnet router and uses the kubenet CNI …
Minor Mismatch When using a database in an application, there are many ways the idioms from the database ecosystem can disagree with the idioms in the programming language used to write the application. The object-relational impedance mismatch is one such example of this, but that is mostly about differences between …
Motivation Over time an application becomes a living thing. When operating a running service, it's crucial to not only ask the question Does the code being deployed work? it's also crucial to ask Can I safely deploy this change given the currently running version of my service? This second question …
Motivation As features are added, changed or deleted, the data model used by an application usually changes as well. For most database-backed applications, this means migrations are needed. With this in mind, the fundamental goal of database migrations: The current database schema should match the currently running code. However, achieving …
I enjoy watching math videos targeted at advanced high school or undergraduate students — even if the topics are far away from research level mathematics. I was recently watching a "quick and easy challenge in algebra" video from SyberMath 1 that used a neat trick to solve the problem. As the …
Building a web application in Go and using PostgreSQL as the database is a real joy. Both of these tools are produced by incredibly vibrant open source projects. These projects represent some of the best attributes of the open source movement. When used in combination, occasionally idioms from Go don't …
In a previous post , I described a use case for customer provided keys with Vault. One of the implications of this was the need for decryption after a bulk data export. In that post, I gave a concrete example of decrypting Vault ciphertext directly with a customer provided key. However …
Recently, I was analyzing some initialization code in Go with a teammate . The value being initialized was meant to be used in concurrent Go, so initialization had some requirement of atomicity. The code essentially boiled down to: func ( t * T ) Start () { if atomic . LoadInt32 ( & t . State ) == Started { return // Early Exit …
Motivation Throughout the Go monorepo we use context.WithValue() to "stash" a global value on a root context. For example ctx = logger . WithLogger ( ctx , log ) // ... later ... log := logger . GetLogger ( ctx ) The implementations for stashing a logger.Log are in the same general form as most context wrapping helpers: type loggerKey …
PostgreSQL Statement Timeout For most applications that use a database, user-facing queries must complete in a reasonable amount of time. In order to ensure a maximum query time, PostgreSQL supports a statement_timeout which will cause a query to be cancelled if it exceeds the timeout: $ psql monsters_inc=> SHOW statement_timeout; statement_timeout …
Contents Motivation "Importing" via Restore Decrypting Exported Ciphertext Key Backup Format Motivation To understand why it's helpful to import external keys into Vault, it's important to understand How and why encrypted data is stored in databases How Vault provides features that aid in encrypting and decrypting data Reasons for data …
TL;DR : Using the ca field to specify custom CAs (certificate authorities) in Node.js is a footgun . It replaces (rather than appends to) the root trust store which can lead to unintended consequences. I've seen this behavior cause outages in production when a third party server does a routine …
This is a story of a brief outage caused by a slightly unintuitive API 1 that has some very sharp corners for the uninitiated. The outage, though brief, was of the "wake up at 4am" variety so the lesson was especially salient. This is not a post trying to tear …
The "obvious" way to write a custom GitHub Action is using Node.js, however it's not the only way. As it turns out, a GitHub Action really just communicates with the "orchestrator" via environment variables (as inputs) and STDOUT (to produce custom outputs). Options When defining an action, there are …
This is a tiny little note that can help with debugging in some situations. We'll use netcat ( nc ) to view the raw data sent to and returned from an HTTP server 1 . Capture a Request We'll run a dummy listener via nc and directly inspect the body of an HTTP …
I recently put on my detective hat and tracked down a bug in network error recovery in a popular PostgreSQL library. Below, we'll walk through the process of bugfinding and iteratively making the feedback loop smaller and smaller. In order to find and fix the bug I Confirmed the root …
In my first summer of graduate school my code suddenly stopped working because Fortran and Python (via pow() in C) do exponentiation differently. Once I debugged and understood the problem, I learned about the highly optimized assembly code produced by Fortran for integer exponents. To give a sample of the …
TL;DR : Running Datadog ( dd-agent ) during local development can help confirm metrics and traces are sent as expected and can help debug when things go wrong. To run dd-agent locally just clone the dhermes/local-dd-agent 1 repository and make run . Being able to quickly iterate with a local dd-agent helped …
Adopting a service mesh like Istio is a huge undertaking. (Let's set aside for this discussion whether it's a good idea to undertake.) A fairly common issue when getting a mesh up and running is misconfiguration. When trying to debug and determine where and how things are misconfigured, the network …
I've been writing a library for running PostgreSQL migrations in Go. One of the primary pieces of advice I keep coming across is Beware of lock queues, use lock timeouts In other words, each migration stage should happen instantaneously (or almost instantaneously). For real-time applications, if a migration runs "for …
Looking Inside a Slice The first thing to realize is that a slice in Go is really just a struct that wraps "header" information about (1) a pointer to the "real" underlying data, (2) the length and (3) the capacity without directly exposing those fields. We can use a struct …
I recently 1 had the privilege of co-authoring a blog post on the Blend engineering blog with my esteemed colleague Austin Poore . Our engineering organization is growing like crazy right now and it's a lot of fun so if any of this post is interesting to you, check out our …
Note : In a docker container, a server can only be available outside of the container / pod if it is bound to the "any host" IP 1 . Binding a server to localhost / the loopback IP 2 will mean the server is only reachable within the container / pod. Consider the following Express …
TL;DR : You can remove a misbehaving pod from a service without deleting it. Use kubectl label pod ... cyberdyne-service- ... to remove a label / labels. Once the labels are gone it will be removed from the Kubernetes service that routes traffic to pods. When a Kubernetes node is misbehaving, it's common …
TL;DR : Prefer inclusion of the protocol in configurable environment variables VAULT_ADDR=https://vault.sandbox.invalid:8200 over VAULT_HOST=vault.sandbox.invalid since this enables targeting a local server, e.g. http://localhost:8200 without any code changes. We utilize sandbox, staging and other siloed environments to test changes before …
In order to run vault locally (I did this because I was on an airplane), first start the server with a known root token export VAULT_TOKEN=root VAULT_ADDR=http://localhost:8200 vault server -dev -dev-root-token-id="${VAULT_TOKEN}" vault version # As a baseline, this is the version of `vault` I am using …
Why? Using app.use('trust proxy', true) is likely too permissive, this post explains concretely why. Example Applications Consider two Express applications index-first.js that uses app.set("trust proxy", true) const express = require ( "express" ); const app = express (); const port = 3000 ; app . set ( "trust proxy" , true ); app . get ( "/" , ( req , res …
I've been load testing quite a bit recently. So far I've been impressed by the wrk project's use of sockets to pour on a ridiculously high amount of load. When I had stretched the system under load past its limits, finding the fortio project was a breath of fresh air …
In order to learn a bit more about how both the TCP and HTTP/2 protocols work, I recently created the tcp-h2-describe reverse proxy in Python. I was excited about some of the insights I was able to glean from the process, in particular a full example tracing a connection …
In my day job at Blend, I write a lot of TypeScript 1 . One great feature of TypeScript is the ability to specify an enum with a finite set of values as a union type : type Coordinate = 'x' | 'y' which then gives compile time checking for values of this type …
First, let me say I'm not posting this to shame any company or language community. Getting to the scale Twitter reached in a short amount of time can quickly make past engineering decisions look foolish in hindsight. But almost always they are decisions made in good faith with the current …
Linear Algebra is an incredibly powerful tool. A joke among mathematicians is that the only way we can solve a hard problem is to boil it down to Linear Algebra 1 . Harnessing the numerical power of Linear Algebra has been done via LAPACK for the last 40+ years. In my …
After five years, I'm excited to say I've finished my PhD . I certainly learned a lot and met a lot of great people along the way. One of the biggest lessons I learned was about myself: I need to say no more often. I continued to say yes, to side …
Caveat : I stopped using a Mac for my development machine way back in the fall of 2014. You probably shouldn't listen to me. What Not to Do The first rule of using Python on any operating system: don't use system Python . The OS relies on that Python, including the packages …
About a year ago, I was reading the Go source for computing log ( x ) \log(x) lo g ( x ) and was very surprised by the implementation. 1 Even three years into a PhD in Applied Math (i.e. numerics), I still managed to learn something by diving in and trying …
Finding zeros of any old function is a common task, and using Newton's method is one of the best tools for carrying out this task. I've even written an old post that used this method. However, Newton's method loses some of it's luster around repeated roots. Consider the iteration x …
While watching today's Seahawks-Vikings game , my wife asked: How did the Seahawks score 9 points? Did they get a field goal and miss an extra point after a touchdown? I had been head down coding and didn't know the answer. I quickly jotted down the possibilities (like solving the coin-change …
A fairly common interview question is What is the "hardest" bug you've dealt with? I've both asked it and answered it in interviews. It's pretty rare that the answer is useful and actionable, but I'll hold off on commenting on the state of the art in tech interviews today. (Usually …
I was recently catching up on articles and videos I'd been putting off and came across a great quote from Elon Musk : A well thought out critique ... is as valuable as gold. And you should seek that from everyone you can, but particularly your friends ... It doesn't mean your friends …
After a happy 3-year run with Blogger , I've decided to revamp my blog with viewers in mind. Why? When I started my blog, I was still learning the ropes as a programmer. I was happy to have a place to share the things I learned, but didn't have much context …