More than a decade of technical ramblings
The practice of writing code has been changing fast. Really fast. I–and a large portion of the industry–write code in a completely different way from a year ago. And by that I mean that I largely manage a sophisticated coding machine that does the work. Writing code is now a process very similar to some of my previous experience as VP of Architecture at Community : I help co-design an architecture…
At Mozi , we needed to connect a new Elixir Phoenix LiveView app to an existing Go backend. This is how we did it. Background We have a backend built in Go, which is fully-evented along the lines of the patterns we used at Community, described in a previous post . In order to support all of that, we have some hefty internal libraries and existing patterns, and get a lot for free by building on top…
If you work with Protocol Buffers ( protobuf ), you can really save time, boredom, and headache by parsing your definitions to build tools and generate code. The usual tool for doing that is protoc . It supports plugins to generate output of various kinds: language bindings, documentation etc. But, if you want to do anything custom, you are faced with either using something limited like…
This post covers how to get a personal Kubernetes (K8s) cluster running on an instance with a public IP address, with HTTPS support, on the cheap. I’m paying about €9/month to run mine and without VAT in the US you will pay less. This post is not how to build or run a large production system, but some of the things you learn here could apply. We’ll get a cloud instance, install Kubernetes, set up…
Distributed systems pose all kinds of challenges. And we’ve built them in the web age, when the tech of the wider Internet is what we use in microcosm to build the underpinnings of our own systems. Our industry has done somersaults to try to make these systems work well with synchronous calls built on top of HTTP. This is working at scale for a number of companies and that’s just fine. But if you…
At Community.com , we had a problem where a bunch of workers needed to pick up and process a large amount of data from the same DB table in a highly scalable manner, with high throughput. We wanted to be able to: Not require any coordination between the workers Be able to arbitrarily change the amount each worker would pick up Not have to shard the table I came up with a solution that has been in…
Apache Spark is a perfect fit for processing large amounts of data. It’s not, however, a perfect fit for our language stack at Community . We are largely an Elixir shop with a solid amount of Go, while Spark’s native stack is Scala but also has a Python API. We’re not JVM people so we use the Python API—via the Databricks platform. For most of our work, that’s just fine. But, what happens when you…
I have been lucky enough to be healthy so far and have been using lockdown time to get back into electronics and microcontrollers, a hobby that is very compatible with being stuck at home. I have a bunch of Arduino, STM32, ESP8266, and ESP32 boards, but something was calling me more to the retro side of things. My first computer was an Apple //e and I learned to do some 6502 assembly when I was a…
A kernel “bug fix” that happened at the end of last year may be killing the performance of your Kubernetes- or Mesos-hosted applications. Here’s how we discovered that it was affecting us and what we did about it at Community.
We needed a specialized load balancer at Nitro . After some study, Mihai Todor and I built a solution that leverages Nginx, the Redis protocol, and a Go-based request router where Nginx does all the heavy lifting and the router carries no traffic itself. This solution has worked great in production for the last year. Here’s what we did and why we did it. Why? The new service we were building would…