Recent content on Kevin Sookocheff
Any sufficiently advanced technology is indistinguishable from magic. — Arthur C. Clarke At times I feel like Claude Code is this magical black box that I can give tasks to and get working code out of. This post attempts to shine a light on that black box by building our own agent harness around a model. To do this, I will use Anthropic’s API to build a simple code review harness, showing how to…
I work on Workiva’s cloud platform engineering team, where we design and operate Kubernetes and AWS infrastructure supporting more than 300 microservices across multiple AWS regions. The platform spans networking, observability, CI/CD, compliance, and internal developer tooling. Lately, our team has been busy developing and codifying our strategy. I’m not a big fan of reinventing the wheel, and…
I, like many of you, am wrestling with how to best apply AI to my career as a software engineer. As we continue to adopt to AI-assisted coding, there is becoming more and more evidence that we may be limiting the ability of our future selves to own and support software systems. In my last post , I made the argument that by writing this blog by hand, it forced a deeper understanding of the…
This blog has been a part of my life for over fifteen years ago. When I first started, I was inspired by Patrick McKenzie’s article Don’t End The Week With Nothing . This post left a memorable impression on 20-something me who had yet to consider what work really was — trading time and (sometimes) skill for money. You work for a week, collect your paycheck on Friday, and then you and the company…
In an attempt to move more of my music workflow off of my Mac and onto real hardware, I recently bought a set of used Korg Volca synthesizers for experimenting with synthesis and drum programming. I also have an Akai MPK mini plus that I mainly used for controlling Logic Pro that I wanted to repurpose for controlling the Volca synths through the MIDI out on the Akai MPK mini. Since I couldn’t find…
This is an article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. Before the late 1970s, secure communication largely meant symmetric encryption. This included the complexity that comes with maintaining and sharing secrets every time two parties wished to communicate. RSA changed that by showing that encryption and…
This is an article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. Modern cryptography relies heavily on number theory. One of the simplest but most important tools in the number theorist’s toolkit is the Euclidean algorithm. This algorithm, and its extension, the extended Euclidean algorithm, form the basis for…
When deploying microservices on Kubernetes, how many replicas should I run? Intuition suggests that two replicas is enough to give you high availability; if one pod dies, the other keeps the service online. Industry standard, however, suggests running three replicas. Why is this the case? In this post, I will explain reasons to choose three replicas over two, and some best practices based on the…
This is an article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. When the Data Encryption Standard (DES) was introduced in the 1970s, it was considered a solid block cipher. But DES has one major flaw by today’s standards — a small key space. With only 56 bits of key material, DES can be brute-forced with modern…
This is an article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. Last time we explored the Data Encryption Standard (DES) , and how its Feistel network structure worked. We also saw why DES eventually became obsolete: its 56-bit key size was simply too small in the face of modern computing power. To replace DES, the…
This is an article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. The Data Encryption Standard (DES) is based upon the principles of a substitution-permutation network that we learned about in the last post in this series . The design of DES is based on the principles of Feistel ciphers , which are a particular…
This is an article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. A block cipher is a type if cipher that combines an encryption/decryption algorithm that works on blocks of data with a mode of operation that details the techniques to process sequences of data blocks. While many different block cipher algorithms…
CloudFront supports two logging modes: v2 logging and legacy logging . Naturally, you’d expect to use the newer v2 logging—but if you’re provisioning CloudFront with CloudFormation, it’s easy to end up with the legacy setup by accident. If you use the Logging property in your AWS::CloudFront::Distribution , CloudFormation will default to legacy logging , writing directly to S3.…
If you have a Dell monitor and an M-series chip from Apple, you may have encountered flickering issues with your monitor which — at times — can render it completely unusable. The issue appears to be fairly widespread and can affect a number of Dell model and Apple silicon chip combinations, and has been acknowledged by Dell . The flicker is caused by the monitors LCD panel’s Image Compensation…
This is the fifth article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. When discussing encryption, you’ll often hear talk about the strength of a security system as measured in “bits.” For example, systems might advertise their strength as having 128-bits or 256-bits. But what exactly does security measured…
This is the fourth article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. Modular arithmetic is the foundation for asymmetric cryptography like RSA, elliptic curves, or Diffie-Hellman — all of them rely on the properties of modular arithmetic to guarantee security and secrecy. Since modular arithmetic is so important…
This is the third article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. Without randomness, cryptography would be impossible because all operations would become predictable and therefore insecure. — Jean-Philippe Aumasson, Serious Cryptography The cryptographic strength of most systems lies in their ability to…
This is the second article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. A cipher is only useful if it is secure. What makes a cipher secure? Let’s remember the use case for classical ciphers — keeping messages confidential . Caesar’s cipher and Vigenère’s cipher both suffer from a similar flaw, which makes them…
This is the first article in a series on Cryptography for the Everyday Developer . Follow along to learn the basics of modern cryptography and encryption. The best way to begin learning about encryption is by example. And thankfully, there exist many examples of encryption throughout history that we can draw from. One of the earliest well-known examples of encryption is the Caesar Cipher , and we…
In this blog post, we will explore how to use simulated annealing to cluster a Design Structure Matrix (DSM). We will also discuss how this approach differs from the implementation developed by Ronnie Thebeau as part of his master’s thesis. What is a Design Structure Matrix (DSM)? A Design Structure Matrix (DSM) is a compact, matrix representation of a system or project. It is used to model…
At AWS re:Invent 2023, Amazon CTO Werner Vogels delivered a talk on the laws of frugal architecture . While I initially filed away those insights to review later, a year of cloud architecture experience crystallized a fundamental truth: in cloud computing, cost isn’t just a financial consideration — it is a first-class architectural concern through which we should design and optimize our…
If you work with US government entities or corporations in regulated markets the subject of FIPS compliance may come up, especially in the context of FedRAMP authorization. FIPS 140-2 and FIPS 140-3 are a set of cryptographic standards that your application may need to adhere to dictating the appropriate ciphers or cryptographic functions that are in use. It can take a lot of effort to run a…
Most new applications are built as a single codebase and deployed as a monolith until product-market fit or the need for additional scale is required. Such monolithic applications are easy to write and maintain, and allow you to iterate quickly when there are a smaller number of users and the traffic is low. However, as traffic increases you will need to scale the application to handle the…
eBPF is a technology allowing you to dynamically program the kernel using a virtual machine approach. Several development frameworks exist, with the top choice for Rust developers being Aya . Because of the deep integration between eBPF and the Linux kernel, it can be challenging to create a local development environment that works across different development machines. To date, I’ve taken to…
One of the larger hurdles to climb when becoming FedRamp moderate is encrypting all data in transit using FIPS validated encryption modules. This article describes one approach to satisfying this requirement for anyone running a Kubernetes cluster on AWS. For those lucky souls who are unfamiliar with FIPS, it stands for Federal Information Processing Standard (FIPS), which is a joint US and…
This article provides guide to the different developer tools available for writing eBPF programs, covering the core tools BCC , bpftrace , ebpf-go , and libbpf . These tools satisfy a different use case depending on whether you are trying to use eBPF for running analysis scripts, or full-fledged programs, in what programming language you wish to use, and the maturity and feature parity with the…
A common pattern when scaling a SaaS application is to deploy your stack to multiple regions across the globe. Deploying to multiple regions helps distribute the application and data closer to the user, and isolates infrastructure in case of an outage in a single region. If your application is deployed in this configuration, you have the new problem of routing users to the correct region based on…
A short tagline from the book Learning eBPF describes eBPF as a revolutionary kernel technology that allows developers to write custom code that can be loaded into the kernel dynamically, changing the way the kernel behaves. The key word in this phrase is dynamically . eBPF allows you to write custom code that changes the way the kernel behaves without having to implement a kernel module or…
DynamoDB provides built-in support for cross regional data replication using a solution AWS calls global tables . This article shows how to build and run an application in Kubernetes that uses global tables to replicate data between regions. In the event of a regional disaster, a secondary Kubernetes cluster in a secondary region has all the data from DynamoDB replicated locally to continue…
Disaster events are one of the biggest challenges that a software organization can face. Natural disasters like earthquakes or floods, technical failures such as power or network loss, and human actions such as unauthorized attacks can disable an entire fleet of systems, leading to complete failure for a business. To deal with disaster scenarios requires a proactive approach to prepare and recover…
Check out the project on Github . I recently published a new project for repeatable filesystem benchmarking. This code is based on the benchmarks available in the AWS Mountpoint-s3 project and the Flexible I/O Tester , made into a more general purpose utility for benchmarking arbitrary filesystems. The benchmark consists of a mix of read and write workloads, each run for ten iterations. The…
AWS provides four different storage options for your Kubernetes cluster: EBS, EFS, FSx for Lustre, and Amazon File Cache. Each of these CSI drivers has different performance characteristics, depending on your workload. This post quantifies those performance differences using the flexible I/O tester FIO . Note: For an overview of the different CSI options available on AWS, see Picking the right AWS…
Dealing with storage is a core challenge when running complex applications on Kubernetes. While many applications operate just fine using a cloud database or blob storage, some applications have performance or design requirements requiring local storage. Note: For performance benchmarks see [Benchmarking AWS CSI Drivers](<{{ relref “/post/kubernetes/benchmarking-aws-csi-drivers” }}).…
eBPF is a relatively new extension of the Linux kernel that can run sandboxed programs in a privileged context. It is used to safely and efficiently extend the capabilities of the kernel at runtime without requiring changes to kernel source code or the loading of kernel modules. Because of eBPFs tight integration with the networking stack at the kernel level, it is seeing adoption in networking…
I was recently introduced to a simple article on system design: Users are almost always right . In it, a simple rule is stated: When the users keep doing it wrong, the users are right and your system is wrong. This rule reminded me of the concept of desire paths : unplanned trails created by human or animal traffic that usually represent the shortest or easiest route between two places. Desire…
I was recently testing file upload performance, and needed several large files of different sizes to test with. To make the math easier, it was helpful if I had files with round numbered sizes like 10MB, 20MB, or 100MB. After searching around for files of the right size, it turns out the easiest solution is to generate one yourself using the Linux command line. Depending on your needs, you can use…
A JWT (JSON Web Token) is a type of security token that is used to securely transmit information between parties. It is a compact and self-contained way of representing claims to be transferred between two parties. Many API providers require you to generate a cryptographically signed JWT assertion that includes the authorization you are requesting from the server. If the server accepts the…
Application Portfolio Management (APM) draws inspiration from financial portfolio management, which has been around since at least the 1970s. By looking at all applications and services in the organization and analyzing their costs and benefits, you can determine the most effective way to manage them as part of a larger overall strategy. This allows the architect or engineering leader to take a…
Shortly after deploying a new Kubernetes cluster, one of the first things you will likely want to do is collect some metrics and data about how it operates. There are two projects that are typically used for this, and since they are named similarly it can be confusing to know which one you should use and why. This post hopes to clear up any confusion between the Kubernetes Metrics Server and…
At some point in your career you will be asked to manage a project. This can be intimidating, it can be scary, but it doesn’t have to be. We can leverage some time-honoured techniques, and adapt them to the unique approach required for software projects to deliver on time, on budget, and with success. This article is a collection of techniques I’ve learned for managing projects over time, that…
Amazon’s Elastic Compute Cloud (EC2). EC2 is a web service that provides resizable, on-demand computing capacity — literally, servers in Amazon’s data centers — that you use to build and host your software. It’s important to understand that EC2 is a virtual computing environment. In a virtual environment, there is one physical server with all of the necessary hardware — CPU, memory, hard disk,…
I first learned Vim in university and, since then, it has been a welcome companion for the majority of my software engineering career. Working with Python and Go programs felt natural with Vim and I was always felt productive. Yet Java was always a different beast. Whenever an opportunity to work with Java came up, I would inevitably try Vim for a while, but fall back to IntelliJ and the IdeaVim…
Practical distributed applications are deployed into varied environments and execute on a variety of different machines linked together over a variety of communication infrastructure. The physical machines themselves can differ in the number and speed of processors, the availability of random access and stable storage, and more. The communication infrastructure can differ in the available levels…
A central tenet of databases is that any committed data survives a crash or a failure. Write-ahead logging is a fundamental primitive that ensures all changes to data are first written safely to stable storage before being applied. Coupling that with some careful use of sequence numbers and we can guarantee that changes made to a database can survive system crashes. Motivation Let’s start with a…
Every so often I get stuck with a running process that’s using a specific port, preventing me from running some new application that uses the same port. Then I’m left Googling for solutions or rebooting the machine to make progress. But not any more! I’m recording the solution for my future self (and of course for you, dear reader). > sudo lsof -i :<PortNumber> # returns list of processes using…
When people describe progress, they often describe it in terms of a linear progression taking us from primitive to advanced — an idea or invention occurs as a singular event, and somewhere further down the line of time a new idea or invention completely replaces it, relegating the old to the annals of history. This viewpoint is exemplified by traditional worldviews that organize all beings…
In the book “Thinking in Systems”, Donella Meadows dedicates an entire chapter to explaining why functioning systems seem to work so well. In it, she recognizes three characteristics: resilience, self-organization, and hierarchy. Resilience We can use the standard definition from the Oxford English dictionary to describe resilience: re·sil·ience /rəˈzilyəns/ noun the capacity to recover quickly…
Java was the first programming language I was taught at University, and the language I used for the first decade of my career. It continues to be a reliable companion throughout my software development career. Unfortunately, not having developed with Java professionally for several years, I’ve found there are many aspects of the modern Java language that I’m simply not familiar with. To rectify…
One of the core features of modern Java is lambda expressions. Introduced in Java 8, lambdas provide concise syntax allowing the deferred execution of a block of code. Put a different way, lambdas allow us to pass behaviour as a method parameter. When the method executes, the lambda expression is run. This capability is often referred to as behaviour parameterization . Behaviour parameterization…
As a software system scales it becomes sufficiently large that the number of working parts, coupled with the number of working programmers making changes on it, makes the behaviour of the system extremely difficult to reason about. This complexity is exacerbated by the transition of many organizations towards a microservice architecture, as exemplified by the so-called “death star” architecture,…