Recent content in Josh Liburdi on joshliburdi.com
WTF are Semantics? If you spend even a little bit of time reading about or building distributed systems, eventually you come to understand that there are three types of message delivery guarantees: At-most once: Message delivery is not guaranteed. At-least once: Messages delivery is guaranteed with possible duplication. Exactly once: Message delivery is guaranteed with no duplication. Ensuring…
The Trick Aggregate multiple records into a single aggregated record. That’s it. Why It Works The AWS Kinesis Data Streams service bills based on two criteria: the number of shards running per hour (“shard hours”) and the amount of bytes per record put into each stream. Each shard has a maximum capacity (either 1,000 records written per second or 1 MB written per second) and while there are…
I contributed the chapter “Building Loosely Coupled Threat Detection Systems” to Splunk SURGe’s book, Bluenomicon: The Network Defender’s Compendium. The book is a collection of essays from security professionals on a variety of topics, from threat hunting to incident response to cloud security, and it’s “free” (requires contact info). I also have several…
Over on Brex’s tech blog I wrote a post announcing the release of Substation. The original text is copied below. Introduction We are thrilled to publicly announce the release of Substation, an open source toolkit for creating highly configurable, no maintenance, and cost-efficient data pipelines. Substation solves a problem that every security team has, but few may recognize - the need to…
Preamble In October 2019 I joined the Splunk Global Security organization to build Splunk’s internal threat hunting program. Over a few months we went from an organization with no defined hunting program to one that can do full-scale, high-value hunting. This post describes how individual hunts can be structured to maintain focus and avoid “rabbit holes.” Before diving into this…
Preamble In October 2019 I joined the Splunk Global Security organization to build Splunk’s internal threat hunting program. Over a few months we went from an organization with no defined hunting program to one that can do full-scale, high-value hunting. This post describes the metrics used to evaluate a hunting program. Measuring Success Before we operationalized our hunting program, I knew…
This post is additional content for my presentation at 2019’s BSides SF conference that describes the decision to not build Strelka with a microservices architecture. Here’s my take on microservices: they’re for the 0.01% of application owners. Microservices, like any system architecture model, have advantages and disadvantages; for the majority of systems (including Strelka), I…
This post describes the process of building a secure packet capture (PCAP) retrieval service for Google’s Stenographer using gRPC and Python. The service allows clients to securely request PCAP from a server running Stenographer and stream the PCAP back to the client. Components of Stenographer Stenographer contains several utilities (stenotype, which writes packets to disk; stenocurl, which…
I contributed the chapter “Hunting for Command and Control” to Sqrrl’s book, Huntpedia: Your Threat Hunting Knowledge Compendium. The chapter was originally a blog post on the Sqrrl website and was later reused in the book. I don’t remember when the book was released, but I think it was sometime in 2018 (after I left the company). I may also have a few physical copies if…