Recent content on Justin Miller
Building a chat application where the server never sees plaintext messages requires careful coordination between cryptographic primitives, real-time delivery, and persistent storage. This post examines Seal, an end-to-end encrypted chat application that pairs LanceDB for zero-infrastructure storage with libsodium for audited, high-performance cryptography. The goal is to make it as easy as…
Streaming data to Kafka from distributed data processing pipelines is a common pattern, but the integration details matter. This post examines a Ray Data sink for Kafka built on confluent-kafka, leveraging the high-performance librdkafka library for efficient distributed writes. The Problem Writing Ray Datasets to Kafka presents several challenges that don’t align cleanly with typical batch…
Vector databases like Turbopuffer are becoming essential for AI applications, but integrating them can be a challenge if careful considerations around batching and performance aren’t made. This post examines an integration that handles the details of distributed vector operations. The Problem Vector databases need specific data formats and batch strategies that may not align well with…
Companies often develop internal libraries that expose inputs and outputs to functions via Pydantic. Whether your internal library is for machine learning, API clients, or data validation services, you’ll often need to convert large DataFrames into Pydantic objects and pass batches of them to libraries. Here’s how to do it efficiently using Ray Data. The Challenge Many in-house Python…
In this blog post, I’ll walk through how to automatically generate documentation for your dbt project using OpenAI —completely integrated into your GitHub Actions pipeline. This process will analyze every dbt model and seed file and generate detailed DBT documentation in Markdown with OpenAI’s GPT 4o LLM that’s committed back to the pull request. Technology Stack dbt: The analytics…
In this blog post, I will demonstrate how the Airflow AI SDK and Ollama can be used to develop Airflow DAGs at no cost locally with Astro CLI. This allows for full end-to-end testing of the Airflow DAG without reaching out to a third-party LLM provider and for a broader range of model selection. The example takes a collection of product names, submits to Ollama a request with a system prompt…
In this post, I’ll walk through fine-tuning Gemma-3 using Hugging Face and the Unsloth library. This article is based off an Unsloth Colab notebook and the consolidated source code is available here: Gemma 3 Training with Unsloth To fully use the provided source code (for model export to HF) you must be logged into Huggingface via huggingface-cli . Technologies Used Hugging Face: Platform…
Introduction Orchestrating distributed workloads efficiently is essential to maintaining performance and cost effectiveness of modern data engineering and machine learning workflows. By combining Astronomer , Ray , and Google Cloud Vertex AI , you can dynamically create and manage Ray clusters to execute jobs and clean up resources afterward. In this post, we’ll work through a process to automate…
Extracting information from text can be a time consuming task, especially when you’re looking for a needle in a haystack. Warren Buffett has had a long and successful (perhaps the most successful?) career in investing and provides a number of anecdotes and words of wisdom in the form the Berkshire Hathaway shareholder letters. In this blog post I will lay out a method of extracting this…
This blog post is a combination of notes and references used in my talk on June 8th, 2017 at ProtectWise. SQL ACID transactions but also scalable, consistent, and HA Raft consensus Features both Linearizability and Serializability but without atomic clocks (Spanner) Currently supported languages: C++ Clojure Go Java Node.JS PHP Python Ruby Rust Aphyr’s Review Definitions Linearizability -…
This blog post is the last of a three part series, this section covers: Operations on Scala Collections File I/O Trivia Operations on Scala Collections It’s possible to compose operations on collections, here are a few examples. scala > val listOfSentences = List ( ''Tis better to have loved and lost than never to have loved at all. #lovedthanlost' , 'It ain't what you don't know that gets…
This blog post is the middle of a three part series, this section covers: Sets Maps (HashMap and TreeMap) Array and List Buffers Queues and Stacks For performance characteristics of the various data structures listed below, please see: Performance Characteristics Sets Sets in Scala are relatively straightforward. There are few interesting bits of trivia around Sets which will be covered in the…
This blog post is the beginning of a three part series that will cover a variety of topics around Scala Collections, this part will cover: Mutable and Immutable Collections Lists Traversable and Iterable Sequence Traits (Seq, IndexedSeq, LinearSeq) Mutable and Immutable Collections Collections packaged in the Scala Collections library can have mutable and immutable variants, generally separated by…
This year I wasn’t able to make it to Scala Days (hoping next year I’ll get to go!). Quite a few of the talks though seemed really interesting and luckily the Scala Days talks were made available shortly after the conference. Here’s my notes from a few of the videos (I’ll add more as I have time). First, here’s a link to all the videos: Scala Days NY 2016 Martin…
Lately I’ve become interested in category theory and it’s application to software engineering. After watching a video titled “Programming isn’t Math” by Oscar Boykin, I decided to spend a bit of time watching videos on libraries out of Twitter and conferences around category theory, Algebird, Storehaus and other libraries. Here’s a list of a few videos along…
Recently I was interested in seeing all the places I’ve taken photos and found a way of extracting exif data (specifically lat/long) from images stored on my cell phone. Here are the steps I used. Generate the CSV file To begin, start by installing exiftool. This can done on OS X by running “brew install exiftool” (assuming you have Homebrew installed). The following will…
A quick post to demonstrate the differences in performance when FileVault 2 is enabled vs disabled. Today I re-installed OS X and ran various benchmarking tools (with and without FileVault 2 enabled). Here are the results: FileVault 2 Enabled: FileVault 2 Disabled: As we can see from the numbers above (I ran them repeatedly to ensure there wasn’t much of a difference between runs), the…