from Andy Boothe
Depending on your perspective, Reddit is either one of the last great online communities or a petri dish for the internet’s most cutting-edge bots. Or, maybe, both. One thing that’s become increasingly hard to ignore is how much conversation on Reddit is shaped by accounts that don’t really read like people. The scale of the … Continue reading "Subreddit History Is a Surprisingly Good Lie…
One of my favorite conference talks of all time is Statistics Without the Agonizing Pain, by John Rauser, who was at that time head of data science at Pinterest. In this talk, he explains the statistical argument underpinning the Student’s t-Test in simple, approachable terms using an unforgettable example involving mosquitoes and beer. It’s about … Continue reading "(More) Statistics Without the…
I just tried to use one of the Amazon Nova models in AWS, and I got the following error message: After some tooling around, I noticed that the error message was listing us-west-2 in the error message, even though I was working in us-east-2. This led me down a little rabbit hole that ultimately led … Continue reading "Having Trouble with Bedrock Errors? Check Cross-Region Inference." The post…
I’m using Google BigQuery to do some ETL, and have found OpenAI’s products to be enormously helpful for the task. A new client recently asked for some assistance working with Sunshine Act data. Since I expect additional asks about this data set to come in over time, rather than fuss with the generic UI, I … Continue reading "AI is Getting Really Useful for SQL" The post AI is Getting Really Useful…
Rapier is a code generation companion library for Google Dagger. It is designed to reduce boilerplate by generating Dagger modules for fetching configuration data from common sources. If you’ve ever written Dagger code like this: Then Rapier can help! You can find the official release on my consultancy site. The post Introducing Rapier first appeared on data, code and conversation .
While there are many examples of Jackson serialization to JSON, there are comparatively few resources of Jackson serialization to CSV. Following is an example of working with a TSV-formatted dataset from the ground up, starting with creating the model object, building code samples for parsing CSV to Java objects using Jackson, writing Java objects to … Continue reading "Jackson CSV Serialization…
Need a regular expression to recognize the 249 officially-assigned codes from the ISO 3166-1 alpha-2 and alpha-3 standards? Here they are! ISO 3166-1 is a living standard and changes over time. This page is current as of November 11, 2024. ISO 3166-1 alpha-2 Codes This regex will match all 249 officially-assigned codes from the ISO … Continue reading "Regex for ISO 3166-1 Country Codes" The post…
As convenient and powerful as AWS API Gateway is, it’s not without its quirks. Here are a few lessons I’ve learned the hard way that can help you avoid some common pitfalls when working with AWS API Gateway. 1. Need a Custom CloudFront Distribution for Lambda@Edge If you’re using Lambda@Edge, you may find that API … Continue reading "AWS API Gateway Gotchas & Lessons Learned" The post AWS API…
There’s a lot of discussion about how to use ChatGPT to generate tests for code. Another interesting use case I’ve seen fairly little coverage of is generating DTOs from JSON. Here is an example with the prompt I’ve put together applied to JSON from the manifest of a Distributed Map Run. This was ChatGPT 4o … Continue reading "Generating Java record classes with Jackson Annotations to map JSON…
As part of updates to arachn.io, I’ve started tinkering with object detection machine learning models. During my experiments on AWS SageMaker, I found that AutoPilot does not support object detection models, so I had train using notebooks. As a result, I hit some “gotchas” fine-tuning TensorFlow Object Detection models. While this notebook works a treat … Continue reading "AWS SageMaker Object…
Java has a rich set of tools for processing images built in to the standard library. However, it’s not always clear how to use that library to perform even simple tasks. There are already lots of great guides out there for working with images once they’re loaded… but what can Java do without ever loading … Continue reading "Efficient Image Metadata Extraction with Java" The post Efficient Image…
Because Java 21 virtual threads are very cheap — per JEP Cafe, they are about 1,000 times faster to launch than a platform thread, and use about 1,000 times less memory than a platform thread — they should never be pooled. As a result, ThreadLocal variables are unlikely to be useful in applications that use … Continue reading "Yes, Java ThreadLocalRandom is Safe to Use with Virtual Threads" The…
When writing code or testing, it’s useful to have valid examples of whatever you’re working with for sanity checks. I need some example JWTs with accompanying private JWK and couldn’t find any, so I minted a few. I’m posting them here in the hopes they’re useful to others, too. These keys were created using a … Continue reading "Example JWT with Private JWK" The post Example JWT with Private JWK…
I’m porting Jonas Wagner‘s excellent smartcrop.js, which analyzes an image to recommend a good crop, from JavaScript to Java. When porting code, I always try to understand it along the way so that if (when) there are bugs, I’ll have an idea of where the bug might be. I used ChatGPT 4 to untangle a … Continue reading "Reverse Engineering with ChatGPT: An Example" The post Reverse Engineering with…
Based on the results of this JMH benchmark, string operations are faster than regex operations in Java, even if the two operations are logically the same, like String.equals and Pattern.matcher/Matcher.matches. In some cases, much faster. The Benchmark The measurements include the following benchmarks: The Results In other words: Conclusions If speed matters, then prefer to … Continue reading "In…
The source code never lies. The post Apache HTTPComponents HTTP Header Names are Case Insensitive first appeared on data, code and conversation .
WebFlow is an outstanding website design, development, and hosting platform. The WebFlow API provides webhooks for a variety of important events, and it signs its webhook requests to allow users to validate webhook requests, and therefore confirm that requests actually come from WebFlow. The below code sample shows how to verify WebFlow webhook requests in … Continue reading "Validating WebFlow…
I couldn’t find an example payload for CloudWatch Logs data delivered by a Kinesis Firehose stream to a Lambda function URL, so I set one up. Sample payloads appear below, as well as a walkthrough of the process I used to create the data stream. Use Case I need to deliver API usage data to … Continue reading "Sample AWS Kinesis Firehose CloudWatch Log HTTP Endpoint Payload" The post Sample AWS…
Modern serverless applications on AWS are complex with a lot of moving parts. Mapping a developer workflow onto those applications can be difficult. This article discusses the developer workflow I have developed for complex serverless applications at aleph0, with example CloudFormation template and GitHub Action snippetes to illustrate the concepts. Goals for Developer Workflow Assumptions ……
AWS Lambda is Amazon’s FaaS product. Pound for pound, it’s one of the best serverless computing products on the market. Easy to use, inexpensive to run (among FaaS offerings), and with compelling features like Layers, Extensions, and SnapStart, Lambda is a rock-solid choice for building serverless architectures. However, its managed nature cuts both ways. The … Continue reading "The Lambda…
As part of optimizing the HumanGraphics product, I am investigating different cloud architectures and their tradeoffs. Documenting my current “default stack” for a new webapp with compute (like an API) seems like a good starting point. Here it is: Experienced cloud engineers and architects should should look at this diagram and agree that it’s at … Continue reading "Unpacking My Default AWS Webapp…
I’ve added a new custom base image for Java 21 on Lambda to complement the community base images already available for Java 17, Java 18, Java 19, and Java 20. You can find the images on the ECR Public Gallery and DockerHub and the source code on GitHub. Java 21 is an LTS release with … Continue reading "Community-Managed AWS Lambda Base Images for Java 21" The post Community-Managed AWS Lambda…
By standing on the shoulders of giants (I’m looking at you, Mark Sailes), I was able to turn around an AWS Lambda Custom Runtime for Java 21 on Java 21 launch day. Instructions to use it are in the repo’s README. All files required to create your own function are in the releases, so no … Continue reading "Java 21 Custom Runtime for AWS Lambda" The post Java 21 Custom Runtime for AWS Lambda first…
HumanGraphics is a REST API that converts names, locations, and faces into structured data and demographics estimates. Sound interesting? Take it for a test drive! And it launched today! Check out the website for a special launch offer! Name Parser HumanGraphics’ name analysis endpoints take a full (i.e., “Andy Boothe”) or faceted (i.e., first name: … Continue reading "Announcing HumanGraphics!"…
The excellent PostgreSQL database supports the following syntax for the common upsert SQL pattern: With this syntax, you can insert a new row with the given values, or update the existing row to the given values, using the row’s primary key (or other indexed key). Very handy! But did the resulting upserted row insert or … Continue reading "Postgres Upsert: Created or Updated?" The post Postgres…
The writing system that people use can actually tell you a lot about the person: where they are, what language(s) they speak, and so on. The two main standards for writing systems are ISO 15924 and Unicode. They are closely related to each other, but not every ISO 15924 entry has a corresponding Unicode script, … Continue reading "ISO 15924 Codes to Unicode Scripts… and Back Again!" The post ISO…
I needed a dataset of popular names by country for testing, but I couldn’t find one that had everything I needed. So I made my own! Need a free dataset of popular names by country, including CJK and RTL examples, plus romanization and counts, all for a boatload of countries? Me too! Keep reading to … Continue reading "Popular Names by Country Dataset" The post Popular Names by Country Dataset…
Need a regular expression to recognize the 50 US state names or USPS abbrevations? Here it is! State Names and Abbreviations This regex will match all 50 US state names (e.g., Texas, California) or USPS abbreviations (e.g., TX, CA) case insensitively. If you want the regex to match case sensitively, then simply remove the leading … Continue reading "Regex for 50 US States" The post Regex for 50 US…
A new version v15.0.1 of my emoji processing library, emoji4j, for Java 8+ just dropped. Here are the updates: There is also now a Cookbook in the emoji4j wiki to help users solve hard or common problems with emoji4j. Enjoy! The post emoji4j v15.0.1 Released first appeared on data, code and conversation .
If you ever need to convert “plain” social media text like: Hey @importantguy, check out my project https://www.mycoolproject.com/ #PrettyPlease into “rich” social media text like: Hey @importantguy, check out my project https://www.mycoolproject.com/ #PrettyPlease Then you need a social text parsing library. The industry standard is twitter/twitter-text, but it doesn’t work for everything. (For…
I’m a software architect, software engineer, data scientist, and analyst by trade. So far in my career, I have faced more and different problems than the average bear. This is a list of some of the techniques I have used to crack some of the harder nuts in my career. They’re pretty simple, so don’t … Continue reading "Techniques for Solving the Toughest Problems" The post Techniques for Solving…
I’m using AWS Step Functions to do some complex orchestration of services that could span more than 25,000 state transitions and exchange data sets larger than 256KB, so I’m making heavy use of the new distributed map feature. It definitely makes things easier than the old everything-is-a-child-execution approach! However, the ResultWriter field is not particularly … Continue reading "AWS Step…
The excellent vector database Pinecone has a very useful API, but client support is sparse. While the API and its clients are in theory based off of an OpenAPI spec, no one seems to be able to find it. But I needed an OpenAPI spec for the API, so I reverse engineered one (read: copy and pasted from the documentation). … Continue reading "Community-Managed OpenAPI Spec for Pinecone API" The post…
I’ve added a new custom base image for Java 20 on Lambda to complement the community base images already available for Java 17, Java 18, and Java 19. You can find the images on the ECR Public Gallery and DockerHub and the source code on GitHub. All the new features in Java 20 are in … Continue reading "Community-Managed AWS Lambda Base Images for Java 20" The post Community-Managed AWS Lambda Base…
I'm in the process of developing the marketing website for Arachnio. Being an API product, I need to embed code in some of the site's pages and blog posts. The website is built on WebFlow, which I have found to be generally outstanding, but it does not support inline code or code blocks out of the box. Here's how I got code blocks with syntax highlighting working in WebFlow. The post Adding Code…
In my opinion, JDBI is simply the best database access framework for relational databases available for Java today. My new library, JDBQ, is essentially a port of JDBI from relational databases to Google BigQuery. My opinions on ORMs are known, so you can probably guess JDBQ isn’t an ORM. Rather, it is an opinionated framework … Continue reading "JDBQ: A Database Access Framework for Java +…
If anyone needs a regular expression for matching single Emoji in BigQuery/RE2, this is a good place to start:…
I’ve added a new custom base image for Java 19 on Lambda to complement the community base images already available for Java 17 and Java 18. You can find the images on the ECR Public Gallery and DockerHub and the source code on GitHub. If you’re as excited to try Java 19 as I am, … Continue reading "Community-Managed AWS Lambda Base Images for Java 19" The post Community-Managed AWS Lambda Base…
I’ve added a new custom base image for Java 18 on Lambda to complement the community Java 17 base images already available. You can find the images on the ECR Public Gallery and DockerHub and the source code on GitHub. If you’ve been waiting to adopt Java 18 until there was Lambda support, then these … Continue reading "Community-Managed AWS Lambda Base Images for Java 18" The post…
I’ve just released my first commercial API, arachn.io. It’s a simple thing, focusing on performing link unwinding and data extraction affordably at scale. It’s also dead simple to use. Check out the Free Forever plan to try it out yourself! The post Announcing My First Commercial API, arachn.io! first appeared on data, code and conversation .
Thinking about using an AWS Lambda function using the excellent awslabs/aws-serverless-java-container as an AWS Application Load Balancer (ALB) target? There are some configuration gotchas you should know about! ALB Lambda Target Configuration When you set up an ALB target group, you can configure the group using Target Group Attributes. For Lambda functions, the attribute…
I’m (finally) upgrading several projects to Java 17, the current LTS version. I like to deploy my Lambda functions as container images because it keeps the devops simple — everything is a container! — but there still isn’t an officially-supported base image Java 17, even though it’s been out for almost a year. So I … Continue reading "Community-Managed AWS Lambda Base Images for Java 17" The post…
I just released the first beta version of litecene, a Java library that implements a common boolean search syntax for full-text search, with its first transpiler for BigQuery. Litecene Syntax Litecene uses a simple, user-friendly syntax that is derived from Lucene query syntax that should be familiar to most users. It includes clauses for term … Continue reading "Litecene: Full-Text Search for…
This is the third post in a three-post series covering the fundamentals of software performance analysis. You can find the introduction here. You can find Part II here. You can find the companion GitHub repository here. Part II covered the process of using a profiler (VisualVM) to identify “hot spots” where programs spend time and … Continue reading "Fundamentals of Software Performance Analysis…
This is the second post in a three-post series covering the fundamentals of software optimization. You can find the introduction here. You can find Part I here. You can find Part III here. You can find the companion GitHub repository here. Part I covered the development of the benchmark, which is the “meter stick” for … Continue reading "Fundamentals of Software Optimization Part II — Optimizing…
This is the first post in a three-post series covering the fundamentals of software optimization. You can find the introduction here. You can find Part II here. You can find the companion GitHub repository here. The introduction motivated why software optimization is a problem that matters, reflected on the fundamental connection between the scientific method … Continue reading "Fundamentals of…
This is the introduction to a three-post series covering the fundamentals of software optimization. You can find Part I here. You can find the companion GitHub repository here. Performance is a major topic in software engineering. A quick Google search for “performance” in GitHub issues comes up with about a million results. Everyone wants software … Continue reading "Introduction to Fundamentals…
The OpenAPI Generator is a wonderful bit of tech. It allow users to create an OpenAPI spec, and then generate client and server code from it in a variety of languages and platforms. I use it to generate DTOs for all my APIs, and to generate service interfaces to keep my client and server in … Continue reading "OpenAPI Generator Template Customization Example" The post OpenAPI Generator Template…
As part of a side project, I ended up producing a large dendrogram. Roughly 22,000 items were clustered, which produced over 44,000 nodes! Plotting such a large dendrogram is tough. After trying to convince some tools to render a dendrogram this large in a way that is useful, I decided to render it as a … Continue reading "A Dendrogram Layout in Gephi" The post A Dendrogram Layout in Gephi first…
I was investigating the feasibility of putting natively-formatted Twitter data into BigQuery, and got pretty far along the way before deciding to go another direction. I found the schema in the twitter-for-bigquery project to be incomplete for my needs, so I made a new schema of my own. I’m making the schema available here in … Continue reading "BigQuery Twitter Schema" The post BigQuery Twitter…