Recent content on adrianhesketh.com
Setting up a NixOS remote builder for the M1 Mac If you want to build a Virtual Machine disk image, or Docker container using Nix on an M1 Mac, you’ll need to use a remote builder that’s running Linux. Virtual machines We’ll set up two Linux machines, one for ARM64, and another for x86_64, using UTM. ISO Download the Minimal NixOS ISO for both ARM64 and x86_64 from [0].…
DynamoDB diagrams from text I’m working on a talk where I need to create some single table design diagrams. The sort of thing that you can create with the AWS NoSQL Workbench [0]. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html [0] However, I need to create a lot of diagrams with different data in them, and it’s time consuming to modify the data in that…
DynamoDB Stream to Lambda Filtering With Go CDK It’s been bothering me that I have a Lambda function set up to process data from DynamoDB Streams that doesn’t use Stream event filtering. All of the filtering was done in Lambda itself, wasting precious CPU time. Since v2.42.0 [0] of CDK it’s been possible to use Lambda function filtering, but I hadn’t looked into it.…
go-sqlite3 on AWS Lambda with CDK On a recent project, I chose to use sqlite as an embededed database engine for running SQL statements written by a data team as part of an API. By using sqlite, I could horizontally scale the excution of the SQL statements across multiple execution environments instead of centralising traffic onto a database server cluster like RDS running Postgres. sqlite is also…
Alerting on AWS Security Hub Notifications with OpsGenie In my previous post [0], I wrote about meeting CIS AWS Foundations using Terraform and some scripts. /2022/11/02/meeting-cis-aws-foundations-requirements/ [0] However, many of the CIS benchmark controls enforce alerting on suspicious or unusual actions, for example, using the root user, or changing firewall rules. I use CDK to configure all…
Meeting CIS AWS Foundations Benchmark requirements The CIS AWS Foundations Benchmark is a set of best practices that are commonly adopted by organsiations. Using the CIS Foundations Benchmark allows teams to understand and improve their security posture. The full list of “controls” that make up the benchmark are available in AWS documentation.…
Migrating Go and Node.js Fargate tasks and Lambda functions to Graviton ARM processors with CDK I’ve been using AWS’s Graviton ARM processors for personal projects for a while. They’re cheaper, use lower power chips, and might be better for the environment [0]. https://davidmytton.blog/carbon-footprint-laptops-vs-servers-intel-vs-arm/ [0] But… can I actually use them for…
Process for creating a React / Next.js page I’ve recently been working with a team that was new to TypeScript, React and Next.js to build a Web application and found that one of the challenges the team faced was understanding the workflow of how to design the page, break the process down into tasks, and test the outputs. In this post, I’ll cover the process I use. Each of the headings…
From linear to binary search This week, I’ve been playing around with a parsing library I wrote, working out how how I can rewrite it to make (hopefully sensible) use of Go generics. It has an Input which maintains an Index, and you can Peek characters, but also Take characters which moves the Index - i.e. moves a pointer through the string. You use it by building up bigger parsers from…
Backup Github repos to S3 Application and infrastructure code is often the result of months or years of combined effort from a team, costing a large amount of money to create. It makes sense to keep a backup of this digital asset, in case of accidental (or malicious) loss. Years ago, IT teams would take responsibility for backing up the on-premises SVN/SourceSafe/Mercurial/Git servers to tape, and…
Creating a VPC with CDK It’s one line const vpc = new ec2.Vpc(this, 'VPC', {}) Why is this a blog post then? The default configuration is usually not sufficient to pass security audits, since it doesn’t follow AWS best practice. This post explains what else is required. Add an isolated subnet The default configuration for VPCs is sensible. You get 1 public, and 1 private subnet per…
Use the M1 Mac GPU with Go I bought one of the new M1 Macbok pro machines when they came out. I’m really happy with it, everything works great, and they got rid of the Touchbar (which I hated) and put Magsafe, HDMI, and the SD card slot back. But, the chip is the special bit. I went for the M1 Max with 64GB RAM, 10 CPU cores and 32 GPU cores.
Alerting on errors being logged in CloudWatch Logs, AWS Lambda failures, and HTTP 500 errors with Go CDK In my last post [1], I covered setting up dead letter queues to alert on message handling failures in Lambda. /2022/02/28/aws-go-cdk-encrypted-sqs-lambda-dead-letter-queue [1] In this post, I’ll cover two other areas: Getting alerted when Lambda functions are logging errors. Getting…
Setting up an encrypted AWS Lambda dead letter queue with Go CDK I’ve been building a system that uses EventBridge as the event bus to notify other systems about key events. In the project, when a new item is sold, a 3rd party API must be called to register the customer for a service. To do this, I subscribed a Lambda function to the EventBridge bus, and called their API in program code.
Verifying download hashes during Docker build On a system I was working on, I got a security finding back with the comment: Insecure Software Install Mechanism in Container Build This was down to installing binary releases of software using curl inside a Dockerfile which was used to build a Docker image: # Install Go. RUN curl -L -o go1.17.2.linux-amd64.tar.gz…
Export all CloudFormation templates for a pentest Prior to the first release of a project, alongside internal security testing, and security tooling carried out as part of CI/CD, it’s common to have a pentest carried out by a 3rd party security team to check over everything. Security teams often request the CloudFormation templates used to create infrastructure, since having good…
AWS API Gateway V2 to Go Lambda HTTP Handler Adaptor Why use API Gateway V2 instead of V1? API Gateway V2 is faster and cheaper than V1, but packs fewer features, as per the comparison table [0]. https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vs-rest.html [0] AWS refers to API Gateway V2 as the “HTTP API” in documentation, and V2 in the API calls, whereas V1 is…
Using Storybook with Go frontends Storybook [0] is an open source tool for building UI components and pages in isolation. https://storybook.js.org/ [0] I’ve used it with lots of React projects, where it’s been a great way to build out layouts and to allow developers to share, document, and preview components in isolation. However, it’s not just for React. Storybook also supports…
Testing templ HTML rendering with goquery I’ve been building the templ templating language [0] as a way to define and share reusable HTML UI components and layouts for the Go programming language. https://github.com/a-h/templ [0] Developers often use unit testing to describe the expected behaviour of components and layouts that they’re developing, to test that the expectations are met,…
Trigger Fargate tasks on S3 upload with Go CDK If you’ve got a file processing job that might take more than the 15 minutes of time you have with AWS Lambda, or you want to use more then 10GB RAM, you’ll need to consider an alternative way to run it. While it’s possible to break S3 files up into chunks and process them in parallel using Step Functions [0] [1], sometimes you…
Secure your AWS CI/CD pipelines with a Permissions Boundary The permissions required to deploy a serverless project are fairly broad. Lambda functions that access AWS resources such as S3 buckets, or DynamoDB require the creation of a new role that has the appropriate access, or the use of an existing role that has the access. To support this, many teams give their CI user the AdministratorAccess…
Event Sourced DynamoDB A year ago, I wrote up and open sourced an event sourced database design that uses DynamoDB and the TypeScript programming language to update the state of an entity and store it in DynamoDB [0]. /2020/08/28/event-sourced-dynamodb-design-with-typescript-part-1/ [0] The approach is particularly useful for systems that receive a mixture of synchronous (e.g. REST API calls) and…
Github Actions CI/CD for Go AWS CDK projects To get CI/CD working, there’s a few moving parts to take care of. Set up a CI/CD user with appropriate permissions in your AWS account and take a note of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY Create Github secrets for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables Configure the Github Actions execution environment…
Go CDK: Building Go Lambda functions Go Lambda functions are structured as an individual Linux executable that uses the github.com/aws/aws-lambda-go/lambda package’s Start function to run the Lambda runtime to process requests. Requests are then processed by the handle function that is passed into lambda.Start. The signature of the handle method is different depending on the event…
Think before you iPaaS Building out new products is often a case of combining custom code with SaaS services to deliver a unique solution to customers. SaaS services include customer service tooling products used to interact with customers such as ZenDesk, card payment solutions such as Stripe, and management tools such as Shipstation for ecommerce fulfillment. A common challenge faced by teams is…
Cancelling Go network requests I wanted to update my Gemini browser [0] to respond to Ctrl-C to cancel network requests. At the moment, there’s a short timeout to catch when servers are too slow to respond, but I want to increase the timeout and let the user decide when they’ve waited long enough for the server to respond. https://github.com/a-h/min [0] When I’m planning a…
Running a Go app on AWS App Runner with CDK App Runner [1] is a new service for running Web applications on AWS. I’m interested in finding ways to reduce time spent on infrastructure setup and management, and this service claims to be: the easiest way to run your web application (including API services, backend web services, and websites) on AWS. With App Runner, there is no infrastructure…
Building a Hotwired web app with Go and Templ Hotwire [0] is an approach used to build Web applications without using a lot of front-end JavaScript. Most of the content is served as HTML via server-side rendered templates, which makes it an ideal partner for the templ [1] templating language. In fact, I designed templ with exactly this scenario in mind. https://hotwire.dev/ [0]…
Templ - hot reload with Air My colleague Joe [0] introduced me to air [1] this week. He’d been using it while building a web app with the templ [2] templating language to rebuild templates, and restart his Go web server. https://github.com/Joe-Davidson1802 [0] https://github.com/cosmtrek/air [1] https://github.com/a-h/templ [2] I’d never seen it before, but it was just what I was…
Thoughts on team metrics You probably already know from the title of this post that you’re going to read some variation of the phrase “you can’t improve what you can’t measure”, a phrase that’s often attributed to Peter Drucker, so let’s just get that out of the way first. Much like good song lyrics, there’s a few readings of the phrase you can take, based on how you feel. You could…
Introducing templ About templ Templ is a language, command line tool and set of IDE extensions that makes it easier to write HTML user interfaces and websites using Go. It strives for simplicity, developer experience and performance - in that order. The command line tool generates Go code from the templates (templ generate) provides formatting (templ fmt) and supports the VS Code and vim…
Trying out npm and yarn workspaces I’m doing most of my work with CDK at the moment, to deploy Serverless projects. These projects often consist of multiple projects in the same repo - i.e. a monorepo. There’s the CDK project itself, containing the definition of the infrastructure, and one or more projects containing application code that will be deployed as AWS Lambda functions. For…
Launch a Gemini capsule on AWS with the CDK I got into the Gemini protocol [0] last year and ended up writing a server [1] and a browser [2] in Go. https://gemini.circumlunar.space [0] https://github.com/a-h/gemini [1] https://github.com/a-h/min [2] But… I didn’t start hosting my blog using Gemini for a while because of the work involved in migrating the content over, and setting…
Using AWS X-Ray with a TypeScript Lambda A while back, I set up a repository to demonstrate using CloudWatch Embedded Metric Log format [0]. Before this was introduced, I used to write log entries in JSON format, and then create a metric extraction filter to turn the log files into metrics. This was a bit time consuming (just another thing to set up), so it’s great that in AWS Lambda, if you…
Google OAuth token CLI tool I’ve been setting up some APIs to use Google Authentication as the identity provider. The API needs to know the user’s identity (well, just their email address), and that’s it. This requires getting a JWT out of Google’s identity system. First, you have to set up an OAuth 2.0 client ID in the Google Cloud Console, and get the client ID and client…
Setting up AppSync GraphQL subscriptions with TypeScript and CDK I’ve been working on some systems that would benefit from pushing realtime data to clients via Websockets. I tried out using Web Sockets in API Gateway, which was really easy to set up, but the service doesn’t handle subscription management, it’s up to you to track connections by their ID, what each connection is…
AWS CLI - authenticating with SSO AWS CLI v1 didn’t support AWS SSO, but the new AWS CLI does. Before this, you had to do a complicated dance of configuration, or use a tool to save yourself the trouble. Prior to AWS CLI v2 being released, I used the SSOFresh tool: [0] which took away the complication of the various command line incantations that were required without it.…
Running DynamoDB Local with nix I’ve been using Nix [0] as a package manager for a while. Rather than installing packages globally like brew, the idea with Nix is to have a minimal set of base packages, then to start shell sessions containing just the tools you need. This reduces the amount of cruft just sitting around on your machine. https://nixos.org/ [0] I wanted to run a local DynamoDB…
Idempotency in Lambda - 1 - What is it and why should I care? This is part 1 of a 3 part series. Part 1 - What's the problem and why should I care? Part 2 - Using DynamoDB to manage once-only functionality Part 3 - AWS and Stripe APIs that support idempotency. What is it, and why should I care? Idempotence is the property of certain operations in mathematics and computer science whereby they can…
Idempotency in Lambda - 2 - Dealing with it This is part 2 of a 3 part series. Part 1 - What's the problem and why should I care? Part 2 - Using DynamoDB to manage once-only functionality Part 3 - AWS and Stripe APIs that support idempotency. Making a Lambda function or API idempotent The strategy here is to ensure that a protected resource (our API call) is only used once.
Idempotency in Lambda - 3 - Idempotent APIs This is part 3 of a 3 part series. Part 1 - What's the problem and why should I care? Part 2 - Using DynamoDB to manage once-only functionality Part 3 - AWS and Stripe APIs that support idempotency. Idempotent APIs Some APIs and services provide us with capabilities to help deal with idempotency. However, each service still requires careful consideration…
Simplifying TypeScript code with Jest mocking If you’ve ever thought that your TypeScript code is too complex due to dealing with dependencies, this post is for you. I’m going to start with a scenario. I have three functions that I want to compose together. Let’s call them “a”, “b” and “c”. My function should call all 3 of them and return an…
AWS - migrate to MFA Like many people, I’ve set up a personal account, but I haven’t really given it much attention. I set up my account, set myself up an IAM user with Admin access, created some access keys and put them into my ~/.aws/credentials file. This isn’t very secure. The credentials are almost as powerful as the root ones, and are just stored on disk. If they were…
Migrating to aws-vault One of the downsides of the default AWS command line set up is that by default, it stores credentials in plaintext in the ~/.aws/credentials file on the local disk. It’s always bothered me, but recently I found [0] https://github.com/99designs/aws-vault [0] aws-vault uses a password manager to store credentials instead. By default, on MacOS, it uses MacOS Keychain, but…
How to send secrets to me If you need to send me an API key, AWS IAM credentials, username/password combination, or other data that you want to ensure remains secure, you can encrypt it for me, and only me, using public key cryptography. This is more secure than using a password. To do this, you will need to install gpg. You can do this using your system’s package manager (e.g. brew install…
Pentest passing S3 bucket CloudFormation config As a consultant, I launch a lot of new services with customers. Before launch, there’s usually a penetration test of the system and any associated AWS infrastructure. One thing that regularly shows up on the AWS configuration side of things are S3 buckets. Common checks include: Check if S3 buckets have access logging enabled. Check if S3…
Event sourced DynamoDB design with TypeScript - Part 1 This is part 1 of a 2 part series: Part 1 - Event Sourcing design Part 2 - From design to implementation Full example: [0] https://github.com/a-h/hde/ [0] Introduction Most applications deal with entities, “a thing that exists”, and the processing of events that effect change in that entity, such as a user wanting to update a…
Event sourced DynamoDB design with TypeScript - Part 2 This is part 2 of a 2 part series: Part 1 - Part 1 - Event Sourcing design Part 2 - Part 2 - From design to implementation Full example: [0] https://github.com/a-h/hde/ [0] Part 2 In part 1, we looked at the difference between Transaction Scripts and Event Sourced database designs, and the DynamoDB features that make Event Sourced database…
Install AWS Amplify CLI with Nix I switched to using nix-darwin [0] to install and update various base packages on my system a few months ago. I intend to get around to using it to define the tools in use within various projects I’m working on, rather than installing things globally, but I’m quite happy with the base setup I’ve got for day-to-day work for the moment.…
Migrate from Wordpress to self-hosting on AWS This year, I switched my blog away from wordpress.com (where it had been for years), over to hosting it myself on AWS because I didn’t like the new editor in Wordpress (and the lack of markdown support). It’s cheaper to host it on AWS, but that wasn’t the main point. I thought I’d document the process for anyone else, that wants…