Recent content on NotesToSelf.Dev
Application configuration in AWS Systems Manager Parameter Store often mixes plain values with SecureString ones. A SecureString read comes back encrypted unless the request specifically asks for decryption. Nothing fails at that point, so you just get the base64 encoded ciphertext. Simulated SSM in Yulin encrypts a SecureString value through simulated KMS when it is written, so a local test reads…
Applications which use presigned S3 URLs can be awkward to test. The URL is just a string, so a test might only check that it looks about right. It would be better if tests could verify the signature, and confirm that the signing principal is allowed to perform the action enabled by the presigned URL. Checking that often means deploying a real S3 Bucket, IAM credentials and so on to confirm that…
A lot of bugs in real systems are related to time. Time logic is difficult to test, so it often ends up covered by narrow unit tests around single functions, rather than wider system behaviours. Those kinds of tests are better than nothing, but they tend not to expose problems that occur across multiple parts of the system. The simulated clock in Yulin helps with testing time logic across more…
Simulated AWS services in Yulin take their timestamps from a clock which belongs to the simulation rather than to the process running it. That clock can be moved, so you can simulate and test behaviour which only becomes interesting once time has passed. Here’s a simulated Bucket with an Object in it, and a Role which is allowed to read that Object. They run in a simulation which starts at a…
Here’s a minimal CDK app with a Lambda function that reads an Object from an S3 Bucket: import * as cdk from 'aws-cdk-lib' ; import * as lambda from 'aws-cdk-lib/aws-lambda' ; import * as s3 from 'aws-cdk-lib/aws-s3' ; const app = new cdk . App (); const stack = new cdk . Stack ( app , 'FooStack' , { env : { account : '111111111111' , region : 'eu-west-2' }, }); const dataBucket = new s3 .…
If you want to check locally that your IAM policies allow and deny what you think they do, you can use the simulated IAM service in Yulin . Sim IAM stores simulated Roles, Users and Policies, and evaluates allow and deny decisions for them. The other simulated services use it to authorise their own actions, so a simulated S3 call from a caller without the right permissions is denied in the same…
Here’s a Bash script to dump a project into a flat file structure in a single directory. It turns nested paths into flat filenames and prepends each file with its original source path. It also includes the current Git hash in each flat filename, so that LLM memory can distinguish different versions if necessary. The script also writes a manifest file that maps each flattened filename back to…
Here’s a minimal CDK app that deploys a CloudFront Function: import * as cdk from 'aws-cdk-lib' ; import * as s3 from 'aws-cdk-lib/aws-s3' ; import * as cloudfront from 'aws-cdk-lib/aws-cloudfront' ; const app = new cdk . App (); const stack = new cdk . Stack ( app , 'TestStack' , { env : { account : '111111111111' , region : 'eu-west-2' }, }); new s3 . CfnBucket ( stack , 'SiteBucket' , {…
AWS CDK has a bucket deployment feature which automatically copies files from a local filesystem directory to an S3 Bucket during CDK deployment. You can simulate this behaviour for tests and local development with the Yulin AWS simulator npm package . A minimal CDK setup might look like this: import * as cdk from 'aws-cdk-lib' ; import * as s3 from 'aws-cdk-lib/aws-s3' ; import * as s3deploy from…
CloudFront Functions can be slightly inconvenient to test and iterate on, because the CFF JS2 syntax is idiosyncratic and does not support module exports. Often the only options are to use the CloudFront Function testing tool in the AWS console, or to deploy to a development account and iterate that way. The AWS simulator library Yulin can simulate CloudFront distributions and CloudFront Functions…
In the last post I wrote about how TypeScript assertion signatures are more useful when they can incorporate existing type information from the calling scope. Sometimes the asserted type is less precise than information TypeScript already has in scope at the calling site. function assertString ( value : unknown ) : asserts value is string { if ( typeof value !== 'string' ) { throw new Error (…
TypeScript has a useful feature called assertion signatures that allows you to write functions that can be used to narrow the type of a value. For example, you can write a function that asserts that a value is a string and tells TypeScript that after this function has been called, that value must be a string. function assertIsString ( value : unknown ) : asserts value is string { if ( typeof value…
A lot of tests use the expect().toMatchObject() assertion in Jest or Vitest, because we’re most often dealing with object structures returned from the subject of the test. The toMatchObject assertion is useful because it applies a deep partial match, so you only need to specify the particular property paths you’re interested in. It doesn’t matter if other properties are added to…
In TypeScript projects it’s common to have separate tsconfig files covering the entire project and separate build contexts which exclude test files. This can lead to failing to check test files with tsc, so type errors in test files creep in unnoticed only to be discovered later when the root problem is less clear. For example the main tsconfig.json file in a project might be similar to…
If you’re using S3 to host a static website and want to test and develop it locally, you can use the @kensio/yulin package. This might seem pointless when it’s easy to serve a static website on localhost in various other ways (such as with hugo serve ), but even for this small use-case Yulin does provide some extra benefits. One immediate small benefit is that Yulin simulates…
The @kensio/yulin library is an open source TypeScript package for simulating AWS for local dev and isolated unit testing. AWS state is simulated internally, so you can test realistic interactions with multiple AWS services. Yulin docs website The word yǔlín (雨林) is Chinese for “rainforest”. This is a roundabout reference to “Amazon” as in Amazon Web Services. The…
AWS CloudFront Functions (CFF) use a special subset of JavaScript called JS2 . JS2 syntax is simpler than full JavaScript, but it’s still nice to have modern tooling around it, such as linting, testing, and IDE autocomplete. This tooling can also help avoid mistakes related to the minimal JS2 feature set. Firstly, these docblock comments in the CFF JS2 file allow the IDE to understand more…
The @kensio/smartass library is an open source TypeScript package that provides type narrowing assertion functions. It uses TypeScript assertion signatures to provide type narrowing information to the TypeScript compiler and IDE autocomplete. The fluent expect().toBe() interface in Jest and Vitest is readable, but is unable to provide type information to TypeScript and IDE autocomplete. This…
In June 2025 I was the second employee to join Fanalysis, as Staff Software Engineer.
The cdk synth command can be pretty slow, even when running for a single small stack. We also often want to run the CDK synth command regularly as part of local development setups, so it’s nice to skip the re-synth if no relevant files have changed. Here’s a bash script that checks for changed TypeScript source files by comparing their timestamp to that of the generated template file…
TLDR: SAM local drops headers that contain an underscore, due to using Flask and WSGI internally, which differs from the behaviour of a deployed API Gateway. I’m working on an existing project that uses AWS CDK to provision API Gateway in front of some Lambda functions. The team would like to be able to run this locally for faster development iterations, so I set that up with the sam local…
Here’s a reusable GitHub Action definition that caches a public Docker image: --- name : Docker public image cache description : Pull and cache a public Docker image for reuse inputs : image_name_tag : description : Docker image name and tag to pull and cache required : true runs : using : 'composite' steps : - name : Restore image cache id : restore_cache uses : actions/cache@v4 with : path…
It’s best if a linter is applied from the beginning of a project, and with default linting rules, as that way the project benefits from the linting with the minimum possible effort. However, we often join existing projects which have already been developed without the use of a linter, by which point there are possibly thousands of linter violations. In that situation, one option is to…
Here’s a bash command to split a text file into multiple separate files by section delimited by a regex pattern matching titled sections in the source file: csplit \ -z \ -f section_ \ -b '%02d.txt' \ document.txt \ '/\nSection [0-9]\+\n/' \ '{*}' The key part is the /\nSection [0-9]\+\n/ regex pattern, which matches section titles such as “Section 18” alone on a line.…
Here’s a Python testing utility function for asserting on an expected Exception type from within an ExceptionGroup : from contextlib import contextmanager @contextmanager def raises_in_group ( expected_exception : type [ Exception ], match_message : str = '' , ): ''' Like pytest.raises, but for an Exception inside an ExceptionGroup, for example coming from asyncio.TaskGroup. ''' matched_type…
I helped HOA Mailers with their AWS infrastructure, backend systems and frontend software.
Here’s an AWS CloudFormation template and deployment script to set up Organizational Units and Accounts for Prod and Dev for a new project or department. Deploying a CloudFormation stack from this template will require a user in the Organization’s management account with the AdministratorAccess permission set. The created Organizational Unit (OU) and Account structure will look this:…
If you’re using Coverage.py or pytest-cov in your Python project, you probably end up with # pragma: no cover comments all over the place to prevent odd lines from being marked as missing coverage, like this: class Foobar : def do_something (): # pragma: no cover raise NotImplementedError () You can reduce the need for these # pragma: no cover comments by configuring Coverage.py to ignore…
Here’s a Github Action that posts a notification message to a Telegram chat, without using any dependencies. --- name : Post to Telegram description : Post a message to a Telegram channel inputs : telegram_bot_token : description : Auth token of the Telegram bot via which to post. required : true telegram_chat_id : description : ID of the Telegram channel to which to send the message.…
With branch coverage enabled in Coverage.py or pytest-coverage , you might see it report missing coverage on a compound boolean condition, like this: def foobar () -> bool : return ( condition_1 () and condition_2 () and condition_3 () and condition_4 () and condition_5 () ) That might lead to a report that the line where the boolean compound condition begins to the exit of the function are…
Here’s a recursive utility function in Python that allows overriding nested paths through objects that can be mutable mappings, mutable sequences or dataclass instances. The function allows doing things like this: override_paths ( { 'foo0' : 'bar' , 'lvl1' : { 'foo1' : 'bar1' , 'list1' : [ 1 , 2 , 3 , 4 , 5 , 6 ], 'foobar1' : Foobar ( foo = 'hello1' ), }, }, foo0 = '123' , lvl1__foo1 = '456'…
Here’s a Github Action job that runs pytest coverage and then posts a summary of the coverage report as a PR comment with Markdown formatting. This is handy as part of a PR workflow to make the test results and coverage summary visible on the PR. coverage : name : coverage runs-on : ubuntu-latest steps : - uses : actions/checkout@v4 - uses : ./.github/actions/setup_python - uses :…
Here’s a Github Action that runs a given bash command, captures the stdout and strderr output, and posts a comment with the output back to the PR if the command has a non-zero exit status. This is useful for running check commands such as linters against a PR, and making a PR comment with the linter output if there are linting errors. name : Command & Comment description : Run a bash command…
Here’s a Github Action that finds and removes PR comments based on a string identifier. This is useful for Github Action workflows that post comments to PRs, as it clears up out-of-date comments from earlier workflow runs. name : Command & Comment description : Run a bash command and comment failure output back to PR inputs : identifier : description : Single word identifier for comments to…
You can capture output from a Github Action by echoing a variable name to the file specified by the "$GITHUB_OUTPUT" variable: echo 'foobar='here is some data'' | tee -a ' $GITHUB_OUTPUT ' This is a bit trickier with multiline output. You can use the bash EOF syntax for multiline strings: FOOBAR <<EOF here is some multiline data EOF The easiest way to use bash multiline strings in a Github Action…
Here’s a Github Action to get the URL to the current job and specific step in the job to open. This is useful for giving users a direct link to specific jobs. name : Get current job URL description : Output the current Github Workflow Job URL inputs : step_name : description : Specific job step to include in the URL required : true outputs : job_url : description : 'URL to the calling job…
I noticed a ResourceWarning like this when using aiofiles to iterate lines of a file in an async context manager from aiofiles.open : ResourceWarning: unclosed file <_io.TextIOWrapper name='.../numbers.tmp' mode='r' encoding='UTF-8'> This is only a warning, so by default it won’t error out of the Python program. I try to have pytest filterwarnings set to error whenever possible, so that any…
Here’s a utility function that safely ensures an event loop is running in a Python asyncio application. This is useful because of the discrepancy in expectations between the asyncio library and how it ends up being used in the real world in a lot of applications. The asyncio library reasonably expects users to be in control of the event loop for the whole application lifecycle, but this…
I’ve been helping to build Doozy’s product as a freelance Software Engineer, using GCP, Firebase and TypeScript.
I worked as a Software Engineer on a contract basis for ElseWhen, using asyncio, CloudBuild, CloudRun, GCP, Gemini, Google Cloud Functions, LangChain, LLMs, OpenAI, PostgreSQL, Python, SQL, Terraform and other technologies.
I worked as a Software Engineer on a contract basis for Omnipresent via YLD, using asyncio, AWS, AWS Lambda, AWS SAM, Docker, ECS, FastAPI, PostgreSQL, Python, SNS, SQL, SQS, Terraform and other technologies.
It’s quite likely that you’ll let other organisations know about the Elastic IPs that you have configured in your AWS VPC. The other organisations could be service providers or your customers, and they might not have automated processes for configuring these static IP addresses on their side (sending configurations by email is surprisingly common). This creates a risk around the…
Here’s a small CDK stack to set up an EC2 instance as a bastion host inside a VPC, sharing a fixed Elastic IP with other services in the VPC via the NAT Gateway. A bastion host is useful for connecting to your own services inside the VPC. With this particular setup, you can also use the same bastion host to connect to external services that require a fixed IP address on your side.
Here’s a Bash one-liner to minify JSON in the current contents of the clipboard and replace the clipboard contents with the minified JSON: xclip -selection clipboard -o | python3 -c 'import sys;import json;print(json.dumps(json.loads(sys.stdin.read()),separators=(\',\',\':\')))' | tee > ( xclip -selection clipboard ) The equivalent on Mac is: pbpaste | python3 -c 'import sys;import…
Here’s a Bash one-liner to JSON-encode the current contents of the clipboard and replace the clipboard contents with the JSON-encoded output: xclip -o | python -c 'import sys;import json;print(json.dumps(sys.stdin.read().strip()))' | tee > ( xclip -selection clipboard ) The equivalent on Mac is: pbpaste | python -c 'import sys;import json;print(json.dumps(sys.stdin.read().strip()))' | pbcopy…
Here’s a small bash script to identify an EC2 instance as a bastion host , and then use SSM Session Manager to forward multiple ports through the bastion on to other hosts. This is useful when you want to access hosts inside your VPC (such as RDS / Aurora), external services that enforce an IP access list (such as MongoDB Atlas), or a mixture of those. I’ve used scripts like this to…
I worked as a Software Engineer on a contract basis for KrakenFlex, using asyncio, AWS, AWS Lambda, AWS SAM, CloudFormation, DynamoDB, Python, SNS, SQS and other technologies.
It’s surprisingly difficult to use AWS SSM Parameter secure strings in CloudFormation templates. If you try and have CloudFormation fetch the value of the SSM Parameter as a AWS::SSM::Parameter::Value<String> type, you’ll get this error: An error occurred (ValidationError) when calling the CreateStack operation: Parameters [/foobar/foo_param] referenced by template have types not…
It’s often beneficial to use Docker to build AWS SAM Lambda Function images. Projects using AWS SAM often also use AWS CodeArtifact to manage private libraries, and poetry to manage Python dependencies. This combination can make it a little bit tricky to get the index auth working during the Docker build. Your Dockerfile for building the AWS SAM Lambda Function might look like this: FROM…
I had a large slide deck in PDF format and wanted to add the individual slides to Anki to revise from. There is some benefit to the learning process in doing this manually, but it was more practical to study the slides first and then automate the import into Anki. The final bash command to produce an importable CSV file for Anki looks like this: pdftoppm my_slides.pdf my_slide \ -progress -png -f…