Recent content on gripdev.xyz
Spoiler: I made a free and open-source way to get an interactive web terminal to your GitHub Action when it fails. Try it out here: https://actions-term.gripdev.xyz/ ( code 🔗 ) 1 Your browser does not support the video element. Building it I think we’ve all been there, your build fails in Actions, but the script works fine locally. You now settle into a slow loop of: Push speculative change…
This adventures starts with a simple eBPF program to transparently redirect DNS requests on port 53 for a single program (or docker container). To do this I used BPF_CGROUP_INET4_CONNECT on a cgroup . That lets me inspect and redirect traffic when syscall.connect occurs from within the cgroup . Here is a simplified version 👇 int handle_connect_redirect ( struct bpf_sock_addr * ctx , __be32…
So you have Flipper in your ruby codebase for handling feature flags but… now you have used them and need a way to find them and help you pay down the tech debt of cleaning them up. Well good news, you can use AST Parsing to find all the instance. This simple parser walks all the Ruby code looking for calls to Flipper.enabled? and makes a list of the features flags used.
Quick post. My goal was, I thought, simple: Run a CLI tool which checks if a JSON document is valid for a JSON schema and returns the line numbers with error details So I head over to json-schema.org/tools and look for one. I try: ajv-cli ⛔ Doesn’t handle field type of date boon ❌ Doesn’t show line numbers Test-Json ❌ built in powershell cmdlet - Doesn’t handle if-then-else…
Ever had a long-running command in terminal but forget to check back on it? I’ve used do_long_think; notify-send "thing finished" in the past to help. I can do other stuff then be interrupted when it finishes. Annoyingly that doesn’t work if you’re over an SSH connection, in a DevContainer or using Codespaces . To fix that up I’m now using WezTerm ’s user-var-changed…
GraphQL can throw up some surprising N+1 style performance issues in Ruby on Rails. Usually this come up as “Why is this particular GraphQL query so slow?” Here I’m going to talk through using a Query Analyzer , StackProf and Speedscope to find out what is slowing down that query so you can fix it. What are we hunting? Well N+1’s typically look something like this 👇 on a…
So you’re building a container where it has packages, like npm ci or go mod download , and would you like to make it quicker? For example: COPY go.mod go.sum ./ RUN go mod download Normally when you edit go.mod or go.sum the go mod download will have to redownload all the packages as its cache was busted by the change in the files. Well - we can use inline bind mounts to avoid that.
So you have a docker build which requires access to an authenticated resource? Docker Secrets can help do this cleanly! Temporarily provide build-time secrets There are times when you want to provide a secret, or file with secrets in it, for use in your docker build. Now if you COPY secert.yaml /etc/secrets that’s going to end up in your public image 🫢😨 Also, secret are commonly outside…
So you have some config that you want to provide to a service started from docker compose . Did you know you can keep things simple by inlining that config in the compose file, rather than messing with mounts? Inline content in docker-compose.yaml In our case we want a json file with 👇 mounted at /etc/consul/client.json { 'node_name' : 'consul-client' , 'data_dir' : '/consul/data' , 'log_level' :…
I’m current working on code which lives in an agent, written in golang , which is running on a fleet of machines. The agent needs to, when a new version of an application is deployed, restart the systemd unit and make sure it comes up healthy and running. Here is the output of what we’re going to build: ☠️ Dead-end: Try calling systemctl and parse the stdout This is the most obvious…
Note, I’ll use Ruby on Rails 💎 as my example for this post but this can be done in nearly all modern language as SQIDs support most of them . They’re also interoperable, so you can generate in Ruby and decode in another C# service. Why does GraphQL need Global IDs? In GraphQL has the concept of Global Object Identification . Each item in the API has a unique Global ID. If you’re…
I love tackling difficult problems, finding new innovative approaches, teaching, learning, blogging and engaging with the OSS community. GitHub LinkedIn Mastodon Opinions expressed are solely my own and do not express the views or opinions of my employer.
Tailscale supports using a GitOps flow to manager ACLs for you’re tailnet. This involves configuring a GitHub Action then you commit the ACLs and the Action runs. Pretty cool right? I can see this being a great flow for larger Tailnets managed by multiple people. For me tho, with my homelab Tailnet, this was a bit heavy weight. What I wanted was: ACLs tracked in my HomeNet git repo which has…
I run a HomeLab for hosting a few bits (for example atuin , omnivore and matrix-bridges ) Like all software these things need to be kept up-to-date. This saps the fun out of hosting things. I do not want any toil , here is the goal: I don’t want to manually do OS updates I don’t want to manually update versions of software that I’m hosting I don’t ever want to have to worry…
My HomeLab runs a few useful things (like atuin , changedetect.io , matrix bridges and homebridge ) either in via Docker Compose or hosted on a single node K3s Kubernetes cluster. In both cases YAML all sits in a versioned git repository. If things go bad I can recreate the lab from scratch without too much pain (along with ZFS snapshots of data drives thanks to TrueNAS ). Using this basic…
Resque is a background job processor for Ruby. Sometimes you need to do something that’ll take a long time and you don’t want that happening as part of the HTTP request lifecycle. It helps you do that. But what happens when you want to do LOTS of things at once, you want to avoid the thundering herd problem by spreading that work out rather than doing it all at once. Jitter is one way…
In my normal day I work in my local machine, in codespaces and in devcontainers. Recently I started using Atuin which is an awesome tool for syncing your command history. Go check it out, it’s really nice! A really cool feature is you can search history in different categories, like commands on this host or commands in this directory. Here I set this up for Codespaces so the host is the name…
This was a quick fun hack to pull back the data from my solar inverter and display it in my status bar. Here’s the code
This is a super quick one, I have a Ryzen server running a bunch of VMs. I noticed it’s running quite hot and pulling a fair bit of power. As none of the VMs running are particularly performance sensitive I wanted to force the CPU to use a more conservative power setting. First up, how do I see what frequencies the CPU is currently running at? For that we’ll use cpufreq-info , this…
I’ve been adding Sorbet and type checking gradually to a legacy Ruby codebase dating back to the 2010‘s. First up was getting all or most files as (a topic for another day): # typed: true Now I’m gradually adding method annotations, using sig annotations, to tell Sorbet what types a method accepts and returns, like so: sig {params(x: SomeType, y: SomeOtherType).returns(MyReturnType)} def foo(x,…
A system is failing. People rely on it. You are on-call to fix it. You don’t know how it works, your team don’t know how it works and the last person to work on it has left the company. Fun times! I’ll be upfront. This was an intense on-call shift. It wasn’t much fun but it did help me learn some new approaches for how to handle these situations. This blog is what I was…
Warning: This expects you already know about rego/opa and is more of a brain dump than a blog. First up take a look at conftest it’s a great little CLI tool which lets you take rules you’ve written in rego/opa and run them easily. In our case we have the following: - ./rules folder containing our rego rules - ./yaml folder containing yaml we want to validate We’re going to write…
This is a brain-dump rather than a fully fleshed out blog. Most of the code was written with an unwell small human sleeping on me and python isn’t my best language, it’s very much a hack. I have two kids, both have asthma and chest issues. Unfortunately, these are things you manage rather than cure, they’re more prone to normal colds escalating quickly and need more medical…
Hyper-v on Server 2019 supports Discrete Device Assignment (DDA) which allows PCI-E devices to be assigned directly to underlying VMs. This through me off as my searches for Device Pass Through didn’t return any results! Typically this is used with Graphics cards and all the docs talk extensively about doing just that. What I wanted to do was pass through an LSI SAS controller to my TrueNAS…
This is a quick one, if you get the following error or similar: 2021/06/07 00:15:35 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 189118 vs 130560 2021/06/07 00:15:35 Failed to copy: corrupted on transfer: sizes differ 189118 vs 130560 These track back to an issue with OneDrives metadata generation altering the size of the file. You can see details on this issue…
Brain dump post, excuse typos and writing getting this out of my head while I still remember it. In this post I’m going to go through how I used query based parameters to setup an Application Insights workbook so it does not have hard coded resource ID’s in it’s definition. This means it’s much easier to use for automated deployments where these ID’s aren’t…
The quickest and easiest way to start running a Windows Container in Azure is using Azure Container Instances (ACI). The problem is that they currently (as of 03/21) don’t support running Windows Containers inside a VNET. This blog is about how I worked around this limitation by automating the deployment and management of a Windows Containers with PowerShell DSC and Terraform. As we needed…
Devcontainers are awesome for keeping tooling consistent over the team , so what about when you need to run your build? There is some great work already done talking about how to use these as part of a normal pipeline ( shout out to Eliise! ), what about if you need your build agent to be inside a virtual network in Azure? The standard approach would be to create a VM, setup tools and join that as…
I’ve had pain in my right wrist from computer usage for a while now, last 5 years or so. To fix it I’ve done lots of stuff, like: Move to using mouse left handed Move to roller ball left handed mouse Move to ergonomic keyboard Move to fully split keyboard In the arms race that is keeping my wrists working I decided to go for the next step and get a (expensive) programmable and columnar…
So you’re trying to use the Terraform azurerm_function_app_host_keys resource to get the keys from an Azure function after deployment. Sadly, as of 03/2021, this can fail intermittently 😢 (See issue 1 and 2 ).+ [Edit: Hopefully this issue is resolved by this PR once released so worth reviewing once the change is released] These errors can look something like these below: Error making Read…
So you want to add a diagnostic setting to your Azure storage account via Terraform and you pass the storage account ID to target_resource_id only to get the following error: Status=400 Code=“BadRequest” Message=“Category ‘StorageWrite’ is not supported.” Here is the fix, the diagnostic target resource actually needs to be a sub-resource of the storage account,…
I’m working on a project which uses pester tests to validate our deployment and system health. We’ve accumulated a lot of tests. Nearly all of these sit and wait on things to happen in the deployment and we’re running them sequentially which takes around 20mins. Each of our tests is in it’s own file. What I wanted to do was, as these are IO bound tests waiting on things to…
Shout out to the awesome work here from Alex Yates! This post builds on that work and updates a few bits. What is the the aim? I have a file called IMAGETAG.txt which contains a simple version v1.0.1 . It is used to build and push a Docker container as part of the build. If the file is changed in a commit, I want to build and push the docker image. Now normally you could use the Path filtering…
Warning: This is a bit of a brain dump. I’m working on a project at the moment which dynamically creates a set of CRDs in Kubernetes and an operator to manage them based off a schema which is provided by the user at runtime . When the code is being built it doesn’t know the schema/shape of the CRDs. This means the standard approach used in Kubebuilder with controller-gen isn’t…
21 June 2021: Updated to resolve bug where wrong private key was passed to client_certs. This was exposed by service change. There is a good guide to generating the necessary certificates and manually editing the openvpn config you can download from the portal in the official docs. Being a sucker for punishment I wondered if I could automate the process (mainly because I always forget the openssl…
I recently upgrade my machine and and installed the latest Ubuntu 20.04 as part of that. Very smugly I fired it up the new install and, as I use devcontainers, looked forward to not installing lots of devtools as the Dockerfile in each project had all the tooling needed for VSCode to spin up and get going. Sadly it wasn’t that smooth. After spinning up a project which uses terraform I found…
[Brain dump so I don’t forget this one] So you want your bash script to exit on an error but you’d like it to clean some stuff up before it closes after the error occurs. No problem a TRAP can do this for you (read detailed docs for caveats). In a very simple form it looks like this: https://gist.github.com/lawrencegripper/9e778601b2a21d7891e46cf0e1765f46 Using Trap to fire cleanup on…
[Update Feb 2021 ] There is now a Terraform Provider for Databricks, it’s a better route - https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs My starting point for a recent bit of work was to try and reliably and simply deploy and manage Databricks clusters in Azure. Terraform was already in use so I set about trying to see how I could use that to also manage Databricks.…
For some testing I’m doing I need a set of images of a specific size to simulate pulling larger vs smaller image. Here is a quick script I put together for generating a 200mb, 600mb, 1000mb and 2000mb image (tiny bit larger as alpine included). Took a while to work out best to use /dev/urandom not /dev/zero as with zero the images got compressed for transfer.…
I found myself last week looking at a bit of code in K8s which I thought I could make better, so I set about trying to understand how to clone, change and test it. Luckily K8s has some good docs , trust these over me as they’re a great gui de. This blog is more of a brain dump of how I got on trying with Devcontainers and VSCode.This is my first try at this so I’ve likely got lots of…
I’ve been working with OPA recently and using KIND to test things out. This works really nicely but when I started using the same approach in CI I saw some errors. Digging into things you can see that the nodes of the KIND cluster aren’t “READY” when the CLI finishes up so you need a bit of extra bash foo to make the process wait on the READY status. This monster line of…
First up, quick refresher - what is a mutating admission controller? Well it’s a nice feature in Kubernetes which lets you intercept objects when they’re created and make changes to them before they are deployed into the cluster. Cool right? All those fiddly bits of YAML or hard to enforce company policies around network access, image stores you can and can’t use, they can all be…
[Braindump - warning] So I’ve been playing with devcontainers for Visual Studio Code, they’re awesome… go play with them. They let you use a Dockerfile to describe all the tooling needed for devs to get started with your project. One of the side effects is that you have a nice Dockerfile which you can then also use it for your build server meaning that you never have an…
I started out with C# since then I’ve learned other languages and one of my favorites is Golang. When I was reading the release notes from C# 8 I saw the new using declaration and through it was awesome … I also realized it could be misused to give C# the defer keyword from Golang. Whats defer in Golang do? defer in Golang lets you define a function that will get run when the code…
Note: This is more a stream of consciousness than a blog post. It’s not detailed and mainly for my own memory on how to set this up. Be cautious. So I’ve recently built a new home server/lab and wanted to use some of it’s power to run my daily dev stuff. I’ve been playing a bit with VSCode Remote for Containers which lets you define you dev environment as a container and…
So you’re making a change to the provider to add a feature, it’s going great and your ready to test it out…. but then you realize things get a bit ropey… ideally you want a visual debugger to step through the code. Well here is how to set that up in VSCode. First make sure you have VSCode setup for golang debugging (delve configured etc). Then it’s easy, say you…
So I’ve found myself writing lots of bash scripts recently and because they tend to do real things to the file system or cloud services they’re hard to test… it’s painful. So it turns out there is an awesome linter/checker for bash called shellcheck which you can use to catch a lot of those gotchas before they become a problem. There is a great plugin for vscode so you get…
So it starts out easy, you write a bit of terraform and all is going well then as more and more people start committing and the code is churning things start to get messy. Breaking commits block release, formatting isn’t consistent and and errors get repeated. Seems a bit odd right, in the middle of your devops pipe which dutifully checks code passes tests and validation you just give…
Update 12/11/2020: This is now supported directly in the Azure Terraform Provider see here. Updated 09/03/2020: This new method in the Azure provider has intermittent issues. I have another workaround here which avoids ARM templates as an alternative. So you’ve deployed your function and you want to get pass the secure url another component in your deployment so it can use it… Well…
So it seems like a brain dead simple one. Don’t push secrets by accident, make sure you check and update the projects .gitignore to ignore sensitive files but the reality is different. One example, you use Terraform and set the ignore file to ignore the state file. Then later another developer moves the folder the Terraform is in and updates the ignore. Now when you merge you get the…