Using Secrets In A Lambda
Here’s a guide on the different ways to use a secret in an AWS lambda .
Home on cbui.dev
Recent content in Home on cbui.dev
Latest posts
Use Just In Time Elevated Access
This week, I came across a SaaS product called Indent . I have no affiliation to them nor have I used them (yet). One thing that their product reminded me of is the need for what I’m calling just-in-time elevated access. For example, I used to work at a company where when you were going on PagerDuty, you had to make a ticket to get approval for elevated access to production systems for your…
Setting A Deployment's Replicas And Having A Horizontal Pod Autoscaler
What happens if you set a Kubernetes deployment’s number of desired replicas and have a horizontal pod autoscaler (HPA) for the same deployment? The short answer is the HPA takes precedence. The HPA will override the Deployment’s replica count based on the observed metrics and the scaling rules defined in the HPA configuration. Even if you manually change the replica count in the…
SQLite in Production
There’s this thread on Hacker News asking about people’s experiences running SQLite in production. In the spirit of achieving more with less, using SQLite for a production web service can fit certain use cases. Several people mention running their SaaS businesses on SQLite. The thread also mentions litestream which is a tool for streaming replication for SQLite.
Caching Docker Layers In GitHub Actions
I recently came across this guide on using Docker caching on GitHub actions. I normally don’t bother with Docker layer caching on GitHub Actions. However, the article presents three ways to approach it.
Don't Define The Same Env Var Multiple Times With Kubernetes
Kubernetes environment variables for CronJobs or Pods are defined as a list. You can technically define the same environment variable multiple times, and it will take the last one on the list. However, I’ve found that this leads to strange behavior with Kubernetes tooling, and I’ve since avoided this practice. For example, I’ve seen ArgoCD fail to generate a diff because two…
Terraform 1.8.0 Is Out
Terraform 1.8.0 has just been released today . Here’s a list of new features: Providers can now offer functions which can be used from within the Terraform configuration language. The syntax for calling a provider-contributed function is provider::provider_name::function_name(). (#34394) Providers can now transfer the ownership of a remote object between resources of different types, for…
How To Defend A Shared Database From Noisy Neighbors
Yesterday, I wrote about the problem of defending a single shared database instance from noisy neighbors . Today, I’m going to talk about one way you might go about doing this. Our approach is to use a cron job that runs every few minutes and then queries the database for its active processes and kills any long-running queries. We considered an approach of running the process inside of the…
Defending A Single Shared Database Instance
Let’s say for some reason you ended up with a single, shared database instance that supports many microservices instead of having one database instance per service. Don’t worry, it happens to all of us. One reason this might happen is to save money. It’s cheaper in terms of infrastructure costs to have a single, large, database instance than to have many small ones generally. Of…
Upcoming Kubernetes 1.30 Changes
Kubernetes 1.30 is coming out soon, on April 17th, 2024. Here’s a link to the list of major changes so you can prepare for any deprecated APIs. The most notable change to me is the addition of container-based pod autoscaling.
Merging Helm Chart Values With Kustomize
A while back, I wrote about using Kustomize and Helm together . It’s a good strategy to get the best of both worlds. You get the nice developer UX from using Kustomize, at the same time, you can use Helm when templating is useful. The main reason I advocate for using both is if you want to create a shared, base set of manifests with Helm that you inflate with Kustomize. It’s like…
Create IAM Users And Roles Instead Of Using Root AWS Account
After setting up MFA on your root AWS account, you’ll want to create IAM users for your team and use those credentials instead of using the root account. Similarly, you’ll want to use IAM roles for granting permissions to AWS services. Here’s why: If you’ve just gotten your first round of funding and AWS credits, it’s overkill to set up SSO account syncing with an…
Enable MFA (Multi-Factor Authentication) On Your Root AWS Account
This post will be the start of an AWS Account Setup series for startups that just received AWS credits. The first thing you should do when setting up a new AWS Account is to enable multi-factor authentication (MFA) for the root AWS account. The root AWS account is the initial account created when you sign up for AWS and has complete access to all AWS services and resources in the account. It is…
What Are Kubernetes Service Accounts Used For?
Today, I had somebody ask if Kubernetes service accounts could be used for zero trust between services deployed to Kubernetes. Short answer, no. But it did drive me to review what Kubernetes service accounts can do: Your Pods need to communicate with the Kubernetes API server, for example in situations such as the following: Providing read-only access to sensitive information stored in Secrets.…
Hidden Costs Of Managed Cloud Databases
I came across this article on the hidden costs of managed databases . I default to using managed databases if I can, but it’s good to be aware of the not-so-obvious costs of using a service like RDS.
Kubernetes ConfigMap Best Practices
The third factor of the twelve-factor app is to store your application’s configuration in the environment. How to use ConfigMaps In Kubernetes, you can have deployments that have their environment variables defined explicitly: apiVersion : apps/v1 kind : Deployment metadata : name : my-deployment spec : selector : matchLabels : app : my-app template : metadata : labels : app : my-app spec :…
Focus On Software Delivery Fundamentals
When you’re in a situation where you feel people are overcomplicating software delivery, take a step back and think. Does the path we’re going down help improve our software delivery fundamentals? Or does it avoid improving the basics altogether? Stephen Curry is one of the most influential NBA players today. Kids growing up today want to be like Steph and shoot three-pointers. They…
Routing ALB Traffic With A Header
Say you’re using the AWS LB Controller and want to route traffic to specific hosts based on a header. You can do that with some annotations on your ingress. See these docs . This is useful if you want to route traffic to different Kubernetes services. For example, you might have two versions of a service, each with different sets of feature toggles. You send everybody that doesn’t have…
Creating A Terraform Workspace If It Doesn't Exist
Today, I needed to add a step to my GitHub Actions workflow to create a Terraform workspace if it doesn’t exist. You can do that like this: terraform init terraform workspace select -or-create <workspace name> See: https://developer.hashicorp.com/terraform/cli/commands/workspace/select
Separate Build And Runtime Secrets
It’s a good practice to separate the secrets you need at build time and at run time. Here’s why: Separating the two types of secrets minimizes the risk associated with secret exposure. Build-time secrets, if compromised, should not give access to environments or resources beyond your CI system. Run-time secrets are often more critical as they might grant access to live databases,…
Deploy More
Here’s an excerpt from Stripe’s 2023 annual letter : Teams at Stripe work tirelessly to deliver this industry-leading reliability, and we decided we should share a little about how they accomplish it. Since many outages at internet companies are in some way triggered by a change gone wrong, we can do so by walking through how new code gets deployed at Stripe— something that happens to…
How To Find AWS VPC's DNS Server
I was setting up Tailscale subnet routers in AWS. I needed to set the VPC DNS servers in Tailscale under the DNS tab to resolve DNS entries in our private hosted zones. How do you get the ip address of an AWS VPC’s DNS server? It turns out that each VPC has one DNS server, and it’s located at the VPC’s CIDR block + 2. This means that if your VPC’s CIDR block is 10.10.0.0/16…
Github Actions Composite Actions Vs. Reusable Workflows
When should you reach for a composite action vs a reusable workflow? They’re similar because a composite action now lets you call other actions within it. Reusable workflows are also a combination of many steps with their actions. Actions are executed on the same runner on which its job is running. You should reach for an action if you need to pass data between job steps. One example is if…
Wait For Daemonset Pods Before Starting Pods
I’m not sure why I made this assumption, but I learned recently that Kubernetes DaemonSet pods aren’t guaranteed to be up before regular pods are scheduled on a node. I had a pod that didn’t have enough resources, which caused the cluster-autoscaler to kick in and spin up a new node. When the new node came online, the pod was instantly started. It ran before kube-proxy and our…
Google's Product-Focused SRE
Google has a more recent paper on product-focused reliability for SREs . The paper is a newer iteration of Google’s SRE practice that considers user flows in the product. If you’ve read the Google SRE book , you’ll want to give it a read.
Create Executables From A Docker Image
I just came across this tool called dockerc . It’s fairly new. It lets you create a binary from a docker image, which blew my mind. It doesn’t support Windows or MacOS yet, nor does it support arm64 yet. I suggest following the project and seeing if it progresses further because I’m excited to see if I can ship portable tooling in languages other than go .
IAM Policies With Principal Org Ids
Let’s say you want to write an AWS IAM policy to allow anybody under your AWS organization to perform an action, like access s3 buckets. You can do this by using the PrincipalOrgId condition. { 'Sid' : 'Allow org access to all s3 buckets' , 'Effect' : 'Allow' , 'Principal' : { 'AWS' : '*' }, 'Action' : [ 's3:*' ], 'Resource' : '*' , 'Condition' : { 'StringEquals' : { 'aws:PrincipalOrgID' :…
Is IaC Only Useful For Larger Systems?
Is infrastructure as code (IaC) useful for working on a small team or by yourself? I’ve seen this question come up several times. Personally, my answer is yes. I’m the only one working on my GitHub/Slack app right now. It’s much faster for me to create resources and set things up using Terraform than clicking through the AWS Console (ClickOps). Here’s why: You essentially…
Speed Up Docker Builds With Cache Mounts
You can speed up your docker builds by adding cache mounts to your Dockerfile . Cache mounts let you specify a persistent package cache to be used during builds. The persistent cache helps speed up build steps, especially steps that involve installing packages using a package manager. Having a persistent cache for packages means that even if you rebuild a layer, you only download new or changed…
How To Only Run GitHub Actions Steps If Files Change
If you want to conditionally run a GitHub Action step or job when files change, then you’ll want to use the paths changes filter action. You can’t use GitHub’s built-in path filters because it only operates on entire workflows. Here’s what it looks like: - uses : dorny/paths-filter@v3 id : changes with : filters : | tf: - 'deploy/tf/**' - run : terraform apply ... if :…
Kubernetes VS Code Extension
Today, I found out about the Kubernetes VS Code Extension . One unique feature is converting existing resources into a Helm template. If you’re not a fan of cli tools like k9s and you use VS Code, you should give this a try.
Loki Vs. Elasticsearch For Log Aggregation
After yesterday’s post on sending Docker logs to Cloudwatch, I researched different log aggregation solutions. Just to understand them. I had been aware of the ELK stack – Elasticsearch, Logstash, Kibana, but didn’t know much about Loki by Grafana Labs. Here’s a good article comparing the differences between the two. The main difference between Loki and ELK is that Loki…
How To Send Docker Logs From EC2 To CloudWatch
You can send logs from docker containers to AWS CloudWatch easily. Docker has a built-in logging driver that can ship to CloudWatch. Background I was working on my side project this morning. It’s deployed as a single docker container running on an ec2 instance. It quickly became annoying to shell onto the host to run docker logs -f to get logs. I didn’t want to set up a heavy…
Encourage Your Platform Engineers To Talk To Their Users
If your company practices platform engineering, having your platform engineers talk to their users is important. Here’s why: You need to be able to empathize with other engineers. It’s easy to work on new tools based on perceived pain points when you’re working in a silo. You’ll end up wasting time building tools that nobody will use. I’ve found it super useful to…
Test Your Helm Charts
My team publishes a shared helm chart “base” for our shared services to consume. We needed to ensure we don’t release broken charts. So, we write unit tests for our helm charts with Gruntwork’s library . I don’t think there’s enough value in doing integration tests for helm charts yet. But I’ve already got a lot of value from having basic unit tests on our…
Mirror GitHub To AWS Code Commit
GitHub has been less than reliable lately. If you’re using GitOps then GitHub can become a single point of failure for your company’s deployment pipelines. I found this guide on how to set up AWS Code Commit to mirror your repositories in case that’s of interest. Let me know what you all think and if you’re also having issues with GitHub.
Don't Couple Your Deployments
You’ve made changes to your service and go to deploy it. Among your changes, you added a new SQS queue. Your deployment pipeline applies the IaC changes to get you that new queue. Then, it updates your code. Later, there’s a problem with your code, so you roll back. However, rolling back the commit will also revert your SQS queue and cause your IaC to destroy it, which you don’t…
Traefik As A TCP Reverse Proxy
In case you haven’t heard of it, Traefik is a load balancer and reverse proxy. Its selling point is that it is simple to use and operate and integrates well with containerized workflows. I plan on using Traefik as an HTTP load balancer for my side project. But I also just learned you can use it as a TCP proxy. This means you can use it for SSH, SMTP, AMQP, etc. Here’s an example: #…
AWS Account ID Best Practices
Tracebit published an article that describes how to find the AWS Account ID associated with an S3 bucket. This is scary. While your account ID isn’t considered a secret like a password or access key, keeping it confidential from the public is highly recommended. Attackers can’t do anything just with your AWS Account ID. But it can provide ways for them to gather more information about…
Dumping/Restoring Postgres With Ansible
I was playing with Ansible again for a side project. One cool thing I learned was that there’s a plugin for Postgresql that makes it easier to do common operations, such as dumping and restoring a database. Here’s a snippet of what it looks like to do both operations: - name : 'Dump database to a file {{ db_name }}' community.postgresql.postgresql_db : name : '{{ db_name }}' state :…
Thoughts on the Kubernetes Secret Store CSI Driver with AWS
Last year, I evaluated using the secrets store CSI driver for Kubernetes with the AWS provider . It’s terrible. Don’t use it. I needed a way to use secrets stored in AWS Secrets Manager in EKS. Our engineers were used to having secrets available to their services via environment variables. Like this: apiVersion : v1 kind : Pod metadata : name : env-single-secret spec : containers : -…
Routing Traffic To Multiple EKS Clusters With AWS ALB
I’m working on figuring out how to set up a second production EKS cluster. The secondary cluster will run the same services as the first cluster. My biggest question is how do we route traffic across both clusters? I just came across this article on AWS’s blog that outlines an architecture that uses an ALB to route traffic to both clusters. The insight I got from this article is using…
Deleting RDS Automated Snapshots
We recently encountered a situation where we needed to delete the automated RDS snapshots. Our RDS data size was hundreds of gigabytes, but our snapshot sizes ballooned to 11 terabytes per snapshot. That’s a story for another day. Our retention policy is set to 30 days, and unlike manual snapshots, you can’t delete them directly. The solution? Modify your RDS backup retention period to…
How To Set startingDeadlineSeconds For Kubernetes CronJobs
The startingDeadlineSeconds property in Kubernetes CronJobs is an important parameter. It determines the behavior of jobs that miss their scheduled time for any reason. startingDeadlineSeconds determines the deadline in seconds for starting the job if it misses its scheduled time. If a CronJob misses its scheduled time for any reason and more time than startingDeadlineSeconds has passed, the job…
How To Use Liveness Probes In Kubernetes
Somebody asked me: We’re spinning up a service that is going to be using an external system as a data source. If that data source is not available, the service can’t do it’s job. Does it make sense to use a livenessProbe/readinessProbe to ensure that there’s basic connectivity to the external service before taking traffic? For some additional context, the external system is an API, not a database.…
Ansible With AWS SSM Inventory
Earlier this week , I mentioned that you should prefer AWS SSM over SSH keys. I use ansible as my configuration management tool of choice. Here’s a quick guide on how to use AWS SSM with ansible: We’ll be using the aws ec2 inventory plugin. One thing to note about this plugin that I found out the hard way is all inventory files must end with a suffix of _aws_ec2.yaml or _aws_ec2.yml…
Setting Default CPU/Memory Requests For A Namespace
Today, I ran into the problem of trying to enforce a minimum amount of resources requested for a Kubernetes namespace. We have a namespace that developers deploy all of our microservices. Not all of the deployments have a minimum CPU/memory set. One way you can do this is to use a LimitRange. It looks like this: apiVersion : v1 kind : LimitRange metadata : name : mem-limit-range namespace : <your…
Infrastructure Decisions At A Startup
I came across this amazing post, (Almost) Every infrastructure decision I endorse or regret after 4 years running infrastructure at a startup The author, Jack Lindamood, outlines his experience running infrastructure in a unique format, and his experience aligns almost exactly with mine at my current company two years in. Given that, I’m going to highlight certain bullet points: AWS VPN: He…
Don't Use SSH Keys With AWS EC2
Managing secure access to virtual machines (EC2) with ssh keys isn’t easy. That’s why you should stop using ssh keys if you’re using AWS. A best practice is to use AWS Systems Manager (SSM) . SSM has a feature called Session Manager that allows you to do what you’d normally do with ssh, such as getting a terminal into a virtual machine. Since SSM is an AWS offering, you use…
Lowering AWS Secrets Manager Costs
I’ve seen people consider switching to Hashicorp’s Vault when they’re already using AWS Secrets Manager because of the perceived costs. It’s expensive to migrate your secret store entirely, so reducing Secrets Manager costs is often easier and faster. First, you need to understand where the bulk of the Secrets Manager costs come from – storage. On us-east-1, a single…