Cloud Operations and Development
AWS Certificate Manager can generate custom TLS certificates that can be used by AWS resources to enable secure TLS communication. In this article, I will delegate authority of a custom subdomain to AWS Route 53, then allow AWS ACM to generate certificates for this subdomain. This subdomain and certificate can then be used within the ... AWS: using Certificate Manager to generate a TLS certificate…
Porkbun is an ICANN accredited domain name registrar with value pricing and an API that allows domain record creation and updates, which makes it especially useful for scripting. This article assumes you already have an account, and have Porkbun managing at least one domain name. Create API Access key As a prerequisite, generate an API ... Porkbun DNS: create DNS records via API using curl
GitLab Pipelines provide the ability to define a workflow that has externalized variables that can be set at the direct project level, or inherited from the group or instance level. These can be viewed in the GitLab web UI, but for scripting and automation can also be queried from the GitLab REST API or GraphQL ... GitLab: query the project and group level GitLab CI/CD variables using curl
GitHub Actions provide the ability to define a build workflow, which per your requirements may include calls to AWS infrastructure. However, for those AWS calls to work you need to establish an IAM identity. In the past, you would have to rely on AWS access keys set in project variables/secrets, or applying an IAM role ... GitHub: workflow authentication to AWS using OIDC
GitLab Pipelines provide the ability to define a build workflow, which per your requirements may include calls to AWS infrastructure. However, for those AWS calls to work you need to establish an IAM identity. In the past, you would have to rely on AWS access keys in CI/CD variables, or applying an IAM role to ... GitLab: pipeline authentication to AWS using OIDC
The AWS CLI provides a way to query for DB instances using aws rds describe-db-instances , but this will return all DB instances including those that are members of a cluster. If we want to query for just the standalone DB instances (not part of a cluster), here are a couple of options. You can use ... AWS: retrieving standalone database instances by filtering out those with cluster membership
jq has a very powerful expression language for querying and transforming json. But when dealing with complex data structures and responses in the real-world, well-tested happy path queries can fail. One common error is: jq: error (at stdin :1): Cannot index array with string "..." The root cause is that your query is attempting to pull ... jq: Resolving Cannot index array with string errors from…
Flux is a GitOps continuous delivery tool that provides a framework for keeping a Kubernetes cluster in-sync with source git repositories, OCI registries, and published Helm charts [1]. In this article, I will show how Flux can deploy a Helm chart, and then subsequently monitor for changes and auto-upgrade when new chart versions are published. ... Flux: automating Helm chart deployments with Flux
Flux is a GitOps continuous delivery tool that provides a framework for keeping a Kubernetes cluster in-sync with source git repositories, OCI registries, and published Helm charts [1]. The recommended way to install Flux on a Kubernetes cluster is to bootstrap using the Flux CLI, so I will go through those details in this article. ... Flux: installing Flux on a Kubernetes cluster with bootstrap…
Github Actions provide the ability to define a build workflow, including the packaging and publishing of a Helm chart. This allows tools like Helm to refer to the URL of the public source project, add it as a remote Helm repository, and then use the packaged chart to deploy a workload to a Kubernetes cluster. ... GitHub: automated publish of Helm chart using GitHub Actions
GitLab Pipelines provide the ability to define a build workflow, including the packaging and publishing of a Helm chart to the GitLab Package Registry. This allows tools like Helm to refer to the public URL of the Gitlab Package Registry, add it as a remote Helm repository, and then use the packaged chart. Pipeline job ... GitLab: pipeline to publish Helm chart to GitLab Package Registry
The kubectl get all command only returns a limited set of resources, namely: pods, services, daemon sets, deployments, replica sets, jobs, cronjobs, and stateful sets (not Ingress, Secrets, ConfigMap, CRD, etc.). While there is a rationale for this, it is often the case we need to see all the resources and custom resources defined in ... Kubernetes: showing all resources in a namespace
Enabling the use of the GPU on your Mac M1 with the tensorflow-metal plugin can be challenging because there is a lot of conflicting documentation and older forum questions and replies. I ve written this article for a Mac M1 running on macOS Sequoia 15.1.1. As of December 2024, you should pair Python 3.11 with TensorFlow ... Mac: tensorflow-metal pip module on M1 chip for GPU support
The random_id Terraform resource generates a value that can be used to create remote infrastructure that requires a unique identifier. The primary attribute it exposes is .id which contains upper+lower+number characters, but it also has .dec and .hex equivalent representations that can be used to support infrastructure requiring a limited character set. As an example, ... Terraform: converting hex…
If you need to import a Terraform module , it is critical to understand that importing a module state is not a single bundled operation. Instead, you must import each of the resources inside the module individually. It is unfortunate that you cannot simply supply the module identifier and its variable values to import all its ... Terraform: importing a module by its individual resources
Kyverno is an open-source project that manages and enforces policies within a Kubernetes cluster. The policy definitions are defined as yaml and deployed as Kubernetes objects. Kyverno has become popular for its Kubernetes-specific policy engine and declarative rule definitions (as opposed to a general policy engine like OPA/Gatekeeper that use a domain specific language). It ... Kubernetes:…
If you have specific intentions for a Kubernetes node pool/group (workload isolation, cpu type, etc.), then you can assign labels to attract workloads in conjunction with taints to repel workloads that do not have explicit tolerations applied. And although the generalized kubectl utility can assign labels and taints to specific nodes, the assignment of labels ... Kubernetes: targeting workloads to…
If you have a set of resources in Terraform that are conditionally included based on the same criteria, instead of appending a count/for_each on every resource definition, consider refactoring them into a module. The conditional can then be placed on the module definition instead of polluting each resource definition. For example, if you had several ... Terraform: module for conditional include of…
The output of a Terraform child module is not shown by the root configuration/module. In order to have the child module output shown, you must explicitly define a root level output for it. For example, if you root had a child module defined like below, and you wanted to display the output of that child ... Terraform: enabling the output of a child module
If you are using awk with a relatively indexed NF variable (number of fields), and get an error like below, this is because the input being parsed does not have the number of fields expected. awk: run time error: negative field index $-1 The root problem is the awk expression was expecting a certain number ... Bash: resolving awk run time error, negative field index
Istio Ambient mode is a data plane model that eliminates the need for an envoy sidecar proxy on each of your workloads. This reduces resource overhead, timing issues between sidecar lifecycle and your containers, and the need to restart your workloads to upgrade proxy versions. In this article, I will show you how to install ... Minikube: Istio Ambient mode on Minikube
At the Bash command line interface, there is the concept of programmable completion and regular file/directory completion. This means that when you press the TAB , the alternatives can be provided by a custom program or the filesystem hierarchy. There is always the chance that a program may introduce undesirable behavior to your auto-completion, and if ... Bash: falling back to file autocompletion…
Github Actions provide the ability to define a build workflow, and for projects that are building an OCI (Docker) image, there are custom actions available for running the Trivy container security scanner. In this article, I will show you how to modify your GitHub Action to run the Trivy security scanner against your image, and ... Github: security scanning built into GitHub Actions image build
GitLab Pipelines provide the ability to define a build workflow, and for projects that are building an OCI (Docker) image, there is a convenient method for doing container security scanning as part of the build process. Include Container Scanning As described in the official documentation, add the following include to your .gitlab-ci.yml pipeline definition. include: ... GitLab: security scanning…
Google Pub/Sub is a managed messaging platform providing a scalable, asynchronous, loosely-coupled solution for communication between application entities. It centers around the concept of a Topic (queue). A Publisher can put messages on the Topic, and a Subscriber can read messages from the Subscription on a Topic. In this article, I will first use the ... GCP: publishing and reading from Google…
KEDA is an open-source event-driven autoscaler that greatly enhances the abilities of the standard HorizontalPodAutoscaler. It can scale based on internal metrics as well as external Scaler sources. In this article, I will illustrate how to install KEDA on a GKE cluster that has Workload Identity enabled, and then how to configure KEDA scaling events ... GCP: Installing KEDA on a GKE cluster with…
Although the simple gcloud container operations list command is the easiest way to find recent upgrade events on your GKE cluster or nodepool, it returns only the recent events and does not provide a historical record. If you need to look at historical events, you can use Logs Explorer web UI or use the gcloud ... GCP: historical log of GKE cluster and nodepool upgrades and scaling
Trivy is an open-source tool that can scan your containers and produce reports on known critical issues at the binary and OS package level. In this article, I will describe how to scan images directly from your local Debian/Ubuntu machine, whether you built the image locally or pulled it down remotely. Installation on Debian/Ubuntu Per ... Kubernetes: Trivy for container scanning from CLI
If you find yourself low on disk space, check the size of the Systemd journal logs as a way to easily free up some space. $ sudo journalctl --disk-usage Archived and active journals take up 528.0M in the file system. # reduce size to 300 Mb sudo journalctl --vacuum-size=300M There are automatic jobs that trim ... Linux: reducing disk usage of Systemd journal logs
The openssl utility can be used to show the details of a certificate, including its Not After expiration date in string format. This can be transformed into how many days till expiration with a bit of Bash date math. Create test certificate and key Using a line provided by Diego Woitasen for non-interactive self-signed certification ... Bash: calculating number of days till certificate expiration…
GitLab pipelines are a convenient way to expose deployment/delivery tasks. But with their rudimentary web UI for variable input, it can be challenging for users to populate the required list of variables. One way of making it more convenient for end-users is to provide them a URL pre-populated with the specific branch and pipeline variable ... GitLab: URL shortcut to override pipeline variable…
Terraform has now been open-source and forked with the OpenTofu project. The tofu binary is a drop-in replacement for terraform, and this article will show you how to install on Debian/Ubuntu. After installation, we will then use the Debian/Ubuntu Alternatives concept to supersede existing calls to terraform to instead invoke tofu . Setup OpenTofu apt repository ... OpenTofu: installing OpenTofu…
If you are using Terraform as a way to provide infrastructure/services to outside teams, your Terraform definitions and variables may initially be owned by your team alone, with all tf apply operations done by trusted internal group members at the CLI. But once there is a certain amount of maturity in the solution, the Terraform ... Terraform: external yaml file as a contribution model for outside…
Base64 encoding a string may seem like a straight-forward operation, but there are a couple of gotchas even when dealing with just simple ASCII strings. Avoid embedding a new line character into the encoding If you use the most straight forward method of Base64 encoding shown below, you have to remember that echo by default ... Bash: avoiding newline artifacts when Base64 encoding a string
Mike Farah s yq yaml processor has a a rich set of operators and functions for advanced usage. In this article, I will illustrate how to update deeply nested elements in yaml. This can be done for both known paths as well as arbitrarily deep paths. Sample yaml We will use the following yaml files to ... yq: updating deeply nested elements
Mike Farah’s yq yaml processor has a a full-featured validation command that is very detailed in its reporting, but the yaml specification itself is very lenient, which means yq may accept scenarios you did not expect (e.g. an empty file). yq -v file.yaml /dev/null ; echo "final result = $?" Luckily, the yq tips-and-tricks section ... yq: validate yaml syntax
Keeping the Ubuntu system-level Python version and modules independent from those desired at each project level is a difficult task best managed by a purpose-built tool. There are many solutions in the Python ecosystem, but one that stands out for simplicity is pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions ... Ubuntu: pyenv for managing multiple…
It is relatively easy to experiment with a base LLama2 model on Ubuntu, thanks to llama.cpp written by Georgi Gerganov. The llama.cpp project provides a C++ implementation for running LLama2 models, and works even on systems with only a CPU (although performance would be significantly enhanced if using a CUDA-capable GPU). Download model and run ... Ubuntu: LLama2 model on Ubuntu using llama.cpp
If you need to dig deeper into the dependencies that brew has installed on your Mac, you can show a complete dependency tree using: brew deps --tree --installed If you need to see who relies on a specific target Formulae: brew uses --installed targetFormulae REFERENCES brew.sh, deps command syntax apple.stackexchange, brew deps showing all ... Mac: list deep dependencies of Homebrew formulae
It is relatively easy to experiment with a base LLama2 model on M family Apple Silicon, thanks to llama.cpp written by Georgi Gerganov. The llama.cpp project provides a C++ implementation for running LLama2 models, and takes advantage of the Apple integrated GPU to offer a performant experience (see M family performance specs). Download model and ... Mac: LLama2 model on Apple Silicon and GPU…
minikube makes it easy to spin up a local Kubernetes cluster on macOS, and adding an Ingress is convenient with its built-in Addons. In this article, I want to take it one step further and show how to expose the Ingress via TLS (secure https) using a custom key/certificate chain. Prerequisites MacOS Brew package manager ... minikube: installing minikube on Mac with secure TLS ingress
The Apple Virtualization Framework (AVF) provides the ability to run completely independent virtual machines on top of M family Apple Silicon. For example, you can run multiple versions of MacOS virtualized for validating an application or its dependencies against different environments. Additionally, cloning an existing VM (with little cost thanks to APFS copy-on-write) allows you ... Mac:…
Although you could use brew to install Python directly, the cleaner way to manage Python versions and isolate Python virtual environments is by using pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions of Python, while pyenv-virtualenv provides isolation of pip modules, for independence between projects. Install brew package manager Install ... Mac:…
If you are using ssh private/public keypair authentication, and get an almost immediate error like below: $ ssh -i id_rsa.pub myuser@a.b.c.d -p 22 Received disconnect from a.b.c.d port 22:2: Too many authentication failures Disconnected from a.b.c.d port 22 Then try again using the IdentitiesOnly option. ssh -o 'IdentitiesOnly yes' -i id_rsa.pub myuser@a.b.c.d -p 22 The ... Bash: fixing “Too many…
If you want a variable to reference another variable in Bash, that is possible using a concept called indirect reference. Below is a simple example where varname contains the name of another variable foo . # our variable foo=bar # our reference varname=foo # the following syntax will ERROR and DOES NOT work !!! echo "${$varname}" ... Bash: indirect reference to evaluate a variable value
If your systemd service is failing with the following error message: XXX.service: Start request repeated too quickly The first thing to do is fix any underlying issues. Use systemctl status service , journalctl -u service , and search any log files produced by the service to understand why the service failed multiple times and exceeded its StartLimitBurst. ... Ubuntu: resolving systemd error,…
The Vault Secrets Operator is a Vault integration that runs inside a Kubernetes cluster and synchronizes Vault-level secrets to Kubernetes-level secrets. This secret synchronization happens transparently to the running workloads, without any need to retrofit existing images or manifests. In this article, I will show how to: Install the Vault Secrets Operator (VSO) Configure the ... Vault:…
minikube makes it easy to spin up a local Kubernetes cluster, and adding an Ingress is convenient with its built-in Addons. In this article, I want to take it one step further and show how to use a custom key/certificate to expose a service using TLS (secure https). Prerequisites A container or virtual machine manager ... minikube: exposing a deployment using ingress with secure TLS
If there is a command that needs to run on startup/reboot, you can use the @reboot directive in cron to schedule it. Here is a simple example of logging the date to a file at startup. (crontab -l 2 /dev/null; echo "@reboot date ~/startup.log") sort -u crontab - The sort command avoids duplicate ... Bash: schedule a command that will be run at reboot using cron
In this article, I will detail how to use Vault JWT auth mode to isolate the secrets of two different deployments in the same Kubernetes cluster. This will be done by using two different Kubernetes Service Accounts, each of which generates unique JWT that are tied to a different Vault role. JWT auth mode is ... Vault: JWT authentication mode with multiple roles to isolate secrets