Recent content on (p)retired
I bought a friend an Atari VCS Wireless Joystick as a Christmas gift thinking ( incorrectly ) it would work with an Atari 2600+ . My only defense is that, I didn’t realize the VCS is different, the joystick looks the same and, I’m stupid for not reading the fine print. I subsequently bought my friend the correct, CX40+ . I am intrigued to see whether I can now repurpose the joystick as…
I have a new router that runs OpenWRT and it provides a Prometheus (Node) Exporter (written in Lua ) NOTE The prometheus-node-exporter-lua Exporter package is a 4KB (!) script As I’m sure is a common experience for others, it’s great that there’s an Exporter and a wealth of metrics exposed by it. One problem is that there’s no guidance on how to use these metrics to…
A simple deployment of cAdvisor to K3s and to confirm the ability to expose Ingresses using Tailscale Kubernetes Operator (TLS) and – since it’s already installed with K3s – Traefik (non-TLS). local image = "gcr.io/cadvisor/cadvisor:latest" ; local labels = { app: "cadvisor" , }; local name = std.extVar( "NAME" ); local node_ip = std.extVar( "NODE_IP" ); local port = 8080 ; local…
I’m migrating from MicroK8s to K3s . This post explains the installation and (re)configuration steps that I took including verification steps: Install K3s Use standalone kubectl and update ${KUBECONFIG} ( ~/.kube/config ) Install System Upgrade Controller Install kube-prometheus stack (including the Prometheus Operator) Disable Grafana and Node Exporter Tweak default scrapeInterval Install…
A Stack overflow question involving transcoding HTTP/JSON to gRPC, piqued my interest. I had a hunch on the solution but was initially dissuaded from attempting a repro because of the complexity: recreate proto implement stubs deploy gRPC-Gateway I then realized that FauxRPC would probably address much of the complexity and it did. I created foo.proto : syntax = "proto3" ; import…
The “W” signifies that the board includes wireless. Interestingly, the wireless chip is an Infineon CYW43439 ) which is itself a microcontroller running its own ARM Cortex chip (M3). The Pico’s USB device includes another ARM microcontroller. So, with the dual Cortex (or Hazard) chips that are user programmable, and the 8 PIOs, these devices really pack a punch. As a result of…
This one works! Virtual WS2812s diagram.json I’d gone cough many years and never heard of 1-Wire and, suddenly, it’s everywhere. Addressable LEDs are hugely popular in tinkerer circles. Addressable LEDs come in myriad forms (wheels, matrices) but commonly they’re sold as long strips. The part number is WS2812 and they use 1-Wire too. Each, often multi-color (RGB) LED (often known…
I’ve been working through Google’s Comprehensive Rust and, for the past couple of weeks, the Bare Metal Rust standalone course that uses the (excellent) micro:bit v2 that has a Nordic Semiconductor nRF52833 (an ARM Cortex-M4; interestingly its USB interface is also implemented using an ARM Cortex M0). There’s a wealth of Rust tutorials for microcontrollers and I bought an…
Update 2025-07-08 Gemini CLI supports HTTP-based MCP server integration So, it’s possible to replace the .gemini/settings.json included in the original post with: { "theme" : "Default" , "mcpServers" : { "ackal-mcp-server" : { "httpUrl" : "http://localhost:7777/mcp" , "timeout" : 5000 }, "prometheus-mcp-server" : { "httpUrl" : "https://prometheus.{tailnet}/mcp" , "timeout" : 5000 } },…
Solved! Ugh. Before I continue, one important detail from yesterday’s experience which I think I didn’t clarify is that, unlike the Copilot agent, it appears (!?) that Gemini agent only supports integration with MCP servers via stdio. As a result, the only way to integrate with HTTP-based MCP servers (local or remote) is to proxy traffic through stdio as mcp-remote and the Rust example…
Former Microsoftie and Googler: Good documentation Extend your agent with Model Context Protocol Not such good documentation: Using agentic chat as a pair programmer Definition of “good” being, I was able to follow the clear instructions and it worked first time. Well done, Microsoft! This space is moving so quickly and I’m happy to alpha test these companies’ solutions but…
I couldn’t summarize this in a title (even with an LLM’s help): I wanted to: Run a Tailscale service discovery agent On a Tailscale node outside of the Kubernetes cluster Using Podman Quadlet Accessing it from the Kubernetes Cluster using the Tailscale’s egress proxy Accessing the proxy with a kube-prometheus ScrapeConfig In order that Prometheus would scrape the container for…
I was unable to find a Model Context Protocol (MCP) server implementation for Prometheus . I had a quiet weekend and so I’ve been writing one: prometheus-mcp-server . I used the code from the MCP for gRPC Health Checking protocol that I wrote about previously as a guide. I wrote a series of stdin and HTTP tests to have confidence that the service is working correctly but I had no MCP host. I…
Model Context Protocol (MCP) is “all the rage” these days. I stumbled upon protoc-gen-go-mcp and think it’s an elegant application of two technologies: programmatically generating an MCP server from a gRPC protobuf. I’m considering building an MCP server for Ackal but, thought I’d start with something simple: gRPC Health Checking protocol . I was surprised to learn as…
I’m building an emulator for Cloud Run . As I considered the solution, I assumed (more later) that I could implement Google’s gRPC interface for Cloud Run and use Envoy to proxy HTTP/REST requests to the gRPC service using Envoy’s gRPC-JSON transcoder . Google calls this process Transcoding HTTP/JSON to gRPC which I think it a better description. Google’s Cloud Run v2 ( v1…
I have built Prometheus Exporters for multiple cloud platforms to track resources deployed across clouds: Prometheus Exporter for Azure Prometheus Exporter for crt.sh Prometheus Exporter for Fly.io Prometheus Exporter for GoatCounter Prometheus Exporter for Google Analytics Prometheus Exporter for Google Cloud Prometheus Exporter for Koyeb Prometheus Exporter for Linode Prometheus Exporter for…
I’m a little obsessed with creating Prometheus Exporters: Prometheus Exporter for Azure Prometheus Exporter for crt.sh Prometheus Exporter for Fly.io Prometheus Exporter for GoatCounter Prometheus Exporter for Google Cloud Prometheus Exporter for Koyeb Prometheus Exporter for Linode Prometheus Exporter for PorkBun Prometheus Exporter for updown.io Prometheus Exporter for Vultr All of these…
Obsessing on gRPC Firestore Listen somewhat but it’s also a good learning opportunity for me to write stuff in Rust. This doesn’t work against Google’s public endpoint (possibly for the same reason that gRPCurl doesn’t work either) but this does work against the Go server described in the other post. I’m also documenting here because I always encounter challenges…
A question on Stack overflow Firestore gRPC Listen does not send deletions piqued by interest but created a second problem for me: I’m unable to get its Listen method to work using gRPCurl . For what follows, you will need: a Google Project with Firestore enabled and a database (default: (default) ) containing a Collection (e.g. Dogs ) with at least one Document (e.g. freddie ) googleapis…
After recently discovering FauxRPC , I was sufficiently impressed that I decided to use it to test Ackal ’s gRPC services using rust. FauxRPC provides multi-protocol support and so, after successfully implementing the faux gRPC client tests, I was compelled to try gRPC-Web too. For no immediate benefit other than, it’s there, it’s free and it’s interesting. As an aside, the…
Read FauxRPC + Testcontainers on Hacker News and was intrigued. I spent a little more time “evaluating” this than I’d planned because I’m forcing myself to use rust as much as possible and my ignorance (see below) caused me some challenges. The technology is interesting and works well. The experience helped me explore Testcontainers too which I’d heard about but not…
Cloud Run supports gRPC startup|liveness probes which I’d not used before. I’m using Cloud Run v2 and specifically projects.locations.services.create and Service PROJECT = "..." REGION = "..." REPO = ".." # Must be in an Artifact Registry repo IMAGE = " ${ REGION } -docker.pkg.dev/ ${ PROJECT } / ${ REPO } /..." # Run v2 ENDPOINT = "https://run.googleapis.com/v2" PARENT = "projects/ ${…
I build (and therefore) manage many container images. It’s easy (common?) to overlook that these images contain vulnerabilities, hopefully vulns that are fixed and that the images must be rebuilt to accommodate these changes. I have used Google’s very expensive container vulnerability scanning tool but wanted something cheaper. I found this list of open source solutions on Reddit and…
A lazy Sunday afternoon and my interest was piqued by XML-RPC Client A very basic XML-RPC client wrapped in a Cloud Functions function: main.py : import functions_framework import os import xmlrpc.client endpoint = os . get_env( "ENDPOINT" ) proxy = xmlrpc . client . ServerProxy(endpoint) @functions_framework.http def add (request): print(request) rqst = request . get_json(silent = True ) resp =…
For the first time, I chose Rust to solve a problem. Until this, I’ve been trying to use Rust to learn the language and to rewrite existing code. But, this problem led me to Rust because my other tools wouldn’t cut it. The question was how to represent oneof fields in Kubernetes Custom Resource Definitions (CRDs) . CRDs use OpenAPI schema and the YAML that results can be challenging to…
An interesting question on Stack overflow prompted me to understand how to use Visual Studio Code and Delve to remotely debug a Golang app running on Kubernetes (MicroK8s). The OP is using Gin which was also new to me so the question gave me an opportunity to try out several things. Sources A simple healthz handler: package main import ( "flag" "log/slog" "net/http" "github.com/gin-gonic/gin" )…
This is so useful that it’s worth its own post. I write many gRPC services. As these generally run securely, it’s best to test them that way too but, even with e.g. Let’s Encrypt , it can be challenging to generate appropriate TLS certs. Tailscale makes this trivial. Assuming there’s a gRPC service running on localhost:50051 , we want to avoid -plaintext : PORT = "50051"…
General You’ll need a Google Cloud project with Cloud Translation ( translate.googleapis.com ) enabled and a Service Account (and key) with suitable permissions in order to run the following. BILLING = "..." # Your Billing ID (gcloud billing accounts list) PROJECT = "..." # Your Project ID ACCOUNT = "tester" EMAIL = " ${ ACCOUNT } @ ${ PROJECT } .iam.gserviceaccount.com" ROLES =(…
I’ve written before about Ackal ’s use of Firestore and subscribing to Firestore document CRUD events: Routing Firestore events to GKE with Eventarc Cloud Firestore Triggers in Golang using Firestore triggers I find Google’s Eventarc documentation to be confusing and, in typical Google fashion, even though open-sourced, you often need to do some legwork to find relevant sources,…
I naively (!) began exploring JSON marshaling of Protobufs in rust. Other protobuf language SDKs include JSON marshaling making the process straightforward. I was to learn that, in rust, it’s not so simple. Unfortunately, for me, this continues to discourage my further use of rust (rust is just hard). My goal was to marshal an arbitrary protocol buffer message that included a oneof feature.…
Interested to explore Fly Kubernetes after being accepted into the closed beta. The folks at Fly are innovative in their technology uses and, having been a long-time Kubernetes user, I was intrigued to learn that Fly.io has implemented Kubernetes atop Fly . My first Deployment failed: Authentication required to access image "ghcr.io/{image}" It was confirmed to me that FKS does not support pulling…
I responded to a question Prometheus metric protocol buffer in gRPC on Stackoverflow and it piqued my curiosity and got me yak shaving. Prometheus used to support two exposition formats including Protocol Buffers, then dropped Protocol Buffer and has since re-added it (see Protobuf format ). The Protobuf format has returned to support the experimental Native Histograms feature. I’m…
Spent time today yak-shaving which resulted in an unplanned migration from MicroK8s ‘prometheus’ add-on to the new and not fully-documented ‘observability’ add-on: sudo microk8s.enable prometheus Infer repository core for addon prometheus DEPRECATION WARNING: 'prometheus' is deprecated and will soon be removed. Please use 'observability' instead. ... The reason for the name…
I wrote about Navigating Koyeb’s Golang SDK . That client is generated using the OpenAPI Generator project using Koyeb’s Swagger (now OpenAPI) REST API spec . This post shows how to generate a Rust SDK using the Generator and provides a very basic example of using the SDK. The Generator will create a Rust library project: VERS = "v7.2.0" PACKAGE_NAME = "koyeb-api-client-rs"…
This Stackoverflow question piqued my interest: retry policy configuration for grpc not working Service Config in gRPC is new to me but, my initial suspicion (albeit incorrect) was that the JSON types were incorrect. I decided to try using the Protocol Buffer source service_config.proto to verify the JSON. To do so I needed to compile the source…. it was gnarly. There are 2 repos used:…
Ackal deploys gRPC Health Checking clients in locations around the World in order to health check services that are representative of customer need. Koyeb offers multiple locations and I spent time today writing a client for Ackal to integrate with Koyeb using the Golang client for the Koyeb API . The SDK is generated from Koyeb’s OpenAPI (nee Swagger) endpoint using openapi-generator-cli .…
The deployment of Kube State Metrics for Google Managed Prometheus creates both a PodMonitoring and ClusterPodMonitoring . The PodMonitoring resource exposes metrics published on metric-self port (8081). The ClusterPodMonitoring exposes metrics published on metric port (8080) but this doesn’t include cronjob -related metrics: kubectl get clusterpodmonitoring/kube-state-metrics \ --output =…
Referring to Accessing Google Services using gRPC , I wanted to query a project’s Cloud Logging for log-based metrics using gRPC. In summary: ENDPOINT = "logging.googleapis.com:443" ROOT = "/path/to/googleapis" # https://github.com/googleapis/googleapis PACKAGE = "google/logging/v2" # NB Not logging.proto PROTO = " ${ ROOT } / ${ PACKAGE } /logging_metrics.proto" TOKEN = $( gcloud auth…
CRD linting Returning to yesterday’s failing tests, it’s unclear how to introspect the E2E tests. kubectl get namespaces NAME STATUS AGE ... allns-s2os2u-0-90f56669 Active 22h allns-s2qhuw-0-6b33d5eb Active 4m23s kubectl get all \ --namespace = allns-s2os2u-0-90f56669 No resources found in allns-s2os2u-0-90f56669 namespace. kubectl get all \ --namespace = allns-s2qhuw-0-6b33d5eb NAME…
For ackalctld to be deployable to Kubernetes with Prometheus Operator , it is necessary to Enable ScrapeConfig to use (discovery|target) proxies #5966 . While I’m familiar with Kubernetes, Kubernetes operators ( Ackal uses one built with the Operator SDK ) and Prometheus Operator , I’m unfamiliar with developing Prometheus Operator. This (and subsequent) posts will document some…
TL;DR Enable ScrapeConfig to use (discovery|target) proxies I’ve developed a companion, local daemon (called ackalctld ) for Ackal that provides a functionally close version of the service. One way to deploy ackalctld is to use Kubernetes and it would be convenient if the Prometheus metrics were scrapeable by Prometheus Operator . In order for this to work, Prometheus Operator needs to be…
Yet another cloud platform exporter for resource|cost management. This time for Koyeb with Koyeb Exporter . Deploying resources to cloud platforms generally incurs cost based on the number of resources deployed, the time each resource is deployed and the cost (per period of time) that the resource is deployed. It is useful to be able to automatically measure and alert on all the resources deployed…
Responded to Get Custom K8s Resource using Python and found the CustomObjectsApi documentation unclear. If you have a cluster and a kubeconfig file with a correctly configured current-context , so that you can successfully: PLURAL = "checks" kubectl get ${ PLURAL } \ --all-namespaces NOTE I’m using Ackal ’s CRDs in these examples. Then you can use the following code to access the…
Thanks to Joe Mooring for his solution to my question as to how to do this. I’ve decided to ditch Google Analytics and am evaluating using GoatCounter with Hugo with Ananke theme. This was the layout of the site: . ├── archetypes ├── content │ └── posts ├── go.mod ├── go.sum ├── hugo.toml ├── layouts ├── public └── static This is the structure (of layouts ) with the necessary changes: . ├──…
Python’s Protocol Buffers code-generation using protoc has had significant changes that can cause developers… “challenges”. This post summarizes my experience of these mostly to save me from repreatedly recreating this history for myself when I forget it. Version change Generated code change Implementation Backends I’ll use this summarized table of proto and the Pypi…
Google announced Firestore … integration with Eventarc . Ackal uses Firestore to persist Customer and Check information and it uses Google Cloud Firestore Triggers to handle events on these document types. Eventarc feels like the strategic future of eventing in Google Cloud and I’ve been concerned since adopting the technology that Google would abandon Google Cloud Firestore Triggers.…
I’ve been running a DigitalOcean Apps static site for Hugo using the Hugo Buildpack . I’ve migrated a set of Hugo sites to use Hugo Modules which includes the addition of go.mod (and go.sum ) files to the Hugo project in order to manage e.g. themes. Unfortunately, the Hugo Buildpack used by DigitalOcean Apps does not support Hugo Modules. DigitalOcean support recommended that I switch…
I’ve been spending time recently optimizing Ackal ’s use of Google Cloud Logging and Cloud Monitoring in posts: Filtering metrics w/ Google Managed Prometheus Kubernetes metrics, metrics everywhere Google Metric Diagnostics and Metric Data Ingested Yesterday, I read that Robusta has a new open source project Kubernetes Resource Recommendations (KRR) so I took some time to evaluate it.…
I’ve been on an efficiency drive with Cloud Logging and Cloud Monitoring . With regards Cloud Logging, I’m contemplating (!) eliminating almost all log storage. As it is I’ve buzz cut log storage with a _Default sink that has comprehensive sets of NOT LOG_ID(X) inclusion and exclusion filters. As I was doing so, I began to wonder why I need to pay for the storage of much logging.…
I’ve written Prometheus Exporters for various cloud platforms. My motivation for writing these Exporters is that I want a unified mechanism to track my usage of these platform’s services. It’s easy to deploy a service on a platform and inadvertently leave it running (up a bill). The set of exporters is: Prometheus Exporter for Azure Prometheus Exporter for Fly.io Prometheus…