Recent content on IMTI
The clearest reason to put your own hardware in the hybrid cluster is the GPU. Sustained AI work on rented cloud GPUs runs up a large bill, and a GPU you own pays for itself quickly when you actually use it. This is the final piece of the platform, and it is what makes the optional in-cluster AI, self-hosted model inference and embeddings, affordable: adding GPUs to Kubernetes, the modern way,…
The platform so far runs on a set of cloud virtual machines, and that is the right default. But some of the most useful hardware you can put a workload on is not in a cloud: a GPU box in your office, a server in a colo, devices at the edge near where data is produced, machines you already own and are not paying a cloud margin on. The cloud answer to mixing these is a separate managed cluster in…
The 2020 book had a chapter on Blockchain. It built a private Ethereum network on Kubernetes, miners and all, and it was one of the more involved chapters in the book. This rebuild does not have that chapter, and rather than quietly drop it, I want to account for why, because the reasons are a useful lesson in betting on technology and in telling the difference between a capability and a fashion.
Some work on a platform is a small piece of logic that should run on demand and cost nothing when idle: enrich a record, transform a payload, call out to a model, respond to an event. A full long-running service is too much for it, and the cloud answer is functions, Lambda and its kin, which are useful and locked to the provider. The self-hosted answer used to have an obvious open choice, and that…
Postgres is the platform’s default database, and most of the time it is the right answer. Some workloads it is the wrong answer for: ingesting a firehose of writes that never stops, scaling linearly to many nodes, staying available for writes even when a node or a whole data center goes down. That is what Cassandra was built for, and it still earns its place in 2026 for those jobs. This post…
A common cloud pattern is a file landing in a bucket that automatically kicks off some work: a CSV arrives and gets validated, an image gets a thumbnail, a log file gets compressed. The cloud does this with S3 events wired to Lambda functions, which works but locks you to one vendor. The object storage you already run can do the same thing, and the compute that responds runs as ordinary Kubernetes…
The two benchmark reports I published in July, the knowledge-layer ablation and the stored-knowledge study , both came out of the same piece of software: a benchmark harness in the bench/ directory of mcp-data-platform , roughly 36,000 lines of Go in its own module, kept out of the platform’s build on purpose. What I value most in that harness is turnaround. On July 31 I committed a probe…
The platform stores, moves, and serves a lot of data, and the people who get value out of it, analysts, data scientists, engineers, need a place to work with it: write some Python, run a query, build a model, explore. The cloud sells this as SageMaker, Vertex AI Workbench, or a Databricks seat, metered by the hour and by the user. The open answer is a multi-user JupyterHub , and on the platform…
A data platform that takes in device and sensor data needs a way to receive it that suits how those devices actually work: small, intermittent, often on flaky networks, sometimes a microcontroller with kilobytes of memory. That is what MQTT is for, and the Kafka backbone from earlier, while excellent for the platform’s internal event log, is the wrong shape for talking to a thousand sensors…
Every component in this series has been exposing metrics: Postgres, Kafka, OpenSearch, Cilium, the cluster itself, each one had a line about the monitoring stack scraping it “later.” This is later. The 2020 book treated monitoring as an afterthought, a single page pointing at a project’s README. Observability is the difference between operating a platform and hoping it is fine. I…
Agents running on mcp-data-platform do something the design never asked for: they store world state as knowledge. “Three monitors provisioned.” “Zero monitors exist yet.” These are observations, and the capture tool persists them as durable notes. State changes and notes do not, so I had a specific worry: stale observations would accumulate in the knowledge layer, and…
When you built the cluster , one cilium install gave it pod networking, replaced kube-proxy, and encrypted traffic between nodes, three jobs the 2020 edition did with three separate pieces. This post is the deeper look at why Cilium is the networking choice in 2026, what its eBPF data plane buys you, and how to use the security and observability that come with it.
When you built the cluster , the container runtime under the kubelet was containerd, with Docker nowhere in sight. That trips people up, because for a decade “containers” and “Docker” were the same word. They are not, and on a Kubernetes node in 2026 Docker has no role at all. This post is the deeper look at the runtime your whole platform actually runs on: what containerd…
Give a frontier model raw SQL access to a warehouse and ask it for last quarter’s top account by revenue. It will write clean SQL, execute it correctly, and hand you a confident answer that is wrong three different ways: it read a column of integer cents as dollars, it computed gross revenue when the house definition is net of discounts, and its “quarter” is the calendar quarter,…
The 2020 book covered building software on the platform, and it stopped there. It said so directly: deployment was out of scope. That left the most important half of the loop unwritten, because building an image is only useful if something reliably gets it running on the cluster. The answer in 2026 is GitOps, where a git repository is the source of truth for what should be running and a controller…
The Forgejo pipelines from the last post build container images, and how they do it matters more than it looks. The obvious way, the way a lot of CI still does it, is to mount the host’s Docker socket into the build job. That one shortcut hands the job root on the node, and it is exactly the kind of thing you do not want running on a cluster that holds your platform’s data. I build…
A platform needs a place to keep its code, run its pipelines, and store the images it builds. The 2020 edition ran GitLab CE on k3s for this, and that was the right call then. In 2026 I reach for something lighter and governed differently, which matters for the same reasons that run through this series: Forgejo, a self-hosted git forge that runs on the platform you have already built, backed by…
A frontier model can run SQL against everything you have built. That is not the same as understanding it. The model can query a table without knowing that cust_id holds PII, that the table was deprecated last month, or who to ask when a number looks wrong. Hand it raw access and it will confidently do the wrong thing, the way a new hire would on their first day with no one to ask. The fix is to…
The platform now holds data in a lot of places: tables in Postgres, topics in Kafka, indices in OpenSearch, and Iceberg tables in the lakehouse . No one, human or machine, can hold the meaning of all of it in their head: what each table is, what a column actually represents, where a dataset came from, who owns it. A metadata catalog answers those questions, and the best open one is DataHub . It is…
A data platform is not finished until people can see the data. Everything built so far stores, moves, and queries it; the last piece is the front end where someone who is not writing YAML builds a chart and ships a dashboard. The SaaS answer is Tableau, Looker, or Power BI, per seat, with your data pulled into their cloud. The open answer is Apache Superset, and it has grown into a Tableau…
A platform spends most of its life moving data between systems: pulling from an API, reshaping a payload, draining a queue into a table, fanning one feed out to three destinations. The SaaS answer to that is an integration platform, Workato, MuleSoft, Boomi, Fivetran, billed per task or per connector and holding your flows in their cloud. The open answer is Apache NiFi, which runs on your own…
The platform is about to grow tools that people log into: a dataflow canvas, a BI front end, dashboards. You do not want a separate username and password for each of them, and you do not want to rent identity from Okta or Auth0, billed per user, for something this foundational. One identity provider gives every tool single sign-on, central user management, and multi-factor auth from one place.…
A platform’s analytical questions, the roll-ups, joins, and slices over years of data, want a warehouse. The cloud answer is Snowflake or BigQuery: rent the storage and the compute, hand over your data, and pay on both meters forever. The open answer is a lakehouse, and it does the same job on infrastructure you own. SQL engine on top, open table format in the middle, your object storage…
A data platform needs object storage: the S3-compatible bucket layer that holds a data lake’s raw files, the destination for database and stream backups, and the storage under the lakehouse tables coming next. The 2020 edition built this on MinIO. I would not start there today. Over 2025 MinIO took its own open-source edition apart, piece by piece, so this platform uses SeaweedFS, and it…
Some questions are slow on a relational database no matter how you index them. The one I hit most: Top-K spenders by loyalty tier, bucketed by average spend, across hundreds of millions of rows of transactions spanning a decade. OpenSearch answers that in near real time with almost no special configuration, where a relational database needs crafted indexes and still strains. That, more than the…
When a platform needs to move events in real time, many producers, many consumers, durable and replayable, Kafka is the backbone. The 2020 edition ran it the only way you could back then: a Kafka cluster sitting on top of a separate Apache ZooKeeper ensemble that held its metadata. That whole second cluster is gone now. I run Kafka in KRaft mode on the storage set up earlier , managed by an…
I gave an agent one brief and pointed it at my own platform: connect as a real MCP client, exercise every tool, and find every place where what the tool says and what the tool does disagree. Review it as adversarially as possible. It came back with a list. An evaluating agent can confabulate as readily as the ones it judges, so I took none of it on the agent’s word and confirmed each item…
Every platform needs a database, and for most of what you build it should be PostgreSQL. The managed versions, RDS and Cloud SQL and the rest, are convenient, and they bill you for that convenience while tying your data to one provider’s backup format and failover behavior. I can run it on the storage I just set up instead: a highly available Postgres that fails over on its own, backs itself…
Halfway through a query session an analyst types a sentence that is worth more than the query: “timestamps before March 2024 are UTC, after that they are America/Chicago.” The agent takes the correction, fixes its WHERE clause, returns the right numbers, and the sentence is gone. The next session, with a different analyst or the same one tomorrow, starts blind and makes the same…
The 2020 edition exposed services to the outside world with the Nginx Ingress controller, a daemon on every node terminating ports 80 and 443. That controller is gone. The Kubernetes project retired ingress-nginx in 2026 after the IngressNightmare vulnerabilities made its annotation-driven configuration model untenable, and the broader industry had already moved to its successor. The Ingress API…
Agents skip the steps you want them to take. Point one at a data platform and it will reach for trino_query and write SQL against a table it never looked up, ignoring the catalog that would have told it the table was deprecated last month and that one of its columns holds PII. Put “search the catalog first” in the tool description and the model will read it, repeat it back to you, and…
The cluster from the last post can schedule containers, but it cannot yet keep anything. Pods are transient and their filesystems go with them, so the moment you want to run a database, an index, or a queue, you need storage that outlives the pod that wrote it. You could rent managed disks from your cloud, but that is exactly the per-provider lock-in this series avoids. Instead we give the cluster…
The standard way to give an agent access to three systems is to run three MCP servers. The host config lists a Trino server, a DataHub server, and an S3 server, each its own process, each its own transport, each its own auth. It works, and for a single backend it is the right shape. The moment you want a Trino result to arrive already carrying its DataHub ownership and PII tags, the fleet model…
Every platform in this series sits on one foundation: a Kubernetes cluster you own outright. Not a managed control plane you rent from a cloud and cannot pick up and move, but a cluster you stood up yourself on plain virtual machines, one that runs the same on any provider or on bare metal in a rack. That portability is the entire point. It is what keeps you out of lock-in, and it is the layer…
An MCP server is written by an agent, used by an agent, and now often evaluated by an agent, and every one of them can confabulate. The agent that builds the server can report a tool that works when it does not. The server it builds can lie to the agents that call it, with a description that overpromises and a result that misstates what it did. The agents that call it can misreport what it is…
Ninety percent of what I search for now lands me on an article that was clearly generated, never read by a human, and written from nothing. No experience, no expertise, no conviction. Just output shaped to catch a click. It is the trash of the internet, and content farms built it on purpose. So when I tell you AI touches every note on this blog, I want to be clear about what that means, because it…
In 2020, Apress published my book, Advanced Platform Development with Kubernetes . It runs about five hundred pages and builds a complete data-centric platform end to end: streaming ingestion, search and analytics, data lakes and warehouses, IoT collection, and machine learning, all assembled from best-in-class open source on Kubernetes. It is still on Amazon , and I stand by every idea in it.…
Every piece of the protocol has been on the table on its own. This post puts them in one server: a small knowledge base that exposes tools, resources, a prompt, and completion, calls back to the client with sampling and elicitation, runs over both transports, sits behind OAuth, and is tested end to end. One server, one session trace, the whole protocol in one place. Reading it should now feel like…
The first two parts of this series ran entirely on a laptop. A docker-compose brings up Postgres with pgvector and a CPU-only Ollama, the platform connects to both, and semantic search works. Moving that to a cluster sounds like it should be a formality, and mostly it is, with one exception. Every time I propose this stack, the first question is where the GPU node pool goes. It does not need one.…
The last primitive is the newest, and the only one the spec still marks experimental. Tasks, added in 2025-11-25 , turn a request into call-now-fetch-later: instead of blocking until the work is done, the receiver returns a handle immediately, the requestor polls for status, and the real result is fetched later. The Go SDK does not implement tasks yet, so this post reads the spec’s wire…
The authorization post controlled who may reach a server. It said nothing about what a server does once reached, and that is the larger problem, because a server is the one party in an MCP session whose words flow straight into the model’s context. Tool descriptions and tool results are text the server writes and the model reads. A malicious or compromised server puts instructions there.…
Over stdio the connection is trusted: the server is a local subprocess the client launched. Over Streamable HTTP it is not, and a remote server has to prove who is calling before it answers. MCP’s answer is OAuth 2.1, with the server cast as a resource server. Authorization is optional in the protocol, but when a server uses it, this is the shape. This post protects a real server, reads the…
A server has two ways to tell a client something happened without being asked. One is a log message: structured, severity-tagged output the client can filter and display. The other is a list-changed notification: a signal that the server’s tools, resources, or prompts are no longer what the client last saw. Both are server-to-client notifications, and this post reads both, the logging…
The primitives are done, the four the client calls and the three the server calls back. What is left is the plumbing they all share. Five cross-cutting utilities run underneath every method: progress reporting, cancellation, ping, pagination, and the reserved _meta field that carries some of them. None is a primitive on its own, and all five show up in real traffic, so this post reads each on the…
Elicitation is the server asking a person for input in the middle of a task. elicitation/create runs server to client to user, the third and last of the inverted requests after sampling and roots . It has two modes: a form for ordinary structured input, and a URL handoff for the sensitive data a form is forbidden to touch. This post reads both on the wire, and the rules that keep a server from…
Roots are the smallest of the server-to-client requests, and the easiest to dismiss as a convenience. They are not. A root is a directory the client tells the server it may use, and roots/list is the request the server sends to ask for the list. It runs the same inverted direction as sampling : the server asks, the client answers. What the client answers with is the boundary the server is meant to…
Every method so far has run one direction: the client asks, the server answers. Sampling runs the other way. sampling/createMessage is a request the server sends the client, asking it to run the host’s model. The server gets to use a language model without holding an API key, and the host stays in control of which model runs and whether it runs at all. This post reads the full request, the…
Generating an embedding is slow. On a CPU-only Ollama running nomic-embed-text , a single short text takes one to three seconds, and a batch of thirty-two runs somewhere between thirty and ninety seconds. That number decides the entire architecture. You cannot embed a record on the write path, because no one will wait three seconds to save a note. You cannot embed on the read path either, because…
The series has deferred this method three times. Resource templates have variables a user fills, and prompt arguments do too, and both times the note was that those values can be auto-completed. Completion is the method that does it. completion/complete suggests values for an argument as the user types, the way an IDE completes code. This post reads it on the wire, including the context field that…
Prompts are the third thing a server can offer, after tools and resources, and the one most people read wrong. A tool is invoked by the model. A resource is attached by the application. A prompt is selected by the user. It is a template the human picks from a menu, fills in, and drops into the conversation. This post reads prompts/list and prompts/get on the wire, and renders a parameterized…