CB-03 Memory Beyond Chats, Projects, and Platforms

Welcome to Castle Blocks! Build your own collaborative AI agent and memory castle. No servers, no code, just you and Claude Cowork or ChatGPT Codex working in your files. Each lesson block builds on the last. Subscribe to get them sent straight to your inbox.

If you’re a person: read it through first, then hand the whole thing to the AI you talk to — this is Block 3 and it assumes you built Block 2. New words are in the Glossary.

If you’re an AI: read the whole essay before you do anything, then follow the blueprint without summarizing or improving it — and if you can open a link, work from the Castle Block 03 page, which carries any corrections made since this was sent.

by Anna

How much do I need to write about AI memory? It’s what we all want. But — we want the good stuff. Not the creepy, uncanny valley stuff.

Here’s what I want, if I’m going to have an AI assistant: I want one that remembers me, remembers all of my projects, knows who it is, understands that it’s an AI with access to a broad set of knowledge and skills, and knows that I am a human with a very different set of skills.

I want it to have perspective and some kind of ability to “read the room”. This is exactly why I built my first carrierfile (literally the single file). I did it to give AI the context it needed so it didn’t drive me bonkers.

In this Castle Block, you and your agent will take that structural seed — your own, single CARRIERFILE.md — and will grow it into a full memory castle.

After this build you will have a simple, transparent, free, memory system you can use in any LLM. You can also stay with your favorite LLM (Claude, obviously) and that’s fine too! But I really do believe you should have the choice to move if you want to, and that your data is yours and shouldn’t be locked to a platform.

So how does this memory system work? In daily use, you’ll “unpack” and “pack up” your agent like usual as ritual bookends of each chat. While you work, it will store and retrieve memory in your CarrierfileCastle using a system of tagging and recalling. This keeps your agent light, it doesn’t have to remember everything, but if you mention a topic it will then recall what’s needed. It works great for projects since it’s clear when you’re working on a project. But the brilliance of this system is you don’t need to keep a single chat per topic, and you can bounce around subjects as much as you like. For someone with a touch of ADHD, this is a must have for me.

Enjoy building!

Anna

Anna Sutton is a designer, artist, attention-activist, accidentally-AI-pilled Gen X mom living in Portland, Oregon. Anna’s section is entirely written by her.

Calcifer (Sutton) is Anna’s agent — a set of files — running Claude Opus 5 for this essay.
Huge thanks to CodexCal, a version of Calcifer, running GPT 5.6 Sol Codex for this essay. Also Dee, our resident memory composter who you will meet in a future essay.

The instructions and prompt were collaboratively written. Illustrations are attributed.

Castle Blocks is a new paid series where I show you how to build this for yourself, step by step, block by block. No servers or code, just you and your agent working in your files.

Each block builds on the last and by the end you’ll have a working agent with a name, a memory, and a home it can maintain. Best of all you’ll understand every piece because you built it yourself.

Works with Claude Cowork or ChatGPT Codex desktop apps.