The Screenshot Is the API

The screenshot is the API.

That’s the whole thesis. Everything I’ve wanted from a decade of reading tools — and everything they couldn’t deliver — comes down to whether that sentence is true. It just became true.

Here’s the itch. I read constantly — The Economist, The Information, Substacks, tweets, papers, physical books — and it all evaporates. I’ve always wanted one consolidated log of my reading: something I could look back through, search, and actually refer to. “What was that piece about crypto credit plumbing from last spring?” should be a ten-second lookup, not an archaeology project across four apps and a browser history.

Every few years a new tool promised this. Instapaper. Pocket. Readwise. Evernote’s web clipper. Each one worked, sort of, right up to the edge of its parser.

Because that’s what these tools actually were underneath: parsers. Software that knew how to extract an article from a webpage’s DOM, tuned per site, forever chasing markup changes. And a parser ties you to a medium. Links worked — as long as the site cooperated. Paywalls broke everything. A native app with no URL to share? Invisible. Video? Forget it. A screenshot? That’s a picture, not content, as far as the tool was concerned.

You were input-limited. The tool decided what counted as readable, and the answer was always “a public webpage, on a good day.”

And there was a second limit that killed these systems even when capture worked: you were summarization-limited. Saving the article was the easy half. Getting value back out meant highlighting, tagging, writing notes, and — hardest of all — remembering your notes existed. An entire industry rose around this admission: Obsidian, Roam, zettelkasten Twitter, “second brain” courses. The distillation work was a real job, and you were the only employee. So the read-it-later pile became the read-it-never pile, and the note vault became a graveyard with backlinks.

Two limits, one root cause: the intelligence lived in you, and the software only handled formats it was explicitly taught.

Both limits just disappeared, for the same reason.

A frontier multimodal model is a general-purpose parser. It reads a screenshot of anything — a webpage, a tweet, a native app, a paperback on your desk — and hands back structure: publication, title, author, full text, a clean summary, topic tags. No DOM. No format. No “sorry, we don’t support that source.” The parsing intelligence moved out of the client and into the model, and the model doesn’t care what medium you read in.

That kills the input limit. If you can see it, you can log it.

The same model also summarizes — which kills the second limit, the one Obsidian never could. Every capture arrives pre-summarized, pre-tagged, filed into a searchable log. The job you never did because it was a job is no longer a job.

But the model alone isn’t the system. The last piece is the agent.

An agent framework — I run OpenClaw, but the pattern is bigger than any one tool — wraps the model in workflows. It doesn’t just parse what you send; it acts on it. Drop a half-cropped screenshot with a partial headline, and the agent identifies the source, finds the canonical URL, and fetches the full article. Drop a paywalled link, and the agent logs into the subscription — my credentials, my server — and pulls the complete text like the paying subscriber I am. It dedupes repeats, notices what I keep coming back to, and publishes the whole log to a public page on my site every night while I sleep.

And then there’s the feature I’ve wanted longer than the log itself. Nearly every book I read gets to me the same way: someone mentions it on a podcast, or an article cites it in passing. For years the workflow was: hear the recommendation, tell myself I’ll remember it, forget it by the next stoplight. Occasionally I’d pause a podcast mid-drive to thumb a title into my notes app, where it would join a graveyard of other titles I never looked at again. What I actually wanted was obvious and impossible: something that listens to my reading for me, catches every book recommendation as it goes by, and puts the book where I’ll actually buy it.

That now exists. When something I logged recommends a book, the agent notices — pulls the title out of prose, no special markup, no “add to list” button — finds the exact edition on Amazon, and adds it to my wishlist overnight, with a note about where the recommendation came from. Think about what that sentence would have required five years ago: software that reads arbitrary articles and podcast show notes, understands that a title mentioned mid-paragraph is a purchasable object — while knowing a hedge fund named after a famous book is not — and operates a retail website on my behalf. There was no version of that as a product. It wasn’t hard; it was not possible. It’s the purest example in this whole system of a want that sat unbuildable for a decade — not because nobody thought of it, but because no parser could do the noticing. An LLM notices for a living.

Notice what capture became: nothing. A squeeze of the iPhone’s Action Button. A hotkey on the Mac. WhatsApp-ing a screenshot to my own agent like I’d text a friend. The capture layer used to be the product — companies were built and acquired on the strength of their parsers. Now capture is just transport.

And this isn’t really a story about reading logs. A personal reading log with paywall access, automatic summarization, and a public dashboard was a venture-scale product roadmap five years ago. Now it’s a side project. This is the same collapse I wrote about in Software for a Market of One — the fixed cost of turning intent into durable software falling below the value of problems only one person has. Last time the raw material was a neighborhood group chat; this time it’s the unstructured exhaust of my own reading. A reading log is market-of-one software in its purest form: exactly one user, and no parser company could ever afford to serve him properly. And reading logs are nowhere near the most interesting thing this pattern unlocks. They’re just the one I wanted first. Whole categories of software were defined by what their parsers could digest; a general-purpose model plus an agent that can run workflows is medium-blind and maintenance-free. The surface area of what one person can automate for themselves didn’t grow by some percentage. It changed kind.

That’s the story. What follows is the architecture, for the folks who want to build one. Everyone else can stop here — the log is live here, and once you see your screenshots as API calls, you cant unsee it.

Built deliberately, in staged phases, over nights and weekends. The design goals, written down before any code:

1. Two taps or less to capture, from anywhere. Friction kills logging habits.

2. Content-agnostic ingestion. A screenshot of a native app is as loggable as a clean URL.

3. Never silently drop a capture. Failures get stored and flagged, not discarded.

4. Lazy capture, agent completion. Partial input is acceptable by design; the agent finishes the record.

5. Total isolation. The reading agent gets its own workspace, database, and memory, walled off from every other agent in the deployment.

6. The agent owns the auth lifecycle. Logins, session refresh, retries, and change alerts for paywalled sites are the agent’s job, not a monthly ritual.

  iPhone Action Button ──┐
  iOS Share Sheet ───────┤        ┌──────────────────────────────┐
  Mac hotkey (Chrome) ───┼──POST──►  /hooks endpoint (auth).     │
  WhatsApp DM ───────────┘        │                              │
                                  │  readlog agent (isolated)    │
                                  │  ├─ vision extraction call   │
                                  │  ├─ URL fetch + enrichment   │
                                  │  ├─ per-domain auth registry │
                                  │  ├─ dedupe / merge           │
                                  │  └─ SQLite (own db)          │
                                  └──────────────┬───────────────┘
                                                 │ nightly cron
                                                 ▼
                              public JSON → static site → heatmap page

The capture layer is deliberately dumb: every client is just “get an image or a URL to one authenticated HTTP endpoint.” All intelligence lives server-side. Adding a new capture surface is a transport problem, never a parsing problem.

One extraction call replaces every parser. Screenshot ingestion is a single vision call returning strict JSON: publication, title, author, full text, a 2–3 sentence summary, 1–4 tags from a growing tag set, a partial-capture flag, and an attempted canonical URL. URL ingestion runs a conventional article-extraction fetch (canonical URL resolution, tracking-param stripping), with tweets enriched via the platform API. When a fetch comes back thin or blocked, the entry is stored anyway with whatever metadata is available, flagged partial, returned as HTTP 202. The failure modes are first-class citizens of the schema.

The division of labor should look familiar if you read the losaltos.space build: the model proposes, deterministic code disposes. Vision calls suggest structure; schemas, dedupe rules, and the never-drop contract decide what actually lands in the database. The model is never allowed to define the invariants — only to fill them in.

Video and audio fell too. “Video? Forget it” was the old-parser reality, so it’s worth being specific about how it died. A YouTube link now yields the full transcript — and getting it was a small war story. YouTube gates caption data behind a proof-of-origin token minted client-side by the watch page’s own JavaScript; you cannot just fetch the caption URL, even with valid cookies — the server returns 200 with an empty body. So the agent does what a person would: opens the real watch page in its browser, clicks “Show transcript,” and intercepts the network response carrying the caption data. (The final unlock was a single Chromium launch flag — --disable-blink-features=AutomationControlled — the entire difference between silent empty responses and an 18,000-character verbatim transcript.) Videos without captions degrade gracefully to description-based summaries, tagged so the log knows the difference. Apple Podcasts was gentler: episode pages look like 49 characters of JavaScript shell to a plain fetch, but every one embeds a JSON blob with full episode metadata and show notes — extract that, and an episode logs as cleanly as an article. Same pipeline, same schema, no special cases downstream.

Dedupe as a signal. Same canonical URL or publication+title within 24 hours merges into one row — across modalities too, so a screenshot and a URL capture of the same article converge, each filling the other’s gaps. The merge increments a capture_count, which quietly became a metric I didn’t design: re-capture frequency is a proxy for what I keep coming back to. Merge logic turned into attention analytics for free.

A per-domain auth strategy registry. The capability no prior reading tool could offer: the agent logs into my subscriptions itself. Each site registers {isLoggedOut, login}, keyed by domain, with a persistent headless-browser profile per domain. Adding a publication is one strategy file and two lines of registration — a registry, not a hairball. My three subscriptions turned out to be a neat taxonomy of login mechanics: The Economist is a classic email+password form (the base case); The Information adds a possible emailed verification code; Substack has no password at all — magic links only. (No, Substack doesn’t offer authenticated RSS for paid written posts; the private-feed mechanism is podcast-only. I checked before building.)

Two of the three require the agent to read an email mid-login — and that’s one capability, not two. A generic awaitSignInEmail helper, parameterized by Gmail query, a cutoff timestamp, and a pattern that extracts the link or code, serves both. The email inbox is an auth API. It always was one; we just don’t usually let software read it. Once your agent has mailbox access, every “we sent you a code” flow on the internet becomes automatable.

The obvious objection: the agent holds my credentials. Correct, and deliberate — they live in the deployment’s existing secrets store, server-side, and I chose full automation over credential-isolation purity. Your threat model may differ; mine already trusts this box with more sensitive things than a magazine login.

Real Chromium on a box where you can’t install anything. Paywalled fetching needs a real browser (bot-detection rejects plain HTTP clients), but the VPS is an unprivileged container — no root, no package manager. The fix is a fully userland browser runtime: resolve Chromium’s ~113 system-library dependencies via the package manager’s URL-listing mode, extract them into a local tree, inject via LD_LIBRARY_PATH. A portable Debian userland, ~880MB all-in with the browser, touching nothing system-wide. RAM is the scarcer resource, so the design goal was zero cost at idle: a fresh browser launches per fetch and dies after, serialized through a single-flight queue. A fetch transiently costs ~800MB; between fetches, nothing. Worth it? A logged-out fetch of a premium article extracts ~2,400 characters before the gate; authenticated, 7,000–9,000. That’s the difference between logging a teaser and logging the article.

The wishlist pipeline: the log acts on what I read. The mechanics behind the book-recommendation catcher, since it crosses the line from recording reading to acting on it. A nightly job sweeps unprocessed entries for book recommendations — not books I read, books the things I read told me to read. An LLM proposes candidates with confidence levels and the exact snippet that triggered them; deterministic gates dispose. Only high-confidence candidates proceed. Only exact title matches on Amazon get added — ambiguity is a refusal, not a guess. Adds are idempotent, land on one designated wishlist with a note about the source, and the wishlist URL is written back onto the entry that triggered it, so the whole chain is auditable. First real sweep: thirty-three entries, zero false positives — including correctly refusing to buy a book because a hedge fund happens to share its name. It’s the same proposer/disposer split as everything else here, pointed at a new job: the read-it-later pile used to be where recommendations went to die, and now the pipeline quietly closes that loop while I sleep.

The agent watches itself. Automated logins fail eventually — most likely from a redesigned login page, which no retry logic fixes. So the system surfaces change instead of failing silently: after two failed attempts, the entry stores as partial and the agent sends exactly one WhatsApp alert per domain per day, including a screenshot of the stuck page — a picture of what changed beats a stack trace. Every attempt lands in an auth_events audit table (never credential values), and a nightly health check fetches one known subscriber-only URL per domain. I learn about session rot from a cron job, not from a three-week hole in my log. Transient failures get a second chance on their own schedule: a retry job re-runs partial entries whose failure looks temporary — a bot-detection challenge, a timeout — on an exponential backoff, reusing the dedupe layer’s capture_count as the retry counter instead of adding state. Structural failures (an expired login, a broken magic-link flow) are deliberately excluded: those already fired an alert, and blind retries would just hit the same wall. A pipeline you can’t observe is a pipeline that’s already broken; you just haven’t been told.

The publish loop. A nightly job exports the database to a public-safe JSON snapshot — full text, images, and internal fields stripped; titles, summaries, tags, and aggregates kept — and commits it to my website’s repo, skipping no-op commits. A static page renders it (activity heatmap, tag filters), hosted entirely on GitHub Pages. The front end was built by a second coding agent (Claude Code) working against the JSON contract while the pipeline agent owned the server side. Two agents, one interface — neither needed to know how the other worked. I’d default to this now: define the data contract first, then let different agents own each side.

None of the load-bearing pieces are exotic. The pattern needs four primitives: an authenticated HTTP trigger (so dumb clients can hand work to the agent), a multimodal model (the general-purpose parser), a scheduler (health checks, publish loop), and a messaging channel (universal capture fallback + alerting surface). I built on OpenClaw because I already run it and it ships all four — but swap in any agent runtime with those primitives and the architecture ports unchanged. The strategy registry, the email-as-auth helper, the never-drop ingestion contract are design, not framework features. The model does the parsing; the framework just moves bytes and keeps time.

Legacy reading tools were parsers with a UI, bounded by what their parsers could digest. This system has no parser to maintain. Make the capture layer dumb, make the agent smart, and treat “the model can read anything” as an architectural assumption rather than a demo trick.

And the caveat from last time still holds: the model got radically more capable, but the engineering didn’t get optional. The auth registry, the audit table, the health checks, the alert-with-a-screenshot — that unglamorous scaffolding is the difference between a capture demo that works once while you’re watching and a log that’s still quietly filling itself in on a random Wednesday.

If you’re building on this pattern — or you’ve pushed it somewhere further — I’d genuinely like to hear it. Always happy to compare notes.