Forget AI, do more ambitious UX

Happy Wednesday! This week: darn, I did evals with Claude!

This is my Wednesday update on UX stuff I’m learning while building Sputnik Intelligence (high quality podcast and newsletter intelligence for your agents).

Last week I wrote about building with AI and IKEA - how AI can make it easier to focus on the real underlying value for users. In IKEA’s case, that’s beautiful design that’s affordable. For Amazon it’s price, selection and fast delivery.

How can I use AI to create more value for the Sputnik Intelligence customers? We want our data to be fresh, high quality and coming from human. (I made those 3 up, still figuring out what the real 3 things are there.)

Just a few thoughts from the past days of work:

1. (kind of obvious) Really fast UI iteration and experimentation.

2. (more subtle) Being able to quickly explore tech solutions that typically would be out of reach.

I’ll give an example. We gather topics from podcast episodes and newsletter posts. They’re crazy interesting. Here’s a screenshot, for example, showing recent topics.

Being able to iterate quickly on how those topics are created, with Claude Code, made me make them much more useful. Here’s how that went:

1. I had the idea I wanted specific topics - not “elections” but “election interference”. They’re much more useful.

2. I then had AI quickly write a prompt to run that over a few hours of posts, and see what came out.

3. It was not bad, but not good enough. So then I guided Claude to help me set up a set of evals that we can repeatedly run. This was a long discussion about what makes a helpful topic versus a non-helpful topic, but using real data and the real system that was generating them.

This was key, being able to quickly set up an actual system that creates actual data, then let me work with Claude on defining What Good Looks Like for this output, and to improve it.

(This all would have been a month of work with worse outcomes if it was planned in a traditional way.)

I got real data very quickly, we had a real deep dive into What Good Looks Like, and with all that, I then asked Claude to iteratively improve the prompt:

1. Improve the prompt

2. Run the eval

3. Review, and back to 1

Within a few hours of work I had real data, tied to my best understanding of real value for our customers, with real evals.

You know the Wright brothers story? Samuel Langley, their competition, had a ton of funding and failed to launch a working airplane. The Wrights spent about $1,000 of bike-shop money, and when their 1901 gliders underperformed they stopped trusting the published lift tables (Lilienthal's data was off) and built a six-foot wooden wind tunnel to generate their own. They tested around 200 wing shapes in a couple of months that fall.

Evals are your wind tunnel.

If you have an idea of customer value, create real data with the help of AI, then create evals with AI, and figure out What Good Looks Like.

That’s just really exciting.

Health and happiness,
Peter
P.S.: Also see Fred Brooks in the Mythical Man Month: "plan to throw one away" (1975): “the first version of any system is effectively a prototype whether you plan it that way or not, so build it fast and learn from it.” We can do that in a few hours now.