I have far too many conversations like the one I just got off the phone from. People frustrated with systems, frustrated with AI. From where I sit, most of these are optional problems. It doesn’t have to be this hard.
Mostly it breaks down to asking the right questions instead of the wrong ones. The frustration isn’t coming from the tool. It’s coming from the question you’re asking about the tool.
Let me show you what I mean with the biggest wrong question people bring me: “Which thing do I buy?” The right question is “What really matters here?” Watch what happens when you ask it that way. How does it feel in your body?
Please don’t struggle with tools you need to configure and troubleshoot and adapt to your workflow. Just don’t, it’s not worth it. You used to have to, but not anymore. Now you build your own. That may sound scary, but it’s less trouble, less money, and more control. And you’ll be more valuable on the other end of building it.
People selling you tools don’t tend to tell you stuff like this, but the rules have changed.
Here’s how you do it now:
- Determine what your goal is, and how you know when it’s been met.
- Decompose everything to the purest artifacts you can: plaintext, image, video, etc. You want the file types as small as possible with as rich an information set as possible.
- Build the measurement layer. Call it the “eval” or “evidence” or whatever you want, this is the thing that you count. This is what matters to you.
- Then build the tool that delivers that, and test it both ways. Test that it works like you think. Test that it fails like it should. Get as nuanced as reality gets.
- Iterate it in the real world with real conditions, in parallel to whatever your existing system is. Yes, it will report differently than your existing system. It’s supposed to. Tweak until it’s unequivocally better on every axis, then replace.
This goes for most any management system (CMS, LMS, DMS, etc.), any system where you’re juggling checkboxes that represent states. Remember the main thing those systems do is not enable access to content. That’s easy. What they do is restrict access. They keep people from seeing things or doing things.
If you just want your stuff to be available, use a web server. If you want people to find your stuff, put it in the most machine-friendly format possible, so machines can discover it. Fewer megabytes means more speed and lower cost. It really is that simple.
Keep your problems cheap, keep the layers minimal. Add security where you need to, but only as architecture (not policy).
Security is another place where the wrong question does the damage. People ask “How do I lock this system down?” when the question that matters is “What am I giving the agent that I’m not prepared to see it share?” Never give an agent information that it isn’t supposed to share. Because it will share it, and then lie and tell you it didn’t. The agent isn’t broken. Your mental model is. What agents know, they will disclose. Plan on that. People are working on this problem, but as I write this, it isn’t solved.
That was two questions upgraded. Here are the rest, sorted by where they show up.
When you’re deciding what to buy or build:
| Wrong Question | Right Question |
|---|---|
| Which thing do I buy? | What really matters here? |
| Which platform’s API should I integrate with? | Do I actually need the platform, or just the data? |
| Which model is best? | What’s the cheapest, easiest, and most reliable model for my needs? |
| How do I keep up with all the changes in the AI landscape? | What’s actually changed about what I need done? |
| Should I replace my whole system with this? | Am I running it in parallel until it’s provably better on every axis? |
| How complex should this system be? | How cheap and how few layers can I make this? |
| How do I get my team to adopt this tool? | How do I make the right way the easy way? |
| Will AI take my job? | What problem can I take ownership of? |
When you’re working with models and agents:
| Wrong Question | Right Question |
|---|---|
| How do I stop AI from hallucinating? | How do I better manage context? |
| How do I write a better prompt? | How do I give the model better context? |
| Why isn’t the agent doing what I told it to? | What did I tell the agent to work towards? |
| How do I control agents so they stop overriding each other’s work and disagreeing? | How do I better enable collaboration with and between agents? |
| How do I get AI to do this process for me? | How do I break down this process so the most important parts are easy for me? |
| Why did AI get this wrong? | Why did I rely on AI for this answer? |
| How do I know if this is true? | How would someone else disprove this? |
| How do I get AI to remember everything? | What’s actually worth persisting versus regenerating on demand? |
When you’re securing it all:
| Wrong Question | Right Question |
|---|---|
| How do I lock this system down? | What am I giving the agent that I’m not prepared to see it share? |
| How many permission levels do I need? | What am I actually trying to restrict, and from whom? |
| How do I add more security? | Who do I need to secure from what, specifically? |
| Why can’t I find anything in this system? | Is my content in a format machines can actually discover? |
| Why did this get shared? | Why is that information in here at all? |
If you need help with any of that, please let me know. I’ve been solving problems like these for a few years now. But please start without me: pick one wrong question you asked this week, and upgrade it.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.