What AI Won't Tell You About Building with It ⚙️

AI has made entrepreneurship a lot more accessible and solving real problems a lot more interesting.

I've been building my second company, Sift Skills, with AI at its core, and that experience has taught me more about what AI can and can't actually do than any think piece ever has. My experience is also why I roll my eyes every time someone claims AI is replacing humans or gaining consciousness by next quarter.

Models improve over time, but new features drop daily, and that's where most of the noise comes from. Every time a major AI company ships something new, ten “hundreds of startups just died” tweets follow within the hour.

Here are the issues I’ve run into building Sift Skills and how I handled them:

The time you save building is the time you’ll spend testing.
AI builds fast and makes everything appear functional. That’s the facade. Whether it’s a button or a core feature, test everything end to end, exactly as a user would. Never assume AI understood your intent or implemented something correctly just because it said it did.

A simple example: AI routed our “Back” buttons to completely random pages instead of the page a user came from. That’s a small thing. Our mistake was never testing it because a 'Back' button routing to the previous page is common sense to us, so we assumed it was for AI too. Now imagine that same lack of inference applied to a core feature of your product.

Don’t fully trust AI to fix its own mistakes.
Cursor has been my tool of choice while building Sift Skills, largely because of its AI code review agent. Before I commit any code, it surfaces inconsistencies and potential issues for me to review before anything gets pushed to GitHub.

When it finds problems, there’s a “Fix All” option. Instead of me needing to go through the code and review it, it’s able to fix it for me. I suggest not using it blindly. AI doesn’t interpret intention well, and having full codebase access with all the context doesn’t change that. There are documented cases of AI agents wiping entire codebases when given unchecked autonomy. Always review what it flags before you let it touch anything.

If you're not a developer, or even if you are, you can tell your AI tool to explain every issue it finds in plain English before suggesting a fix. Make sure you understand what it’s saying broke and why before you approve AI to solve it.

Even before I merge, I give the code one last pass with a tool called Bugbot that runs through the code for any remaining conflicts that I can review before merging.

AI doesn’t consider security, and the lack of it can get you sued.
AI builds for functionality first and security is almost never a default consideration unless you explicitly ask for it. Be sure to tell your AI tool to do a security review of what it built before you move on. Search for vulnerabilities and inconsistencies, and ask it to flag anything that could expose user data or your own secret sauce. Use it as a starting point to understand what laws require, but verify anything legal with an actual source or attorney.

Do things in batches, not all at once.
Work in batches. Whether you’re writing a long prompt or tackling multiple fixes, don’t try to do everything at once. When it comes to long prompts specifically, repeat the most critical parts at the beginning and end. A prompt with too many instructions is going to lose track of something in the middle.

What I didn’t anticipate early on is that repetition costs money. More tokens means higher usage costs, and if you’re building a product that calls a model repeatedly, that adds up fast👇🏾(solution)

More instructions means more tokens, and more cost.
If you're building a product on top of AI, look into prompt caching. Think of it like a memory shortcut. Instead of AI re-reading the same instructions from scratch every single time, it saves them so it doesn't have to reprocess them. That alone can cut your costs significantly, and it speeds things up too. Less reprocessing means faster responses.

These are some of the issues I’ve run into and solutions I’ve implemented. Even with these solutions, AI can still hallucinate, be inconsistent, and break, like all software, it just does it more confidently.

1. Kentik | Senior Technical Writer | Remote in US | $110,000 – $135,000

2. Kentik | Staff Software Engineer, NMS | Remote in US | $190,000 – $225,000