Chain of Thought

You know that feeling when you solve a tough problem, and you have to slow down, think through each step, maybe even jot a few notes on paper? That’s what psychologist Daniel Kahneman calls “System 2” thinking: slow, careful, methodical. Most of the time, though, we cruise on autopilot - answering questions, making snap judgments without even realizing it. That’s “System 1.”

Believe it or not, computers and AI systems aren’t so different. Sometimes they just blurt out an answer - quick, but not always right. Other times, they work through a problem step by step, much like we do when we really need to concentrate. This step-by-step approach is what people in the AI world call “chain of thought.”

Why does this matter? Because for some problems, guessing just isn’t good enough. If you’ve ever hit a logic puzzle that stops you cold, or tried to untangle a multi-stage math question, you know: skipping the steps leads to mistakes. The same goes for AI.

This post is all about that step-by-step thinking—why it works, how it compares to our own ways of reasoning, and where it really makes a difference. We’ll keep things practical, with plenty of real examples and a look at when this approach is worth the extra effort.

Chain of thought refers to the process of generating intermediate reasoning steps—essentially, a step-by-step explanation or deduction—before arriving at a final answer. Instead of treating the problem as a black box and trying to leap straight to the solution, the model (or a human) lays out the logic, calculations, or deductions needed along the way.

This is particularly relevant for large language models (LLMs) and reasoning models. When faced with a tough question, prompting the model to “think out loud” encourages it to generate a series of logical steps, making the answer more accurate and sometimes even more interpretable.

Let’s make this concrete. Imagine a language model is given the following logic puzzle:

John is taller than Sarah. Sarah is taller than Mike. Who is the shortest?

If the model simply tries to guess the answer in a single step, it’s likely to get confused or make a mistake. But with chain of thought prompting, it can lay out intermediate steps:

• John is taller than Sarah, so Sarah is shorter than John.

• Sarah is taller than Mike, so Mike is shorter than Sarah.

• That means Mike is shorter than both Sarah and John.

• Therefore, Mike is the shortest.

This step-by-step reasoning makes the answer clear and reliable.

Now, let’s dial up the difficulty. Consider a complex math problem:

What is the value of (17 + 5) x 2 - 6 / 3?

If you try to solve this in your head without breaking it down, it’s easy to get lost. Most of us would prefer to write out each step, just as the model does with chain of thought:

• 17 + 5 = 22

• 22 x 2 = 44

• 6 / 3 = 2

• 44 - 2 = 42

Without this breakdown, both humans and machines are prone to errors. Trying to leap straight to the final answer, skipping the intermediate steps, is where mistakes happen - especially with more involved logic or arithmetic.

What’s fascinating is that advanced reasoning models are specifically trained to handle these multi-step thinking processes. Behind the scenes, when asked to solve a complex question, they’re not just guessing at the answer in a single jump. Instead, their training data includes countless examples of problems being worked out step by step. Over time, they learn to recognize the structure of reasoning and to generate those intermediate steps automatically, even if the user doesn’t explicitly prompt them for it.

This ability to construct and express a “chain of thought” doesn’t just make answers more transparent—it actually makes the underlying problem-solving much easier for the model. Without intermediate steps, the model faces the difficult task of generating the correct next token (word or number) that must implicitly capture all prior correct reasoning and calculations. This often leads to mistakes, especially as problems get more complex. By “easing the load” and letting itself generate intermediate reasoning, the model doesn’t have to keep track of everything in one step, reducing the chance of errors and often producing more reliable outcomes.

This is what sets reasoning models apart from simpler models that might only see the question and try to predict the answer in one go. By learning these logical pathways during training, they become much more robust at tackling tasks where each step builds on the last.

Let’s see what a real interaction looks like, both with and without chain of thought reasoning. Here’s a math problem and how an assistant might handle it in each case.

Prompt (chain of thought):

What is the result of (12 – 3) x 4 + 6 / 2? Please show your reasoning step by step.

Assistant response:

First, calculate 12 – 3, which is 9.
Next, multiply 9 by 4 to get 36.
Then, divide 6 by 2, which equals 3.
Finally, add 36 and 3 to get 39.
So, the answer is 39.

Now, compare this to a single-step prompt, without asking for reasoning:

Prompt (no chain of thought):

What is the result of (12 – 3) x 4 + 6 / 2?

Assistant response:

36

The difference is not just about transparency—it’s also about reliability. When the model skips intermediate steps and tries to generate the final answer in one go, it has to “compress” all the logical reasoning and calculations into each next token it generates. This makes it much more likely to make mistakes, especially as the problems become more complex. By allowing itself to “cheat” a little—breaking the problem down into smaller steps—the model reduces its own cognitive load and produces more accurate results.

Despite their advantages, reasoning models have some practical downsides. They tend to be slower and more computationally expensive than models that go straight for the answer. Generating a step-by-step explanation means more tokens, more processing, and higher costs—sometimes significantly so. For simple lookups or straightforward answers, it’s often more efficient to use a faster, cheaper model that doesn’t explicitly reason step by step.

Reasoning models shine in situations where accuracy, interpretability, or dealing with complex, multi-step logic is critical. But for routine tasks, they might just be overkill. It’s about choosing the right tool for the job: use reasoning models where their strengths matter, but don’t hesitate to pick simpler solutions when speed and cost are more important than deep reasoning.