Most teams adopting reasoning models make the same mistake: they turn on “thinking mode” for everything and wonder why their costs went up 8× while quality declined for half their traffic.
preview: preview
Guide: Guide link
Apply coupon code below 100% FREE for paid subscribers 👇👇👇
The uncomfortable truth is that reasoning is not free. Every chain-of-thought token is a computation you’re paying for. For a simple extraction task like pulling an invoice number out of a sentence, that deliberation is pure waste. The model already knew the answer on the first forward pass.
But for a multi-step arithmetic problem with constraints? That deliberation is the difference between a right answer and a confidently wrong one.
The question isn’t whether we should reason. It’s when.
We built a lab to make this visible.
Lab 1 runs the same user prompt through two paths, side by side:
Fast path. A single LLM call. No decomposition, no tools, no verification. The model sees the prompt and immediately gives its best shot.
Deliberate path. The full reasoning pipeline. Decompose the problem into sub-goals, plan execution order, solve each sub-problem with bounded tool use, verify results, and compose a final answer.
Both paths run on the same model. The only difference is the orchestration around it.
When you hit “Run,” the lab streams every reasoning step to the browser in real time. You don’t stare at a spinner for 10 seconds and then get a JSON dump. Instead, you watch the system think:
● [Start] Starting Lab 1: Fast path vs. deliberate path 0ms
● [Fast Path] Running fast path, single forward pass… 2ms
✓ [Fast Path] Fast path complete 380ms
→ "INV-2025-4471" | 312 tokens | 380ms
● [Decompose] Decomposing problem into sub-goals… 390ms
✓ [Decompose] decomposition attempt 1 820ms
✓ [Reason] reason (node: extract_number) 1200ms
✓ [Tool] tool call: validate_format 1450ms
✓ [Reason] reason (node: confirm_context) 1800ms
✓ [Answer] composition 2200ms
✓ [Complete] Deliberate path complete 2200ms
→ "INV-2025-4471" | 4,200 tokens | 2.2s
Same answer. 13× the tokens. 6× the latency. For an extraction task, the fast path was already correct, and the deliberate path just burned money, agreeing with it.
Now run a harder prompt:
“A 4,250 unit order carries an 18% volume discount and then a 2.5% handling surcharge. What is the net unit count charged?”
● [Fast Path] Fast path complete 420ms
→ "3,572 units" ← WRONG
● [Decompose] Decomposed into 2 sub-problems across 2 layers 910ms
✓ [Reason] reason (node: s1, compute discount) 1300ms
✓ [Tool] calculator: 4250 * 0.82 = 3485 1500ms
✓ [Reason] reason (node: s2, apply surcharge) 1900ms
✓ [Tool] calculator: 3485 * 1.025 = 3572.125 2100ms
✓ [Answer] composition 2500ms
→ "3,572.125 units (3,572 units rounded)" ← CORRECTThe fast path hallucinated. The deliberate path decomposed, used a calculator tool for the arithmetic, and got the right answer. That is when deliberation pays for itself.
fast_answer, usage = await text_call(prompt)That’s it. text_call sends the prompt to the LLM and returns the response. No ceremony. The model either knows the answer or it doesn’t.
The deliberate path is orchestrated by solve_hybrid(), a function that provides the system with structure at the top (via planning) and resilience at the bottom (via bounded ReAct loops within each node).
Stage 1: Decompose into a DAG
dag = await decompose(goal, router.names(), traj)The LLM produces a structured Decomposition, which is a list of sub-problems with explicit dependencies:
{
"sub_problems": [
{"id": "s1", "question": "Compute 18% discount on 4,250",
"kind": "compute", "depends_on": []},
{"id": "s2", "question": "Apply 2.5% surcharge to result of s1",
"kind": "compute", "depends_on": ["s1"]}
],
"global_constraints": ["All arithmetic must use the calculator tool"]
}The decomposition is validated using Kahn’s algorithm to ensure it’s a DAG (no cycles) and that every dependency references a real node. If the LLM produces garbage, the system repairs it once, then degrades to a single-node fallback rather than crashing.
Stage 2: Execute Layers in Parallel
for layer in dag.execution_layers():
outs = await asyncio.gather(*[
react_node(node, results, dag.global_constraints,
router, traj, max_steps=4)
for node in layer
])Independent sub-problems run concurrently. s1 has no dependencies, so it runs immediately. s2 depends on s1, so it waits in the next layer.
Stage 3: Bounded ReAct per Node
Each sub-problem gets its own ReAct loop. Critically, it’s bounded:
async def react_node(node, established, constraints,
router, traj, max_steps=4):
for _ in range(max_steps): # will NOT run forever
text, usage = await text_call(context)
if "ANSWER:" in text: # model solved the sub-problem
return text.split("ANSWER:", 1)[1].strip()
if "TOOL:" in text: # model wants to use a tool
name, args = _parse_tool_call(text)
result = await router.call_with_retry(name, args, traj)
context += f"\nOBSERVATION: {observation}"The max_steps=4 cap is the critical detail. Without it, a model that calls the wrong tool will retry indefinitely, reintroducing exactly the runaway cost that planning was supposed to prevent.
If a node’s output fails its success predicate (for example, it’s supposed to return a number but returned prose), the system replans that subtree instead of failing the entire run.
Stage 4: Compose Under Constraints
return await compose(goal, results, dag.global_constraints, traj)The LLM sees all sub-results and global constraints and writes the final answer. It’s told explicitly: “Do NOT perform arithmetic here. If a number is needed and not present above, say so.” This prevents the composition step from silently inventing numbers.
One design choice worth calling out: the budget is enforced externally, not by the model:
class Budget(BaseModel):
max_steps: int = 12
max_tokens: int = 24_000
max_wall_ms: int = 60_000
max_tool_calls: int = 8
def exceeded(self, traj) -> str | None:
if len(traj.steps) >= self.max_steps:
return f"step budget exhausted ({self.max_steps})"
...After each layer, the orchestrator checks whether we have exceeded the budget. If yes, it returns a partial answer with an honest caveat:
“I stopped before finishing because the token budget was exhausted. The remaining sub-problems were not attempted, so treat the above as incomplete rather than as a conclusion.”
Why external? Because a model asked to police its own budget keeps deliberating. That’s the behavior reinforcement learning on outcome reward selects for. More thinking usually means higher reward, so the model never wants to stop. The budget must come from the infrastructure, not the prompt.
Before this refactor, the lab ran the entire computation server-side and returned one massive JSON blob. The user stared at a spinner for 5 to 60 seconds.
Now the backend is an async generator that yields events at each checkpoint:
async def lab_01_stream(prompt, cfg, tenant):
p = LabProgress()
yield p.emit("fast_path", "Running fast path…")
fast_answer, usage = await text_call(prompt)
yield p.emit("fast_path", "Fast path complete",
status="completed",
data={"answer": fast_answer, "tokens": ...})
yield p.emit("decompose", "Decomposing problem…")
deep_answer = await solve_hybrid(prompt, router, deliberate, budget)
for step in deliberate.steps:
yield p.emit(step.kind.value, step.thought,
status="completed", data={...})
yield p.emit("done", "Lab 1 complete",
status="completed", data={...full_result...})
The FastAPI endpoint wraps this in a StreamingResponse with NDJSON (one JSON object per line):
@reasoning_router.post("/labs/stream")
async def stream_lab(req, user):
async def event_stream():
async for event in runner(req.prompt, req.config, tenant):
yield event.to_ndjson()
return StreamingResponse(event_stream(),
media_type="application/x-ndjson")On the frontend, a React hook consumes the stream and updates a timeline component as each event arrives:
const reader = res.body.getReader();
for (;;) {
const { done, value } = await reader.read();
if (done) break;
// Parse each NDJSON line and append to state
const event: LabEvent = JSON.parse(line);
setState(prev => ({
...prev,
events: [...prev.events, event],
}));
}Each event renders as an animated card on a vertical timeline, color-coded by phase with expandable detail panels.
Reasoning models are not universally better. They are a compute-quality tradeoff that you must manage explicitly:
Easy tasks (extraction, classification, formatting): the fast path is faster, cheaper, and equally correct. Deliberation is a waste.
Hard tasks (multi-step arithmetic, constraint satisfaction, multi-hop questions): the fast path hallucinates. Deliberation is the fix.
The hard part is knowing which bucket a question falls into before you’ve answered it. Chapter 8 (test-time compute scaling) addresses this, but it starts with seeing the gap, which is what Lab 1 is for.
The interactive lab makes this visceral. You don’t just read about it. You watch the fast path fail, and the deliberate path succeed, step by step, in real time. Then you look at the cost and decide whether it was worth it.
That decision, when to think and when to just answer, is the fundamental engineering question of reasoning model system design.
Demo Video
Read more 9 Labs explanation from this Guide: Guide link
Book a call with us, a high-impact consultation for engineers serious about landing top AI, ML, RAG Engineering, GenAI, and MLOps roles. Book a call
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.