Learning Agentic AI by Building One

I built an agent for work some time ago, but the field moves fast, and I wanted to build something with the current tooling and see where things are at. So I spent a while reading about agentic AI to get a better grasp of the field.

Best way to learn is by doing, so I built (or more like vibed) airflow-agent to try out some of the concepts I’d read about. It monitors Apache Airflow, investigates failed DAG runs using an LLM, diagnoses root causes, and queues remediation actions for human approval. This isn’t meant to be the definitive or most cutting-edge approach. It reflects what I found to be the most common patterns when building these tools in code, skipping the low-code route (n8n, CrewAI, and similar).

What It Does⌗

Apache Airflow is a tool for scheduling and running data pipelines. Each pipeline is a DAG (a graph of tasks that run in order). When a task fails, someone has to dig into the logs, figure out what happened, and decide whether to retry. The agent automates that investigation.

When a failure is detected, the agent pulls task logs and DAG metadata from Airflow’s API, reasons over them with an LLM, produces a structured diagnosis, and writes it to a dashboard. If the recommendation is a retry, a human has to confirm it first. The agent then executes or skips.

Once an investigation completes, the result lands in the dashboard with the failing task, a diagnosis, and a recommended action.

LangGraph: A State Machine for Agents⌗

LangGraph is a Python library for building stateful, multi-step AI workflows as directed graphs. You define nodes (Python functions), connect them with edges, compile the graph, and run it. State flows through each node in sequence.

The agent has six nodes: recall_memory → investigate → diagnose → report → save_to_memory, then a conditional branch to execute_action. Each one does a single thing. Adding a step means adding a node and an edge.

Despite the “agentic” framing, the graph runs in the exact order you defined. The intelligence lives inside the individual nodes. The structure is just a state machine with a diagram attached.

It makes the workflow predictable, easy to trace, and simple to modify.

ReAct: Reasoning Through Tool Calls⌗

ReAct (Reasoning + Acting) is a pattern where the LLM alternates between deciding what to do and calling a tool, iterating until it reaches a conclusion. Each tool call returns a result, which the model reasons over before deciding whether to call another.

The investigate node runs this loop. It has four tools wired up as named functions: list_task_instances, get_task_log, get_dag_structure, get_import_errors. The LLM calls whichever ones it needs, reads the result, and decides what to check next.

In practice: a failure comes in, the agent lists the task instances, finds the failing one, pulls the last 40 lines of its log, determines whether it’s a runtime error or a parse failure, and produces a conclusion. Most investigations follow a sensible path without any explicit instruction on tool order.

The loop is capped at 20 iterations. Without a cap, a confused model will happily call get_task_log forever. The limit forces it to conclude with whatever it has gathered.

One thing the framework doesn’t handle: Groq occasionally returns malformed tool-call JSON. The agent catches it and re-prompts the model to retry the turn. Defensive handling like this has to be written explicitly.

RAG: Memory from Past Investigations⌗

RAG (Retrieval-Augmented Generation) is how you give the model context it wasn’t trained on. Instead of relying solely on training data, you retrieve relevant documents and feed them in each time it runs.

The agent uses it as a memory system. Before each investigation, it queries a ChromaDB vector store for similar past cases, using dag_id + task_id + error_summary as the search key. The three closest matches get injected into the investigation prompt as examples.

If the agent already diagnosed the same connection timeout last week, it should recognize the pattern faster this time. In practice, the store starts empty, so the first weeks of operation provide no recall benefit at all. The value builds as history accumulates. I tested it with repeated identical failures and the pattern matching worked clearly. With realistic variance across many different DAGs, you’d need real production history before it makes a difference.

Human-in-the-Loop: The Interrupt Pattern⌗

Human-in-the-loop means the agent has to ask a human before taking certain actions. The question is how to implement the pause in a way that’s resumable and persistent, not just a sleep.

LangGraph handles this with interrupt_before=["execute_action"]: the graph halts at that node. The Streamlit dashboard shows the pending decision. The poller picks up the human’s choice and either resumes the graph or closes the run.

A correct diagnosis is still one step away from an unsafe action. The approval gate is what separates a useful tool from one you can’t trust in production.

It also required thinking carefully about state persistence. LangGraph checkpoints graph state to SQLite after each node. If the process crashes between the interrupt and the approval, the graph resumes from where it paused on the next restart. Building that by hand would have been genuinely annoying.

Running It for Free⌗

The whole project ran on Groq’s free tier using Llama 3.3-70B. Groq’s free tier covered the whole development phase. When I hit them, the agent fell back to a local Ollama instance automatically via LiteLLM’s fallback chain:

LLM_MODEL=groq/llama-3.3-70b-versatile
LLM_FALLBACK_MODELS=groq/llama-3.1-8b-instant,ollama/qwen2.5:7b

Setting up Ollama is three commands:

brew install ollama
ollama pull qwen2.5:7b   # ~4.7 GB, good tool-calling support
ollama serve

On Apple Silicon, Qwen 2.5 7B is fast enough for investigation tasks. Not as capable on complex logs, but it handles common failure patterns fine. A local fallback means a long test session never blocks on rate limits.

Total cost: zero. If I’d used paid models, an investigation runs roughly 15k-25k input tokens and 1k-2k output tokens:

100 investigations a month on Claude Haiku is about $2.50. On Gemini Flash it’s under $0.25. The numbers are low unless you’re running thousands of investigations daily.

What I Learned⌗

Structured output over open-ended summarization. Whether the agent is useful depends entirely on whether its output is actionable. Open-ended summarization (“the task failed, check the logs”) is useless. Prompting for specific labeled fields (DIAGNOSIS, FAILING_TASK, RECOMMENDED_ACTION, ERROR_SUMMARY) forces specificity and makes the downstream code reliable. Asking the LLM for raw JSON works until it doesn’t; regex extraction on labeled plain-text fields is more robust.

The approval gate. A correct diagnosis is still one step away from an unsafe action without it. It’s what makes the difference between a useful tool and a dangerous one.

Observability tooling is overstated at small scale. LangSmith and Langfuse are marketed as essential infrastructure for agentic systems. For this project, SQLite and log statements were enough. The tracing tools earn their place at production scale with multiple agents and high request volume. At side-project scale, they’re overhead you don’t need yet.

What’s Still Open⌗

Adaptive recall. The memory store is only useful when past cases are similar to the current one. For novel or infrequent failures, recall returns unhelpful matches and the investigation runs without useful context. The right solution is a query classifier that decides whether retrieval would help before bothering to call it. I haven’t built that layer yet.

Outcome feedback. The feedback loop is incomplete. The agent stores its diagnosis, but has no way to know if the retry actually fixed the root cause or just cleared a transient error. What’s missing is a post-resolution confirmation: was the diagnosis correct? That signal would make the RAG store improve over time. The interaction design isn’t obvious: a yes/no confirmation is low friction but coarse; an editable diagnosis field is more useful but most people won’t fill it in.

Repository access. Task logs show what failed, not why the code is structured that way. Giving the agent read access to the source code would let it reason about the actual logic, not just the exception message. The tradeoff is cost: reading source files multiplies token usage per investigation. Worth it for complex failures, overkill for a missing environment variable.

MCP. The Airflow tools are bespoke LangChain implementations. Model Context Protocol is emerging as a standard connector between agents and external tools, which would make them reusable across different agents without rewriting the integration layer. That’s worth doing if this ever runs somewhere other than locally.

The goal was to build something with the current tooling and see where things stand. The framework held up. The harder problems are the ones the framework can’t solve: whether the diagnosis was actually right, whether the memory is improving, whether the model you’re running can handle edge cases in your logs. Those are worth coming back to.