Agentic AI runs on tools

What is agentic AI? It’s hot, that’s what it is. We might be on an exponential growth curve of people attaching the “agentic” label to projects, systems, and products. Nevertheless, I do believe that the concept is useful and important once we understand the substance behind it, and furthermore that practical applications will indeed grow quickly.

A quintessential agentic system has:

1. Goals

2. Planning

3. Multi-step reasoning

4. Memory and/or persistence

5. Interaction with the outside world

Given a goal, an agentic system constructs a plan for how to approach it, has some ability to work through (and update) its plan in a loop, can effectively employ its own output along the way, and has some interaction with other systems in ways that aren’t fully mediated by a human user. Agentic systems vary on how deep they go on each dimension.

To illustrate this more concretely, contrast a powerful coding agent against a human coder who has a chatbot open either inside or outside of their IDE. Once assigning a task to the coding agent, it may: create a step-by-step plan, generate new code, execute and test the code, ingest error messages, and repeat this process until succeeding. Meanwhile, a human coder with a chatbot may repeatedly insert code from the chatbot into their editor, but the human decides when to edit or run the code and which step to take afterwards. The human is far more involved compared to using the coding agent.

“Agentic” is an adjective, and need not be a binary one. We can describe the agenticness of a system. The coding agent is fairly agentic, while a chatbot that only has text input and output is not very agentic.1

We can test this framework by considering how agentic a reasoning LLM is. With a suitable prompt that explains our task, it could construct a plan, loop through multiple iterations of generating a suitable answer, and effectively use its own prior output as the context for subsequent iterations. Those are all dimensions of agenticness. But unless we trust it with some form of automated interaction with the outside world, it’s just a fancy chatbot. It’s more agentic than a weak and non-reasoning LLM, but its effect on the world is mediated through the human using it.

I define “agenticness” as how much dynamic capability a tool has between steps of human supervision. A chatbot might have very little dynamic capability (it might only output text) and the human-in-the-loop cycles are tight (the human reviews the text after each answer). A coding agent has substantial dynamic capability (it can make large code changes across multiple attempts, interacting with the editor and/or the command line) and the human-in-the-loop cycles are wide (the human reviews the code after substantial changes).

LLMs have substantial capabilities even without external tools. They have an incredible amount of compressed information combined with many specific and general abilities. But those LLMs that can’t use any additional external capabilities will forever have limitations.

The most obvious limitation is the information cut-off. The base model of an LLM is trained on some corpus and cannot know all details of the future beyond that corpus.2 If I ask an LLM who won today’s baseball games, that LLM can only know that information by accessing some up-to-date resource. Similarly, toolless LLMs are at a severe disadvantage for predicting upcoming weather.3

Information cut-offs are one reason we need external capabilities, and not the only one. Consider performing basic math. LLMs learn some math through memorization and can solve select other computations through reasoning loops around learned mathematical principles, but that’s a lot of capability to learn compared to just having access to a precise calculator.

Web search, weather services, and calculators are several types of tools for accurately answering questions. Those are, conceptually if not literally, read-only tools with no side effects. We might also want LLMs to be able to perform the equivalent of write actions, intentionally affecting the world.4 You could have LLMs filter your emails, or even respond to them. Or reorder the items on your todo list and place reminders on your calendar. There’s no limit to the uses people could want. Those who want to entrust agents with wide human-in-the-loop cycles will need to grant some tool control to those agents.

Most tools conceptually fit a pattern where they need some input with which they perform a well-understood behavior to generate some output. We already have a word for such things in mathematics and programming: functions.

We can create functions, give the definitions to LLMs, and tell the LLMs that they are allowed to call the functions. The LLM can choose to output text (or images or other modalities it supports) or it can choose to call a function.5

When the LLM requests a function call we execute the function on our end and give the answer to the LLM, which carries on with text or function calls.

What’s the difference between tool use and function calling? In practice, not much, anymore. Tool use is a capability we want and function calling is part of the implementation. We can represent nearly all external capabilities as function definitions, and indeed we have a lot of experience doing so and thinking in such ways through decades of programming and API development.

We can use functions for built-in capabilities of an LLM and for user-specified tools.

An alternative, particularly common in early tool use implementations, is to use a non-function textual convention for a small set of hardcoded tools built in to the LLM training. WebGPT, for example, fine-tuned a model to learn commands like “Clicked on link <link ID>” and “Find in page: <text>” [2].

Capabilities might not necessarily be defined as functions. Some LLMs use direct text editing, such as inline code generation in coding agents. Others are highly trained on specific capabilities, like browser control as used by OpenAI’s Operator.

There’s a place for specific trained capabilities, but they won’t be the full story for complex agentic systems. If we want LLMs to generalize to arbitrary user-defined tools then we need a way for those to be specified. Function definitions are natural because that is already a known paradigm for specifying inputs and outputs.

Instead of LLMs needing to learn (during training) how to use each tool individually, they only need to learn the general pattern of function calling.

I don’t take it for granted that LLMs, trained mostly on internet text, should understand:

• The definitions of functions, including parameters

• What inputs to provide to functions

• When it is useful to call a function and when it’s better to not call functions

• How to not get confused between function definitions and the non-function text

But they can be! For this they had a head start, since their corpuses naturally contain a significant amount of code and a significant amount of human text that’s about code. This can be further supplemented by adding in custom corpuses for function calling.

The function calling pattern works well enough. Leading LLMs are specifying functions correctly, including the right inputs for the task, upwards of 80% of the time on out-of-sample testing (see the Berkeley Function-Calling Leaderboard) [3].6

However, 80% accuracy is impressive on a task that you don’t expect an LLM to be able to do at all, but if you’re going to do it repeatedly then you’ll quickly trigger mistakes.7

Repeatedly is very relevant for agents. That’s the heart of agentic AI: LLMs running in a loop with tools available. For agents that are expected to perform a multitude of actions during wide human-in-the-loop cycles, you better have a good quality control loop around the LLM.

MCP and A2A aren’t substitutes for function calling, but rather they are meta-standards on top of function calling. They are protocols for publishing API specs, discovering those APIs, and client-server communications for those APIs. These enable AI agents, which still need function calling.

A typical pattern is:

1. Use MCP or A2A to discover or access tools (hosted internally or as external services)

2. Specify those tools to your LLM just like any other functions

3. Call them as necessary (through MCP or A2A as opposed to local functions) when the LLM tells you to

See this example for A2A or this example for MCP, both of which follow the aforementioned pattern.

I don’t know which framework (from these or others) will ultimately win out, or if multiple will coexist. It’s still early. Yet, notably, these frameworks entrench the function calling paradigm rather than competing with it.

You might not like how topical agentic AI is, but it isn’t going away. We should see more AI-based systems that support goals, planning, loops, memory, and interaction with the outside world. Function calling is an incredibly powerful—and fairly recent—paradigm for AIs to call external tools and ultimately interact with the outside world.

If you don’t notice a plethora of useful agentic systems so far, note that these are early days. Handing control over to an LLM requires a greater extension of trust in AI tools, and that trust can become increasingly warranted as LLMs’ skills at function calling improve, which they have been, and as we collectively learn development patterns that best address the inexactness of LLM-based systems. We should expect agentic AI to become more viable for more use cases.

[1] Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J., & Wei, F. (2023). Augmenting language models with long-term memory. Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23).

[2] Nakano, R., Hilton, J., Balaji, S., Wu, J., Long, O., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., & Schulman, J. (2021). WebGPT: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332.

[3] Yan, F., Mao, H., Cheng-Jie Ji, C., Zhang, T., Patil, S., Stoica, I., & Gonzalez, J. (2024). Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html,