Tool calling

Tool calling, also known as function calling, is a structured way to give LLMs the ability to make requests back to the application that called it. You define the tools you want to make available to the model, and the model will make tool requests to your app as necessary to fulfill the prompts you give it.

The use cases of tool calling generally fall into a few themes:

Giving an LLM access to information it wasn’t trained with

• Frequently changing information, such as a stock price or the current weather.
• Information specific to your app domain, such as product information or user profiles.

Note the overlap with retrieval augmented generation (RAG), which is also a way to let an LLM integrate factual information into its generations. RAG is a heavier solution that is most suited when you have a large amount of information or the information that’s most relevant to a prompt is ambiguous. On the other hand, if retrieving the information the LLM needs is a simple function call or database lookup, tool calling is more appropriate.

Introducing a degree of determinism into an LLM workflow

• Performing calculations that the LLM cannot reliably complete itself.
• Forcing an LLM to generate verbatim text under certain circumstances, such as when responding to a question about an app’s terms of service.

Performing an action when initiated by an LLM

• Turning on and off lights in an LLM-powered home assistant
• Reserving table reservations in an LLM-powered restaurant agent

Before you begin

If you want to run the code examples on this page, first complete the steps in the Getting started guide. All of the examples assume that you have already set up a project with Genkit dependencies installed.

This page discusses one of the advanced features of Genkit model abstraction, so before you dive too deeply, you should be familiar with the content on the Generating content with AI models page. You should also be familiar with Genkit’s system for defining input and output schemas, which is discussed on the Flows page.

Overview of tool calling

Genkit by Example: Tool Calling See how Genkit can enable rich UI for tool calling in a live demo.

At a high level, this is what a typical tool-calling interaction with an LLM looks like:

1. The calling application prompts the LLM with a request and also includes in the prompt a list of tools the LLM can use to generate a response.
2. The LLM either generates a complete response or generates a tool call request in a specific format.
3. If the caller receives a complete response, the request is fulfilled and the interaction ends; but if the caller receives a tool call, it performs whatever logic is appropriate and sends a new request to the LLM containing the original prompt or some variation of it as well as the result of the tool call.
4. The LLM handles the new prompt as in Step 2.

For this to work, several requirements must be met:

• The model must be trained to make tool requests when it’s needed to complete a prompt. Most of the larger models provided through web APIs, such as Gemini and Claude, can do this, but smaller and more specialized models often cannot. Genkit will throw an error if you try to provide tools to a model that doesn’t support it.
• The calling application must provide tool definitions to the model in the format it expects.
• The calling application must prompt the model to generate tool calling requests in the format the application expects.

Tool calling with Genkit

Genkit provides a single interface for tool calling with models that support it. Each model plugin ensures that the last two of the above criteria are met, and the Genkit instance’s generate() function automatically carries out the tool calling loop described earlier.

Model support

Tool calling support depends on the model, the model API, and the Genkit plugin. Consult the relevant documentation to determine if tool calling is likely to be supported. In addition:

• Genkit will throw an error if you try to provide tools to a model that doesn’t support it.
• If the plugin exports model references, the info.supports.tools property will indicate if it supports tool calling.

Defining tools

Use the Genkit instance’s tool() decorator to write tool definitions:

from genkit import Genkit
from genkit_google_genai import GoogleAI
from pydantic import BaseModel, Field

ai = Genkit(
    plugins=[GoogleAI()],
    model='googleai/gemini-flash-latest',
)

class GetWeatherInput(BaseModel):
    location: str = Field(description='The location to get the current weather for')

@ai.tool()
async def get_weather(input: GetWeatherInput) -> str:
    """Gets the current weather in a given location."""
    # Here, we would typically make an API call or database query. For this
    # example, we just return a fixed value.
    return f'The current weather in {input.location} is 63°F and sunny.'

The tool name defaults to the function name and the description defaults to the docstring (though both can optionally be overridden using name and description parameters in @ai.tool()); the input schema comes from the type hints. Choose a clear function name and write a descriptive docstring—they’re what the model uses to decide when to call the tool.

Using tools

Include defined tools in your prompts to generate content.

Using generate():

response = await ai.generate(
    prompt='What is the weather in Baltimore?',
    tools=[get_weather],
)

Using define_prompt():

weather_prompt = ai.define_prompt(
    name='weatherPrompt',
    tools=[get_weather],
    prompt='What is the weather in {{location}}?',
)

response = await weather_prompt({'location': 'Baltimore'})

Using Prompt files:

---
tools: [get_weather]
input:
  schema:
    location: string
---

What is the weather in {{location}}?

Then you can execute the prompt in your code as follows:

# assuming prompt file is named weatherPrompt.prompt
weather_prompt = ai.prompt('weatherPrompt')

response = await weather_prompt({'location': 'Baltimore'})

Streaming and tool calling

When combining tool calling with streaming, chunks are ModelResponseChunk values. Use chunk.text for text deltas, and inspect chunk.content for tool parts:

result = ai.generate_stream(
    prompt='What is the weather in Baltimore?',
    tools=[get_weather],
)

async for chunk in result.stream:
    if chunk.text:
        print(chunk.text, end='', flush=True)
    for part in chunk.content:
        if part.root.tool_request is not None:
            req = part.root.tool_request
            print(f'\n[tool request] {req.name}({req.input})')
        if part.root.tool_response is not None:
            res = part.root.tool_response
            print(f'\n[tool response] {res.name} -> {res.output}')

response = await result.response

Limiting tool call iterations with max_turns

When working with tools that might trigger multiple sequential calls, you can control resource usage and prevent runaway execution using the max_turns parameter. This sets a hard limit on how many back-and-forth interactions the model can have with your tools in a single generation cycle.

Why use max_turns?

• Cost Control: Prevents unexpected API usage charges from excessive tool calls
• Performance: Ensures responses complete within reasonable timeframes
• Safety: Guards against infinite loops in complex tool interactions
• Predictability: Makes your application behavior more deterministic

The default value is 5 turns, which works well for most scenarios. Each “turn” represents one complete cycle where the model can make tool calls and receive responses.

Example: Web Research Agent

Consider a research agent that might need to search multiple times to find comprehensive information:

from pydantic import BaseModel, Field

class WebSearchInput(BaseModel):
    query: str = Field(description='Search query')

@ai.tool()
async def web_search(input: WebSearchInput) -> str:
    """Search the web for current information."""
    # Simulate web search API call
    return f'Search results for "{input.query}": [relevant information here]'

response = await ai.generate(
    prompt=(
        'Research the latest developments in quantum computing, including recent breakthroughs, '
        'key companies, and future applications.'
    ),
    tools=[web_search],
    max_turns=8,  # Allow up to 8 research iterations
)

Example: Financial Calculator

from pydantic import BaseModel, Field

class CalculatorInput(BaseModel):
    expression: str = Field(description='Mathematical expression to evaluate')

@ai.tool()
async def calculator(input: CalculatorInput) -> float:
    """Perform mathematical calculations."""
    # Safe evaluation of mathematical expressions
    return eval(input.expression)  # In production, use a safe math parser

response = await ai.generate(
    prompt=(
        'Calculate the total value of my portfolio: 100 shares of AAPL, 50 shares of GOOGL, and '
        '200 shares of MSFT. Also calculate what percentage each holding represents.'
    ),
    tools=[calculator],
    max_turns=12,  # Multiple calculation steps needed
)

What happens when max_turns is reached?

When the limit is reached, Genkit stops the tool-calling loop and raises an error from the generate path (today: GenerationResponseError). That type is not a GenkitError subclass, so except GenkitError will not catch it—catch the exception around your generate / generate_stream call instead.

Pause the tool loop by using interrupts

By default, Genkit repeatedly calls the LLM until every tool call has been resolved. You can conditionally pause execution in situations where you want to, for example:

• Ask the user a question or display UI.
• Confirm a potentially risky action with the user.
• Request out-of-band approval for an action.

Interrupts are special tools that can halt the loop and return control to your code so that you can handle more advanced scenarios. Visit the interrupts guide to learn how to use them.

Explicitly handling tool calls

If you want full control over this tool-calling loop, for example to apply more complicated logic, set the return_tool_requests parameter to True. Now it’s your responsibility to ensure all of the tool requests are fulfilled:

from genkit import Message, Part, Role, ToolResponse, ToolResponsePart

response = await ai.generate(
    prompt="What's the weather like in Baltimore?",
    tools=[get_weather],
    return_tool_requests=True,
)

while response.tool_requests:
    tool_parts = []
    for req in response.tool_requests:
        if req.tool_request.name != 'get_weather':
            raise ValueError(f'Unexpected tool: {req.tool_request.name}')
        output = await get_weather(GetWeatherInput(**req.tool_request.input))
        tool_parts.append(
            Part(
                root=ToolResponsePart(
                    tool_response=ToolResponse(
                        name=req.tool_request.name,
                        ref=req.tool_request.ref,
                        output=output,
                    )
                )
            )
        )

    response = await ai.generate(
        messages=[
            *response.messages,
            Message(role=Role.TOOL, content=tool_parts),
        ],
        tools=[get_weather],
        return_tool_requests=True,
    )

Extending tool capabilities with MCP

The Model Context Protocol (MCP) provides a powerful way to extend your tool-calling capabilities by connecting to external MCP servers. With MCP, you can:

• Access pre-built tools from the MCP ecosystem without implementing them yourself
• Connect to external services like databases, APIs, and file systems
• Share tools between different AI applications
• Build extensible workflows that leverage community-maintained tools

MCP tools work seamlessly with Genkit’s tool-calling system, allowing you to mix custom tools with external MCP tools in the same generation request.

Next steps

• Learn about Model Context Protocol (MCP) to extend your tool capabilities with external servers
• Explore interrupts to pause tool execution for user interaction
• See retrieval-augmented generation (RAG) for handling large amounts of contextual information
• Check out multi-agent systems for coordinating multiple AI agents with tools
• Browse the tool calling example for a complete implementation