Your LLM can’t math (and that’s ok)

Last week we precisely defined an agent: an LLM, in a harness, that calls tools repeatedly in a loop. Most agents have two types of tools: read tools that can observe the world, and write tools that can change the world.

But my definition of tools contained a deliberate simplification that I want to dig into this week. Previously, I said tools run on your computer, but that’s not generally true: tools are run by the harness. In many cases (like with ellmer or a coding agent), the harness runs locally, but when using web chat (like claude.ai or chatgpt.com), the harness runs on the server.

Let’s explore what that means, starting with a simple example using a bare ellmer chat with no tools. What happens if we ask it to multiply two large numbers, a task that we know LLMs are not good at?

library(ellmer)
x <- 20598162
y <- 83106206
chat <- chat_openai()
chat$chat(interpolate("What's {{x}} * {{y}}?"))
#> The product of **20,598,162** and **83,106,206** is:
#>
#> **1,711,067,239,026,172**

We can check the answer with R, which does know how to multiply correctly. This reveals that the model was confidently wrong:

format(x * y, big.mark = ",", scientific = FALSE)
#> [1] "1,711,835,094,393,372"

However, if you ask this same question on chatgpt.com or claude.ai, they both get it right. Why? Because the web chats are also harnesses that come with a variety of useful tools. One of those tools is a calculator, because both OpenAI and Anthropic know that LLMs are bad at math.

We can give our ellmer chat a similar tool. First we’ll implement a basic calculator function that takes a mathematical expression as a string and returns the result:

calculator <- function(code) {
  math_env <- new.env(parent = emptyenv())
  math_env$`+` <- `+`
  math_env$`-` <- `-`
  math_env$`*` <- `*`
  math_env$`/` <- `/`
  math_env$`^` <- `^`
  math_env$`(` <- `(`
  math_env$sqrt <- sqrt
  math_env$exp <- exp
  math_env$log <- log
  eval(parse(text = code), envir = math_env)
}
calculator("15 * 11")
#> [1] 165
calculator("read.csv('foo.bar')")
#> Error in `read.csv()`:
#> ! could not find function "read.csv"

This code uses some environment and evaluation magic (which you can learn more about in Advanced R) to restrict R to a set of safe operations. In the future, I’ll talk about why you might want to free this up to run any R code and how you can do that (mostly) safely.

Next we register the function as a tool. Note that we give some advice to the LLM on when to use it. Like with all LLM prompting, there’s no guarantee that the LLM will listen, but it’s likely to help (in the sense that the average quality of responses with the tool should be better than without it).

chat <- chat_openai()
#> Using model = "gpt-4.1".
chat$register_tool(tool(
  calculator,
  description = "Evaluate a mathematical expression.
    Use this for any arithmetic, since you can't reliably compute it yourself.
    Supports +, -, *, /, ^, parentheses, sqrt(), exp(), and log(), following
    standard precedence rules.",
  arguments = list(
    code = type_string(
      "Mathematical expression to evaluate, e.g. '1 + 2 * 3'."
    )
  )
))

Now if we re-ask the same question we get the right result!

chat$chat(interpolate("What's {{x}} * {{y}}?"))
#> ◯ [tool call] calculator(code = "20598162 * 83106206")
#> ● #> 1711835094393372
#> 20598162 multiplied by 83106206 equals 1,711,835,094,393,372.

The harnesses used by current chatbots provide the LLM with a whole raft of tools. Here are a few that you might have noticed:

• Search the web: when the model needs information outside of its training data (like something recent or hyperspecific), it can search the web. Here the model writes a search query, the tool runs it and returns the results (titles, snippets, and links) back to the conversation.

• Fetch a page: when the model needs to read a webpage, the tool downloads it and returns its contents as text. This often goes hand-in-hand with web search, but is also what lets you paste in a URL and ask the model to summarise it. (I often use this to extract recipes into a format I like: strip the blather at the start, repeat the ingredients needed at each step, and convert to metric measurements.)

• Make a memory: when you tell the LLM to remember something, it edits some central markdown file that’s included in the prompt for all future conversations. This is how an assistant can “remember” your name, or that you use British English, or that you prefer sentence case for headings, without you repeating yourself every time.

• Draw a picture: when you ask the LLM to generate an image, it doesn’t draw anything itself but instead acts as a skilled prompt-writer for a separate, specialised model. The model writes a detailed text prompt, uses a tool call to an image model to generate the image, then adds that to the conversation. This is how ChatGPT and Gemini generate images.

Why does it matter that these are all tool calls and not some intrinsic property of the model? Because it shows that even if the underlying model is static, the abilities of the system can grow over time. And importantly, because you can make a harness with your own tools, you can extend the model in whatever way you want. When you combine that insight with tools like shinychat, you have the potential to create tailored systems that can do better than the best generic chatbots for your specific domain.

There’s one last wrinkle to cover. I said tool calls were the responsibility of the harness, but that’s another simplification: there’s also a selection of built-in tool calls that the models can do themselves. (Or maybe the right way to describe this is that models also come with a limited server-side harness.)

For example, the big three (OpenAI, Anthropic, and Google) all provide a built-in web search tool call:

chat <- chat_openai("Be terse", model = "gpt-5.4-mini", echo = FALSE)
chat$register_tool(openai_tool_web_search())
. <- chat$chat("When is Hadley Wickham's birthday?")
chat
#> <Chat OpenAI/gpt-5.4-mini turns=3 input=8501 output=72 cost=$0.01>
#> ── system ──────────────────────────────────────────────────────────────────────
#> Be terse
#> ── user ────────────────────────────────────────────────────────────────────────
#> When is Hadley Wickham's birthday?
#> ── assistant [input=8501 output=72 cost=$0.01] ─────────────────────────────────
#> [web search request]: "Hadley Wickham birthday birth date"
#> Hadley Wickham’s birthday is **October 14, 1979**. ([en.wikipedia.org](https://en.wikipedia.org/wiki/Hadley_Wickham?utm_source=openai))

This is nice because instead of user -> LLM -> harness -> LLM we can eliminate one HTTP request and response, reducing it to user -> LLM (with tool call), saving some time. You can see that in the result above: the final response acknowledges that a web search was performed before including the text response. (And as an added benefit, you now know when to send me a birthday card 🎂.)