Code Actions as Tools
Evolving Tool Libraries for Agents
Martin Krasser
Latest posts
From single-user to multi-party conversational AI
This article was originally posted here . Single-user AI agents excel at responding to direct queries in one-on-one interactions. A user sends the agent a self-contained query with sufficient context, and the agent processes it directly. Even in group chats, the typical pattern remains the same: users mention the agent with a direct query. This interaction model treats multi-user environments as…
DeepSeek-R1 agents with code actions
I ran a quick experiment investigating how DeepSeek-R1 performs on agentic tasks, despite not supporting tool use natively, and I was quite impressed by preliminary results. This experiment runs DeepSeek-R1 in a single-agent setup, where the model not only plans the actions but also formulates the actions as executable Python code. On a subset 1 of the GAIA validation split, DeepSeek-R1…
Impact of prompt masking on LLM agent step planning performance
I recently experimented with fine-tuning an LLM agent's step planner on synthetic trajectories . During fine-tuning, the step planner module learns to create a task description for the next step and select an appropriate tool to execute that step. The selected tool is responsible for translating the informal task description into tool-specific executable actions and returning a (summarized)…
Planner fine-tuning on synthetic agent trajectories
Notebook Repository In a previous article I experimented with separating step planning from function calling in ReAct -style LLM agents. A central planner module is responsible for describing the task for the next step and selecting an appropriate tool. The selected tool is responsible for translating the informal task description into tool-specific executable actions. Reducing the planner's…
Separation of step planning concerns in ReAct-style LLM agents
Notebook Repository LLM agents require a wide range of capabilities for executing tasks autonomously. They must be able to decompose complex user instructions, plan next actions, interact with their environment using tools, call tools with correct arguments, reason about observations and adjust planning if needed. Instructing an LLM to behave like an agent is often done with comprehensive,…
Schema-guided generation with open LLMs
Update 2024-05-31 : This article has been updated to use Llama-3-8B-Instruct, instead of Llama-2-70B-Chat, and the latest version of bot-with-plan . Notebook ( original version , latest version ) Repository ( original version , latest version ) OpenAI recently introduced JSON mode for its chat models. Anyscale provides a similar service that additionally supports user-defined JSON schemas. Both do…
Schema-guided generation in LangChain agents
Update 2024-05-31 : This article covers early work in the bot-with-plan project and is now superseded by the following articles: Schema-guided generation with open LLMs Separation of planning concerns in LLM agents Planner fine-tuning on synthetic agent trajectories Notebook Repository LLM agents use large language models (LLMs) to decide which tools to use for interacting with their environment…
Extending Visual ChatGPT with image search engines
The source code for this article, an extended version of Visual ChatGPT, is available here . Update: A 🤗 Transformers Agent based implementation of the examples in this article is available here . A recent trend in software development uses large language models (LLMs) to understand user input in natural language and to execute complex user instructions by autonomously decomposing them into…
Training compute-optimal Perceiver AR language models
Notebook and code for this article are available here . Introduction In Training Compute-Optimal Large Language Models [1] (the Chinchilla paper) the authors describe how to determine the optimal model size $N_{opt}$ and number of training tokens $D_{opt}$ for a given compute budget $C$, and how $N_{opt}$ and $D_{opt}$ scale with $C$. These scaling laws are applicable to decoder-only transformer…
A gentle introduction to Rotary Position Embedding
The Transformer model is invariant to reordering of the input sequence. For sequence modeling, position information must therefore be explicitly included. Rotary position embedding is an approach for including relative position information. It is a multiplicative approach, in contrast to most other approaches which are additive. Position information basics This article assumes that you have a…
Using AWS SageMaker with minimal dependencies, part 2
Fault-tolerant model training on spot instances
Using AWS SageMaker with minimal dependencies, part 1
Distributed model training with PyTorch Lightning
Sparse Gaussian processes
Sources: Notebook Repository Series: Gaussian processes Gaussian processes for classification Sparse Gaussian processes (this article) Introduction Exact Gaussian processes cannot be applied to larger training datasets because their time complexity scales with $O(n^3)$ where $n$ is the size of the training set. Approximate or sparse Gaussian processes are based on a small set of $m$ inducing…
Gaussian processes for classification
Sources: Notebook Repository Series: Gaussian processes Gaussian processes for classification (this article) Sparse Gaussian processes This article gives an introduction to Gaussian processes for classification and provides a minimal implementation with NumPy. Gaussian processes for regression are covered in a previous article and a brief recap is given in the next section. Regression recap A…
Reliable uncertainty estimates for neural network predictions
Sources: Notebook Repository I previously wrote about Bayesian neural networks and explained how uncertainty estimates can be obtained for network predictions. Uncertainty in predictions that comes from uncertainty in network weights is called epistemic uncertainty or model uncertainty. A simple regression example demonstrated how epistemic uncertainty increases in regions outside the training…
Latent variable models, part 2
Stochastic variational inference and variational autoencoders
Latent variable models, part 1
Gaussian mixture models and the EM algorithm
Single image super-resolution with deep neural networks
Sources: Notebook Repository This article is an introduction to single image super-resolution. It covers some important developments in recent years and shows their implementation in Tensorflow 2.0. The primary focus is on specialized residual network architectures and generative adversarial networks (GANs) for fine-tuning super-resolution models. Super-resolution is the process of recovering a…
Variational inference in Bayesian neural networks
Sources: Notebook Repository This article demonstrates how to implement and train a Bayesian neural network with Keras following the approach described in Weight Uncertainty in Neural Networks ( Bayes by Backprop ). The implementation is kept simple for illustration purposes and uses Keras 2.2.4 and Tensorflow 1.12.0. For more advanced implementations of Bayesian methods for neural networks…