SafeSearch: Teaching LLM Search Agents to Be Both Smart and Safe

LLMs are rapidly expanding their built-in knowledge from training. However, they still suffer from hallucinations and lack access to private or time-sensitive information, such as personal medical data or real-time breaking news. To overcome these limitations, they need the ability to retrieve external knowledge. Recent advances in search agents (e.g., Search-o1, Search-R1, R1-Searcher, DeepResearcher) have made great progress in this direction, enabling LLMs to autonomously generate queries, retrieve relevant information, and reason over it across multiple turns to answer open-domain questions.

As illustrated in the example above, the LLM alone cannot answer the question because it depends on up-to-date information. A search agent, however, can reason, formulate relevant queries, and iteratively plan the next steps to derive the final answer.

Although this seems promising, our recent paper shows that enabling search also makes LLMs more susceptible to producing harmful outputs. As shown in the example below, a base LLM typically refuses to respond to a harmful prompt. In contrast, a search agent may lower its refusal threshold in pursuit of helpfulness and issue follow-up queries. Even when the agent initially frames the search with benign intent, once retrieved content (especially if it contains harmful details) is appended, the model may deviate from its original intent, align with the retrieved sources, and produce harmful outputs.

To mitigate this safety issue and build a helpful, safe search agent, we built SafeSearch. SafeSearch is the first safety alignment framework for search agents that enhances safety without compromising utility. By conducting experiments across multiple datasets and backbone LLMs, we demonstrate that SafeSearch reduces the harmful rate by up to 70% on red-teaming datasets while maintaining QA performance comparable to utility-only fine tuning.

To systematically evaluate both utility and safety, we test different systems on three red-teaming datasets containing harmful inputs (Redteaming-Resistance-Benchmark, StrongReject, and WildTeaming) and three QA datasets containing open-domain QA pairs (TriviaQA, HotpotQA, and Bamboogle). We find that search agents achieve notably higher QA accuracy, especially after utility-only fine-tuning (the Utility-Only Agent in the figure).

However, when evaluated on red-teaming datasets, search agents are up to 3× more likely to generate harmful outputs than their base LLMs. Moreover, utility-only fine-tuning further increases this harmfulness rate, underscoring the need to jointly optimize safety and utility rather than improving utility in isolation.

To make the search agents useful but also safe, we developed SafeSearch, the first reinforcement learning (RL) framework that jointly optimizes safety and utility for LLM-based search agents. Specifically, SafeSearch trains agents to:

• Generate safe but helpful responses by avoiding blanket refusals to harmful inputs and instead offering informative responses such as high-level legal context and safer alternatives, consistent with GPT-5’s safety alignment goals.

• Maintain strong accuracy on general QA tasks.

For QA performance, SafeSearch uses a final-output reward that evaluates the correctness and format of the model’s answer. For safety performance, it combines two reward signals:

1. Final-output rewards — encourage safe and helpful responses.

2. Query-level rewards — penalize unsafe search queries and reward safe ones, motivated by our observation that unsafe queries strongly correlate with unsafe final outputs. Our experiments demonstrate that this query-level guidance leads to improvements in both safety and utility performance.

Our experiments across different backbone LLMs (Qwen-2.5-3B-Instruct and Qwen-2.5-7B-Instruct) show that finetuning with SafeSearch led to:

• 50–90% fewer harmful outputs

• Comparable QA accuracy to utility-only finetuned agents

• High helpfulness among safe responses — rather than relying on overly conservative refusals that are safe but unhelpful

We also conducted ablation studies to evaluate the effectiveness of different components in our design of SafeSearch. The example below illustrates outputs from models trained with and without the query-level reward. Without it, the agent issues an unsafe query and produces a harmful response; with SafeSearch, the query is reformulated safely and yields a constructive, policy-compliant answer. For more details and analysis, please refer to the paper.

SafeSearch shows that we don’t have to trade safety for usefulness. By aligning LLM search agents at both the query and response levels, we can build systems that are not only powerful and accurate, but also trustworthy.

More details are available in the paper, along with the public code release.