TL;DR — Teach What You Can, Search the Rest

Modern image generators render gorgeously and lie fluently. Ask for the 2025 Osaka Expo mascot and you get a confident, wrong invention. The failure isn't the pixels—it's the knowledge. On AgentGen-Bench, open generators score just 17.5–31.6 out of 100 on search-intensive prompts—with a collapse of up to 41.4 points that standard benchmarks never register.

Search is the obvious fix, the way an illustrator consults references. But naive search backfires: it corrupts prompts the generator already handled. The real problem is a knowledge boundary—the line between what a generator can learn and what it must look up. That line is generator-specific, it moves during training, and it cannot be hand-drawn. It has to be discovered.

We discover it by co-training the generator and the search agent together. Below, we show how the collapse works, why naive search fails, what the knowledge boundary is, and how a co-trained 8B reasoner on a 4B generator matches a frontier reasoner on the same generator.

They Render Beautifully. They Just Make Things Up.

Ask a frontier image model for the mascot of the 2025 Osaka Expo. You get a polished, confident fabrication. Ask for a historically accurate Spartan phalanx and you get anachronistic armor, rendered in exquisite detail.

The lighting is right. The composition is right. The world is wrong.

This is not a rendering failure. It is a world-knowledge bottleneck. Generators train on fixed corpora with hard knowledge cutoffs; user requests draw on new characters, regional symbols, niche typography, historical artifacts, and events that postdate training.

Worse, generators have no way to flag their own ignorance. They are trained to always output an image—never to say, “I don't know what this looks like.” So they guess, beautifully, every time.

Ask for a specific person, a labeled scientific diagram, or live data and today's best generators confidently fabricate. These representative cases span SearchGen's 12 failure categories.

Finding 1 — A 41.4-Point Gap That No Benchmark Shows

Generators that score comparably on standard prompts diverge by up to 41.4 points when search-intensive world knowledge is required.

On NoSearch prompts, the strongest open generator reaches 70.0, overlapping the commercial range (66.3–78.9). Turn to prompts that need live world knowledge and the field splits: open generators span 17.5–31.6, while the five leading commercial systems score 63.6–75.5 and drop by only 2.0–9.7 points. Existing benchmarks test rendering inside known concepts, so they never see this gap at all.

To surface it, we built AgentGen-Bench (previously named SearchGen-Bench): 751 test prompts scored with separate dimensions for knowledge and rendering. The new name reflects the benchmark's broader use for evaluating agentic generation. The split is the whole point. When a generator scores poorly on knowledge checklists but remains strong on image quality, the diagnosis is unambiguous—it can draw; it just does not know.

AgentGen-Bench results. Generators score well on prompts they can answer from memory (gray). On prompts requiring external knowledge (blue), every open generator drops—by as much as 41.4 points—while the leading commercial systems remain comparatively stable. The bottleneck is missing knowledge, not rendering skill.

Overall AgentGen-Bench ranking. GPT-Image-2 leads, followed by Grok-Imagine-2.0-Low and Qwen-Image-3-Max. Scores use Overall9 from the finalized 3d5 evaluator.

Benchmark Leaderboards

Full evaluates all 751 benchmark prompts. Test-mini is the fixed, published 200-prompt slice obtained by selecting rows whose subset array contains the official testmini label—it is not a new random sample:

from datasets import load_dataset
benchmark = load_dataset("JasperHaozhe/AgentGen-Bench", split="test")
testmini = benchmark.filter(lambda row: "testmini" in row["subset"])
assert len(testmini) == 200

Scores are on a 0–100 scale. Each table reports Overall9 for the same 18 generators. Coverage is scored prompts / slice prompts, with missing evaluator scores excluded from aggregation.

Full benchmark (751 prompts)

Test-mini (200 prompts)

testmini_easy and testmini_hard are independent official label slices; they are not necessarily a partition of the 200-prompt testmini slice.

Mini-easy

Mini-hard

Full Benchmark Breakdown

Scores are reported on a 0–100 scale; higher is better. Checklist, Rubric, and Visual ref. are knowledge-sensitive measures that test whether requested facts are present. Image quality, Text rendering, and Physical plausibility capture rendering competence. Together, the components distinguish a model that cannot draw from one that can draw but lacks the required world knowledge.

NoSearch contains 100 prompts; Search-Intensive is the 651-row union of VisualSearch and TextualSearch. Scores use present-only aggregation.

Finding 2 — Search Should Help. Often It Hurts.

An illustrator handed an unfamiliar brief looks up references before drawing. Give the generator the same move—a reasoner spots knowledge gaps, search fills them, and the results feed generation—and you have agentic visual generation. Natural. And, done naively, harmful.

Naive search actively degrades prompts the generator already handles.

Search everything blindly, and every generator gets worse on prompts that never needed help. Qwen-Image-2 drops from 70.7 to 60.4 on the no-search stratum—a 14.6% relative loss on prompts it already aced.

Two distinct failures explain it. Concept corruption: search fires on something the model already knew, and the retrieved reference overrides correct internal knowledge—a gating failure. Copy effect: a reference contains so much detail that the generator copies it wholesale instead of borrowing the one missing fact—a filtering failure.

Search is not free. Fed raw, retrieved content leaks into the image—the model copies a reference boat verbatim or pastes a search-result caption into the artwork. Naive retrieval corrupts the very prompts a model could have answered alone.

The Knowledge Boundary

Some knowledge belongs inside the model. A character's canonical look or a flag's fixed geometry is stable, low-dimensional, and learnable once and for all. Fire search for it and you only add noise.

Other knowledge belongs outside. It changes faster than retraining cycles, sits too deep in the long tail to learn reliably, or needs per-request reasoning. For this, search is structurally necessary. The tail is enormous: 93.1% of the 31,537 entities in our data appear in just one prompt. No feasible training set covers that.

Some knowledge is internalizable and search should not fire for it; other knowledge is contextual and search is structurally necessary.

We call the divide between those sets the knowledge boundary. It is generator-specific, and it moves. As a generator learns, concepts migrate from “must search” to “already knows.” A search policy tuned for a weak generator is wrong for a strong one.

The boundary cannot be hand-specified. It is discoverable: it falls out of training the generator and searcher together.

Method — Gate, Filter, Integrate

The method has two halves: a searcher that does not poison the generator, and a training loop that finds and expands the boundary.

• Gate: decide whether to search. Only critical or important gaps trigger a query; the rest are dropped. Use at most three queries per prompt, or skip search entirely.
• Filter: decide what to keep. Choose the reference that fills the specific gap with the least extraneous baggage.
• Integrate: decide how it enters. Route visual references through language, not raw pixels, so the generator borrows exactly what is named and nothing else leaks in.

The teach-then-search co-training recipe acts on both sides of the boundary:

1. Warm-start: supervised fine-tuning gives an 8B reasoner the gate–filter–integrate protocol.
2. Teach the generator: online iterative Diffusion-DPO helps it internalize stable knowledge and use imperfect references robustly.
3. Recalibrate the searcher: rejection-sampling fine-tuning rewards trajectories where search helps the improved generator and discards the rest.

Phase 1 moves the boundary outward; Phase 2 moves the search policy inward to match. The model is never told where the line is—it learns from which searches actually helped.

Two coupled loops. The agent gates each prompt, fetches and filters references only when a knowledge gap warrants it, then integrates them into an enriched prompt. Online DPO teaches the generator what it can internalize, expanding the knowledge boundary so the agent has less to fetch over time.

Finding 3 — The Boundary Is Discoverable

Co-training discovers the knowledge boundary: a calibrated 8B reasoner matches a frontier oracle on the same generator.

On Klein-4B, the full co-training progression climbs monotonically to 31.8 overall, matching the frontier Gemini oracle on the same generator (31.2). Both phases pull their weight: teaching the generator adds +2.8, and recalibrating the searcher adds +2.6.

It is also selective, which is the hard part. On prompts that do not need search, the calibrated reasoner lifts the no-search baseline from 49.9 to 56.9—it learned when to stay quiet, exactly where naive search did damage. It is also generator-specific: a policy tuned for the strengthened generator scores worse on the base one, confirming that the boundary belongs to the generator–reasoner pair, not to the prompt alone.

The headline is calibration per dollar. An 8B reasoner on a 4B generator reaches the same quality as a frontier commercial reasoner on that generator, and the full recalibration cycle fits in 4 × 8 GPU-hours.

(a) Co-training compounds. Each round—reasoner SFT, generator DPO, reasoner RFT—lifts quality across all three difficulty tiers for Klein-4B and Bagel-7B. (b) Proof the boundary moved: after co-training, per-prompt no-search quality shifts right; more prompts clear a given quality bar without search.

This compares reasoners on a fixed 4B generator; it is not a claim of frontier absolute image quality. In absolute terms, 31.8 remains below GPT-Image-2's finalized full-benchmark Overall9 score of 76.0.

The Released Harness

Reproducing search-augmented generation usually means paying for a search engine and a fleet of generators—and watching results drift as those services change. We froze the whole thing.

We release SearchGen-20K (20,839 world-knowledge-grounded prompts across 12 failure categories and 22 domains, with 5.2 knowledge gaps per prompt on average), the co-training corpus (90,452 reasoning traces and 281,925 generations), and SearchGen-Corpus-1M (145,642 archived image and web search sessions, 559,973 unique URLs, and 370,733 cached downloads).

Because every search is pre-executed and frozen, the pipeline can be replayed offline. No live search API keys. No result drift. An expensive research workflow becomes a stable substrate for preference learning, reward modeling, search-policy design, and retrieval studies.

SearchGen-20K spans 22 real-world domains—from People & Professions and Screen & Performance Media down a long tail through Science, Fashion, and Infrastructure—mirroring how people actually prompt.

Repository Layout

./
├── README.md
├── LICENSE
├── agent/
│   ├── README.md                 # API and input/output contracts
│   ├── QUICKSTART.md             # offline end-to-end demonstration
│   ├── pyproject.toml
│   ├── src/searchgen_agent/      # Agentic Reasoner and Image Generator
│   ├── examples/                 # custom API adapter templates
│   └── tests/                    # offline integration tests
└── evaluation/
    ├── evaluate.py                 # evaluation entry point
    ├── aggregate_scores.py         # result aggregation
    ├── searchgen_eval/             # evaluator implementation
    ├── docs/                       # protocol, schemas, and prompt reference
    ├── examples/                   # manifest and environment templates
    └── tests/                      # regression and end-to-end tests

Agentic Reasoner with Search Tools and Image Generator

The agent/ package contains:

• an Agentic Reasoner with Search Tools that analyzes requests, conditionally performs web/image search, selects useful references, and writes a refined generation manifest;
• an Image Generator that consumes the manifest through a user-defined image API plugin; and
• user extension points for custom chat, search, authentication, and image-generation services.

The agent package defines the complete adaptation contracts:

• input dataset and reasoner output protocol;
• OpenAI-compatible chat-model protocol;
• custom web/image search request and response protocol;
• s4_generation_manifest.json handoff protocol;
• custom Image Generator input and output protocol; and
• generated image and metadata output protocol.

Install it with Python 3.10 or newer:

cd agent
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Follow agent/QUICKSTART.md for a fully offline run. To connect real services, implement the search signer and image-generator callback shown in agent/examples/.

Evaluation Environment

For full reproducibility, we recommend the released AgentGen-Bench-Evaluator, which is trained on top of Qwen3.5-9B. The evaluator requires Python 3.10 or newer:

cd evaluation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp examples/env.example .env

Set an OpenAI-compatible endpoint and judge model credentials in .env:

SEARCHGEN_EVAL_API_URL=https://your-openai-compatible-host.example/v1
SEARCHGEN_EVAL_API_KEY=replace_me

Then export them with set -a; source .env; set +a.

Serve the released evaluator with vLLM using the provided portable script:

pip install vllm
bash evaluation/serve_evaluator.sh
# Multi-GPU: one data-parallel replica per listed GPU
bash evaluation/serve_evaluator.sh --gpus 0,1,2,3

The script enforces --chat-template-content-format openai, which is required for the evaluator's multimodal OpenAI-style messages. For multiple GPUs it uses vLLM data parallelism by default (--data-parallel-size), with tensor parallel size one. See evaluation/serve_evaluator.sh for optional model, port, DP, and TP overrides.

Evaluation Quick Start

Download AgentGen-Bench, then prepare a predictions manifest following evaluation/examples/generated_images_manifest.example.jsonl. From the repository's evaluation/ directory, validate all inputs without API calls, replacing the benchmark paths with the location of your download:

python evaluate.py \
  --metadata /path/to/AgentGen-Bench/eval_metadata.jsonl \
  --benchmark-root /path/to/AgentGen-Bench \
  --predictions-manifest predictions.jsonl \
  --output-dir results \
  --model your-judge-model \
  --preflight

Remove --preflight and add --workers 16 to run evaluation. Completed successful examples resume automatically. Use --dry-run to inspect pending jobs, or repeat --bench-id and --generator to select subsets.

Evaluation

AgentGen-Bench separates no-search and search-intensive prompts so evaluation can distinguish missing world knowledge from rendering failures and damage caused by unnecessary retrieval.

The released evaluation protocol reports checklist satisfaction, adaptive-rubric satisfaction, prompt faithfulness, image quality, text rendering, AI naturalness, composition and aesthetics, physical plausibility, visual-reference consistency, and text-reference consistency. Component scores use a 0–3 scale with half points; non-applicable fields remain unscored.

Aggregate completed results with:

python aggregate_scores.py results --missing-policy skip

See the evaluation/ folder and its README.md for the complete commands, input/output schema, evaluation protocol, prompt reference, and tests.

License

The code release is licensed under the Apache License 2.0.

Citation

@article{searchgen,
  title={Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation},
  author={Wang, Haozhe and Feng, Weijia and Yu, Jinpeng and Liu, Che and Nie, Ping and Lin, Fangzhen and Liu, Jiaming and Huang, Ruihua and Lin, Jimmy and Chen, Wenhu and others},
  journal={arXiv preprint arXiv:2607.05382},
  year={2026}
}