Overview

DeepEye-SQL treats Text-to-SQL as a software engineering process rather than a single-shot generation task. It decomposes the problem into grounding, schema linking, implementation, debugging, and final selection, then coordinates those stages with structured snapshots and execution-aware checks.

The repository contains the research pipeline used for our SIGMOD 2026 paper, with support for BIRD, Spider, and Spider2. From-scratch reproduction instructions live in the dataset-specific runbooks under docs/.

Highlights

Capability	What it provides
Software-engineering pipeline	A staged workflow for grounding, linking, generation, revision, and selection.
Dynamic few-shot retrieval	Automatic training-set indexing with LLM-based question/SQL masking and preliminary-SQL-guided retrieval.
Checker-based SQL revision	Syntax, execution, and result-level repair before final selection.
Execution-aware selection	Candidate SQLs are compared with database feedback instead of relying only on model preference.
Structured snapshots	Long-running experiments are resumable, inspectable, and exportable.

News

Date	Update
2026-07-10	Qwen3.6-27B achieves 78.4 EX on the BIRD test set.
2026-07-02	Added dynamic few-shot retrieval, model-organized config templates, unified dataset wrappers, and public runbooks for BIRD, Spider, and Spider2.

Results

Benchmark	Metric	Score	Model	Public Output
BIRD-Dev	EX	74.5	Qwen3.6-27B	prediction JSON
BIRD-Test	EX	78.4	Qwen3.6-27B	not released
Spider2-Lite	official score	38.2	DeepSeek-R1	outputs
Spider2-Snow	official score	50.5	DeepSeek-R1	outputs

Architecture

Runbooks

The root README is intentionally kept as a project overview. Use the runbooks for setup, config edits, dataset preparation, execution commands, inspection, export, and evaluation.

Dataset	Runbook	Template families
BIRD	docs/bird.md	`config/template/*/config-bird-dev.toml`, `config-bird-test.toml`
Spider	docs/spider.md	`config/template/*/config-spider-test.toml`
Spider2	docs/spider2.md	`config/template/*/config-spider2-lite.toml`, `config-spider2-snow.toml`

Tracked config templates are grouped by model under config/template. Local experiment configs should be copied under config/local/<model>/, which is ignored by git.

Repository Map

Path	Purpose
app/	Core config, dataset, database, LLM, prompt, service, vector index, and pipeline code.
app/few_shot/	Dynamic few-shot masking, indexing, retrieval, and runtime preparation.
config/template/	Public model-organized TOML templates.
docs/	Dataset-specific runbooks for fresh-checkout reproduction.
runner/	Python entry points for individual stages, export, inspection, and evaluation.
script/	Shell wrappers for dataset-level runs.
results/	Released predictions and benchmark outputs.
`workspace/`	Generated local snapshots and intermediate outputs. Ignored by git.

Citation

If you find DeepEye-SQL useful in your research, please cite:

Paper: https://doi.org/10.1145/3802035

@article{10.1145/3802035,
author = {Li, Boyan and Chen, Chong and Xue, Zhujun and Mei, Yinan and Luo, Yuyu},
title = {DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework},
year = {2026},
issue_date = {June 2026},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {4},
number = {3},
url = {https://doi.org/10.1145/3802035},
doi = {10.1145/3802035},
journal = {Proc. ACM Manag. Data},
month = may,
articleno = {158},
numpages = {28},
keywords = {text-to-sql, databases, large language models}
}

License

This project is released under the MIT License. See LICENSE.

Acknowledgement

DeepEye-SQL builds on public benchmark ecosystems and OpenAI-compatible LLM serving stacks. We thank the maintainers of Spider, BIRD, Spider2, ChromaDB, OpenAI-compatible serving frameworks, and the broader Text-to-SQL research community.