
NL2SQL_Handbook
This is a continuously updated handbook for readers to easily track the latest NL2SQL (Text-to-SQL) techniques in the literature and provide practical guidance for researchers and practitioners. If we missed any interesting work, feel free to contact us.
Stars: 481

NL2SQL Handbook provides a comprehensive overview of Natural Language to SQL (NL2SQL) advancements, including survey papers, tutorial slides, and a river diagram of NL2SQL methods. It covers the evolution of NL2SQL solutions, module-based methods, benchmark development, and future directions. The repository also offers practical guides for beginners, access to high-performance language models, and evaluation metrics for NL2SQL models.
README:
From this repository, you can view the latest advancements in NL2SQL. This handbook corresponds to our survey paper: A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?. We also provide tutorial slides to summarize the key points of this survey. Based on the trends in the development of language models, we have created a river diagram of NL2SQL methods to trace the evolution of the NL2SQL field.
If you are a novice, don't worry—we have prepared a practical guide for you, covering a wide range of foundational materials here. We summarized NL2SQL related applications.
@article{liu2024survey,
title={A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?},
author={Liu, Xinyu and Shen, Shuyu and Li, Boyan and Ma, Peixian and Jiang, Runzhi and Zhang, Yuxin and Fan, Ju and Li, Guoliang and Tang, Nan and Luo, Yuyu},
journal={arXiv preprint arXiv:2408.05109},
year={2024}
}
Translating users' natural language queries (NL) into SQL queries can significantly reduce barriers to accessing relational databases and support various commercial applications. The performance of NL2SQL has been greatly improved with the emergence of language models (LMs). In this context, it is crucial to assess our current position, determine the NL2SQL solutions that should be adopted for specific scenarios by practitioners, and identify the research topics that researchers should explore next.
-
Model: NL2SQL translation techniques that tackle not only NL ambiguity and under-specification, but also properly map NL with database schema and instances;
-
Data: From the collection of training data, data synthesis due to training data scarcity, to NL2SQL benchmarks;
-
Evaluation: Evaluating NL2SQL methods from multiple angles using different metrics and granularities;
-
Error Analysis: analyzing NL2SQL errors to find the root cause and guiding NL2SQL models to evolve.
we categorize the challenges of NL2SQL into five levels, each addressing specific hurdles. The first three levels cover challenges that have been or are currently being addressed, reflecting the progressive development of NL2SQL. The fourth level represents the challenges we aim to tackle in the LLMs stage, while the fifth level outlines our vision for NL2SQL system in the next five years.
We describe the evolution of NL2SQL solutions from the perspective of language models, categorizing it into four stages. For each stage of NL2SQL, we analyze the changes in target users and the extent to which challenges are addressed.
We summarize the key modules of NL2SQL solutions utilizing the language model.
- Pre-processing serves as an enhancement to the model’s inputs in the NL2SQL parsing process. You can get more details from this chapter: Pre-Processing
- NL2SQL translation methods constitute the core of the NL2SQL solution, responsible for converting input natural language queries into SQL queries. You can get more details from this chapter: NL2SQL Translation Methods
- Post-processing is a crucial step to refine the generated SQL queries, ensuring they meet user expectations more accurately. You can get more details from this chapter: Post-Processing
- A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?
- Next-generation databas interfaces: A survey of llm-based text-to-sql.
- Large Language Model Enhanced Text-to-SQL Generation: A Survey.
- From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems.
- A Survey on Employing Large Language Models for Text-to-SQL Tasks.
- Natural language interfaces for tabular data querying and visualization: A survey.
- Natural Language Interfaces for Databases with Deep Learning.
- A survey on deep learning approaches for text-to-SQL.
- Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect.
- A Deep Dive into Deep Learning Approaches for Text-to-SQL Systems.
- State of the Art and Open Challenges in Natural Language Interfaces to Data.
- Natural language to SQL: Where are we today?
-
OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment.
-
Cracking SQL Barrier: An LLM-based Dialect Translation System.
-
SNAILS: Schema Naming Assessments for Improved LLM-Based SQL Inference.
-
CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL.
-
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows.
-
ROUTE: ROBUST MULTITASK TUNING AND COLLAB- ORATION FOR TEXT-TO-SQL.
-
Confidence Estimation for Error Detection in Text-to-SQL Systems.
-
DBCopilot: Scaling Natural Language Querying to Massive Databases.
-
Boosting Text-to-SOL through Multi- grained Error Identification.
-
Gen-SQL: Efficient Text-to-SQL By Bridging Natural Language Question And Database Schema With Pseudo-Schema.
-
Utilising Large Language Models for Adversarial Attacks in Text-to-SQL: A Perpetrator and Victim Approach
-
You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQL.
-
Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search
-
NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation.
-
OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale.
-
Is Long Context AIl You Need? Leveraging LLM's ExtendedContext for NL2SQL.
-
MCTS-SQL: An Effective Framework for Text-to-SQL with Monte Carlo Tree Search.
-
SQL-o1: A Self-Reward Heuristic Dynamic Search Method for Text-to-SQL.
-
FEATHER-SQL: A LIGHTWEIGHT NL2SQL FRAME- WORK WITH DUAL-MODEL COLLABORATION PARADIGM FOR SMALL LANGUAGE MODELS.
-
FI-NL2PY2SQL: Financial Industry NL2SQL Innovation Model Based on Python and Large Language Model.
-
FGCSQL: A Three-Stage Pipeline for Large Language Model-Driven Chinese Text-to-SQL
-
Transforming Medical Data Access: The Role and Challenges of Recent Language Models in SQL Query Automation.
-
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation.
-
Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation.
-
Generating Succinct Descriptions of Database Schemata for Cost-Efficient Prompting of Large Language Models.
-
ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems.
-
CodeS: Towards Building Open-source Language Models for Text-to-SQL.
-
FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis.
-
METASQL: A Generate-then-Rank Framework for Natural Language to SQL Translation.
-
Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning.
-
Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark.
-
I Need Help! Evaluating LLM’s Ability to Ask for Users’ Support: A Case Study on Text-to-SQL Generation.
-
PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL.
-
Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruning.
-
Research and Practice on Database Interaction Based on Natural Language Processing
-
XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL.
-
A Plug-and-Play Natural Language Rewriter for Natural Language to SQL.
-
In-Context Reinforcement Learning based Retrieval-Augmented Generation for Text-to-SQL.
-
TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring.
-
LAIA-SQL: Enhancing Natural Language to SQL Generation in Multi-Table QA via Task Decomposition and Keyword Extraction
-
Research on Large Model Text-to-SQL Optimization Method for Intelligent Interaction in the Field of Construction Safety.
-
SQLh-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging.
-
Grounding Natural Language to SQL Translation with Data-Based Self-Explanations.
-
E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL.
-
DB-GPT: Empowering Database Interactions with Private Large Language Models.
-
The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models.
-
PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency.
-
CoE-SQL: In-Context Learning for Multi-Turn Text-to-SQL with Chain-of-Editions.
-
AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries.
-
Text-to-SQL Calibration: No Need to Ask—Just Rescale Model Probabilities.
-
Few-shot Text-to-SQL Translation using Structure and Content Prompt Learning.
-
CatSQL: Towards Real World Natural Language to SQL Applications.
-
DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction.
-
Data Ambiguity Strikes Back: How Documentation Improves GPT's Text-to-SQL.
-
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought.
-
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL.
-
Graphix-T5: Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing.
-
Improving Generalization in Language Model-based Text-to-SQL Semantic Parsing: Two Simple Semantic Boundary-based Techniques.
-
G3R: A Graph-Guided Generate-and-Rerank Framework for Complex and Cross-domain Text-to-SQL Generation.
-
Importance of Synthesizing High-quality Data for Text-to-SQL Parsing.
-
Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL.
-
MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL.
-
SQLformer: Deep Auto-Regressive Query Graph Generation for Text-to-SQL Translation.
We create a timeline of the benchmark's development and mark relevant milestones. You can get more details from this chapter: 📊 Benchmark
- 🎯Sovle Open NL2SQL Problem
- 🎯Develop Cost-effective NL2SQL Methods
- 🎯Make NL2SQL Solutions Trustworthy
- 🎯NL2SQL with Ambiguous and Unspecified NL Queries
- 🎯Adaptive Training Data Synthesis
You can get more information from our subsection. We introduce representative papers on related concepts:
- We collect NL2SQL benchmark features and download links for you. You can get more details from this chapter: Benchmark
- The analysis code for benchmarks is available in the
src/dataset_analysis
directory. Benchmark analysis reports can be found in thereport/
directory.
-
Litgpt Repository Link
This repository offers access to over 20 high-performance large language models (LLMs) with comprehensive guides for pretraining, fine-tuning, and deploying at scale. It is designed to be beginner-friendly with from-scratch implementations and no complex abstractions.
-
LLaMA-Factory Repository Link Unified Efficient Fine-Tuning of 100+ LLMs. Integrating various models with scalable training resources, advanced algorithms, practical tricks, and comprehensive experiment monitoring tools, this setup enables efficient and faster inference through optimized APIs and UIs.
-
Fine-tuning and In-Context learning for BIRD-SQL benchmark Repository Link
A tutorial for both Fine-tuning and In-Context Learning is provided by the BIRD-SQL benchmark.
We collect NL2SQL evaluation metrics for you. You can get more details from this chapter: Evaluation
-
NLSQL360 Repository Link
NL2SQL360 is a testbed for fine-grained evaluation of NL2SQL solutions. Our testbed integrates existing NL2SQL benchmarks, a repository of NL2SQL models, and various evaluation metrics, which aims to provide an intuitive and user-friendly platform to enable both standard and customized performance evaluations.
-
Test-suite-sql-eval Repository Link
This repo contains a test suite evaluation metric for 11 text-to-SQL tasks. It is now the official metric of Spider, SParC, and CoSQL, and is also now available for Academic, ATIS, Advising, Geography, IMDB, Restaurants, Scholar, and Yelp (building on the amazing work by Catherine and Jonathan).
-
BIRD-SQL-Official Repository Link
It is now the official tool of BIRD-SQL. It is the first tool to propose VES and give an official test suite.
You can get some inspiration from the Roadmap and Decision Flow.
- Chat2DB: AI-driven database tool and SQL client, The hottest GUI client, supporting MySQL, Oracle, PostgreSQL, DB2, SQL Server, DB2, SQLite, H2, ClickHouse, and more.
- DB-GPT: AI Native Data App Development framework with AWEL(Agentic Workflow Expression Language) and Agents.
- Postgres.new: In-browser Postgres sandbox with AI assistance.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for NL2SQL_Handbook
Similar Open Source Tools

NL2SQL_Handbook
NL2SQL Handbook provides a comprehensive overview of Natural Language to SQL (NL2SQL) advancements, including survey papers, tutorial slides, and a river diagram of NL2SQL methods. It covers the evolution of NL2SQL solutions, module-based methods, benchmark development, and future directions. The repository also offers practical guides for beginners, access to high-performance language models, and evaluation metrics for NL2SQL models.

EvalAI
EvalAI is an open-source platform for evaluating and comparing machine learning (ML) and artificial intelligence (AI) algorithms at scale. It provides a central leaderboard and submission interface, making it easier for researchers to reproduce results mentioned in papers and perform reliable & accurate quantitative analysis. EvalAI also offers features such as custom evaluation protocols and phases, remote evaluation, evaluation inside environments, CLI support, portability, and faster evaluation.

Awesome-Text2SQL
Awesome Text2SQL is a curated repository containing tutorials and resources for Large Language Models, Text2SQL, Text2DSL, Text2API, Text2Vis, and more. It provides guidelines on converting natural language questions into structured SQL queries, with a focus on NL2SQL. The repository includes information on various models, datasets, evaluation metrics, fine-tuning methods, libraries, and practice projects related to Text2SQL. It serves as a comprehensive resource for individuals interested in working with Text2SQL and related technologies.

Awesome-LLM4Graph-Papers
A collection of papers and resources about Large Language Models (LLM) for Graph Learning (Graph). Integrating LLMs with graph learning techniques to enhance performance in graph learning tasks. Categorizes approaches based on four primary paradigms and nine secondary-level categories. Valuable for research or practice in self-supervised learning for recommendation systems.

tt
Connect multiple CEX and DEX across multi messaging platforms. Place order, inquire your balance, access AI agent and more through plugins. Easily deploy via Docker on self-hosted platform or PAAS.

Awesome-RL-based-LLM-Reasoning
This repository is dedicated to enhancing Language Model (LLM) reasoning with reinforcement learning (RL). It includes a collection of the latest papers, slides, and materials related to RL-based LLM reasoning, aiming to facilitate quick learning and understanding in this field. Starring this repository allows users to stay updated and engaged with the forefront of RL-based LLM reasoning.

Cyberion-Spark-X
Cyberion-Spark-X is a powerful open-source tool designed for cybersecurity professionals and data analysts. It provides advanced capabilities for analyzing and visualizing large datasets to detect security threats and anomalies. The tool integrates with popular data sources and supports various machine learning algorithms for predictive analytics and anomaly detection. Cyberion-Spark-X is user-friendly and highly customizable, making it suitable for both beginners and experienced professionals in the field of cybersecurity and data analysis.

awesome-production-llm
This repository is a curated list of open-source libraries for production large language models. It includes tools for data preprocessing, training/finetuning, evaluation/benchmarking, serving/inference, application/RAG, testing/monitoring, and guardrails/security. The repository also provides a new category called LLM Cookbook/Examples for showcasing examples and guides on using various LLM APIs.

lobe-cli-toolbox
Lobe CLI Toolbox is an AI CLI Toolbox designed to enhance git commit and i18n workflow efficiency. It includes tools like Lobe Commit for generating Gitmoji-based commit messages and Lobe i18n for automating the i18n translation process. The toolbox also features Lobe label for automatically copying issues labels from a template repo. It supports features such as automatic splitting of large files, incremental updates, and customization options for the OpenAI model, API proxy, and temperature.

Awesome-LLM-Post-training
The Awesome-LLM-Post-training repository is a curated collection of influential papers, code implementations, benchmarks, and resources related to Large Language Models (LLMs) Post-Training Methodologies. It covers various aspects of LLMs, including reasoning, decision-making, reinforcement learning, reward learning, policy optimization, explainability, multimodal agents, benchmarks, tutorials, libraries, and implementations. The repository aims to provide a comprehensive overview and resources for researchers and practitioners interested in advancing LLM technologies.

Awesome-System2-Reasoning-LLM
The Awesome-System2-Reasoning-LLM repository is dedicated to a survey paper titled 'From System 1 to System 2: A Survey of Reasoning Large Language Models'. It explores the development of reasoning Large Language Models (LLMs), their foundational technologies, benchmarks, and future directions. The repository provides resources and updates related to the research, tracking the latest developments in the field of reasoning LLMs.

Awesome-LLM-Ensemble
Awesome-LLM-Ensemble is a collection of papers on LLM Ensemble, focusing on the comprehensive use of multiple large language models to benefit from their individual strengths. It provides a systematic review of recent developments in LLM Ensemble, including taxonomy, methods for ensemble before, during, and after inference, benchmarks, applications, and related surveys.

BadukMegapack
BadukMegapack is an installer for various AI Baduk (Go) programs, designed for baduk players who want to easily access and use a variety of baduk AI programs without complex installations. The megapack includes popular programs like Lizzie, KaTrain, Sabaki, KataGo, LeelaZero, and more, along with weight files for different AI models. Users can update their graphics card drivers before installation for optimal performance.

LLMFarm
LLMFarm is an iOS and MacOS app designed to work with large language models (LLM). It allows users to load different LLMs with specific parameters, test the performance of various LLMs on iOS and macOS, and identify the most suitable model for their projects. The tool is based on ggml and llama.cpp by Georgi Gerganov and incorporates sources from rwkv.cpp by saharNooby, Mia by byroneverson, and LlamaChat by alexrozanski. LLMFarm features support for MacOS (13+) and iOS (16+), various inferences and sampling methods, Metal compatibility (not supported on Intel Mac), model setting templates, LoRA adapters support, LoRA finetune support, LoRA export as model support, and more. It also offers a range of inferences including LLaMA, GPTNeoX, Replit, GPT2, Starcoder, RWKV, Falcon, MPT, Bloom, and others. Additionally, it supports multimodal models like LLaVA, Obsidian, and MobileVLM. Users can customize inference options through JSON files and access supported models for download.
For similar tasks

Azure-Analytics-and-AI-Engagement
The Azure-Analytics-and-AI-Engagement repository provides packaged Industry Scenario DREAM Demos with ARM templates (Containing a demo web application, Power BI reports, Synapse resources, AML Notebooks etc.) that can be deployed in a customer’s subscription using the CAPE tool within a matter of few hours. Partners can also deploy DREAM Demos in their own subscriptions using DPoC.

sorrentum
Sorrentum is an open-source project that aims to combine open-source development, startups, and brilliant students to build machine learning, AI, and Web3 / DeFi protocols geared towards finance and economics. The project provides opportunities for internships, research assistantships, and development grants, as well as the chance to work on cutting-edge problems, learn about startups, write academic papers, and get internships and full-time positions at companies working on Sorrentum applications.

tidb
TiDB is an open-source distributed SQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible and features horizontal scalability, strong consistency, and high availability.

zep-python
Zep is an open-source platform for building and deploying large language model (LLM) applications. It provides a suite of tools and services that make it easy to integrate LLMs into your applications, including chat history memory, embedding, vector search, and data enrichment. Zep is designed to be scalable, reliable, and easy to use, making it a great choice for developers who want to build LLM-powered applications quickly and easily.

telemetry-airflow
This repository codifies the Airflow cluster that is deployed at workflow.telemetry.mozilla.org (behind SSO) and commonly referred to as "WTMO" or simply "Airflow". Some links relevant to users and developers of WTMO: * The `dags` directory in this repository contains some custom DAG definitions * Many of the DAGs registered with WTMO don't live in this repository, but are instead generated from ETL task definitions in bigquery-etl * The Data SRE team maintains a WTMO Developer Guide (behind SSO)

mojo
Mojo is a new programming language that bridges the gap between research and production by combining Python syntax and ecosystem with systems programming and metaprogramming features. Mojo is still young, but it is designed to become a superset of Python over time.

pandas-ai
PandasAI is a Python library that makes it easy to ask questions to your data in natural language. It helps you to explore, clean, and analyze your data using generative AI.

databend
Databend is an open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake. With its focus on fast query execution and data ingestion, it's designed for complex analysis of the world's largest datasets.
For similar jobs

sweep
Sweep is an AI junior developer that turns bugs and feature requests into code changes. It automatically handles developer experience improvements like adding type hints and improving test coverage.

teams-ai
The Teams AI Library is a software development kit (SDK) that helps developers create bots that can interact with Teams and Microsoft 365 applications. It is built on top of the Bot Framework SDK and simplifies the process of developing bots that interact with Teams' artificial intelligence capabilities. The SDK is available for JavaScript/TypeScript, .NET, and Python.

ai-guide
This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.

classifai
Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.

chatbot-ui
Chatbot UI is an open-source AI chat app that allows users to create and deploy their own AI chatbots. It is easy to use and can be customized to fit any need. Chatbot UI is perfect for businesses, developers, and anyone who wants to create a chatbot.

BricksLLM
BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM: * Set LLM usage limits for users on different pricing tiers * Track LLM usage on a per user and per organization basis * Block or redact requests containing PIIs * Improve LLM reliability with failovers, retries and caching * Distribute API keys with rate limits and cost limits for internal development/production use cases * Distribute API keys with rate limits and cost limits for students

uAgents
uAgents is a Python library developed by Fetch.ai that allows for the creation of autonomous AI agents. These agents can perform various tasks on a schedule or take action on various events. uAgents are easy to create and manage, and they are connected to a fast-growing network of other uAgents. They are also secure, with cryptographically secured messages and wallets.

griptape
Griptape is a modular Python framework for building AI-powered applications that securely connect to your enterprise data and APIs. It offers developers the ability to maintain control and flexibility at every step. Griptape's core components include Structures (Agents, Pipelines, and Workflows), Tasks, Tools, Memory (Conversation Memory, Task Memory, and Meta Memory), Drivers (Prompt and Embedding Drivers, Vector Store Drivers, Image Generation Drivers, Image Query Drivers, SQL Drivers, Web Scraper Drivers, and Conversation Memory Drivers), Engines (Query Engines, Extraction Engines, Summary Engines, Image Generation Engines, and Image Query Engines), and additional components (Rulesets, Loaders, Artifacts, Chunkers, and Tokenizers). Griptape enables developers to create AI-powered applications with ease and efficiency.