{"id":17820190,"url":"https://github.com/HKUSTDial/NL2SQL360","last_synced_at":"2025-03-18T07:30:49.408Z","repository":{"id":220472562,"uuid":"749810395","full_name":"HKUSTDial/NL2SQL360","owner":"HKUSTDial","description":"Official repository for the paper “The Dawn of Natural Language to SQL: Are We Fully Ready?” (VLDB'24)","archived":false,"fork":false,"pushed_at":"2024-09-24T02:34:31.000Z","size":9017,"stargazers_count":94,"open_issues_count":2,"forks_count":10,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-01T04:55:55.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2406.01265","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HKUSTDial.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-29T12:52:44.000Z","updated_at":"2025-02-25T16:01:33.000Z","dependencies_parsed_at":"2024-03-11T09:30:11.291Z","dependency_job_id":"30e3b62a-20d7-48d6-859f-4a530d493896","html_url":"https://github.com/HKUSTDial/NL2SQL360","commit_stats":null,"previous_names":["bugmaker-boyan/hypersql","bugmaker-boyan/nl2sql360","hkustdial/nl2sql360"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUSTDial%2FNL2SQL360","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUSTDial%2FNL2SQL360/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUSTDial%2FNL2SQL360/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HKUSTDial%2FNL2SQL360/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HKUSTDial","download_url":"https://codeload.github.com/HKUSTDial/NL2SQL360/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244177648,"owners_count":20410993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-27T17:02:01.846Z","updated_at":"2025-03-18T07:30:48.324Z","avatar_url":"https://github.com/HKUSTDial.png","language":"Python","funding_links":[],"categories":["💬 Classic Model","💾 Practical Guide for Novice"],"sub_categories":["🔎How to evaluate your model:"],"readme":"# :mag_right:NL2SQL360\n\n\u003cdiv align=\"center\"\u003e\u003cimg width=\"25%\" src=\"./assets/nl2sql360.png\"\u003e\u003cimg width=\"75%\" src=\"./assets/leaderboard.png\"\u003e\u003c/div\u003e\n\n## :dizzy:Overview\n\n**NL2SQL360** is a testbed for fine-grained evaluation of NL2SQL solutions. Our testbed integrates existing NL2SQL benchmarks, a repository of NL2SQL models, and various evaluation metrics, which aims to provide an intuitive and user-friendly platform to enable both standard and customized performance evaluations. Users can utilize **NL2SQL360** to assess different NL2SQL methods against established benchmarks or tailor their evaluations based on specific criteria. This flexibility allows for testing solutions in specific data domains or analyzing performance on different characteristics of SQL queries. \n\nIn addition, we propose **SuperSQL**, which achieves competitive performance with execution accuracy of **87%** and **62.66%** on the Spider and BIRD test sets, respectively.\n\n## :tada:News\n\n[24/9/23] We release NL2SQL360 `1.1.0` version, which supports two new metrics **Reward-based VES** (RVES), **Soft-F1 Score** (F1), from [BIRD-Mini-Dev ](https://github.com/bird-bench/mini_dev) dataset. **Please update your package with** `pip install --upgrade nl2sql360`.\n\n[24/9/1] We have released our **[Homepage \u0026 Leaderboard](https://nl2sql360.github.io)!**\n\n[24/8/2] We have released CLI usage / Code usage tutorials. **Please [check out](#rocketquick-start)!**\n\n[24/7/30] We have refactored the code and released the official python package([nl2sql360 · PyPI](https://pypi.org/project/nl2sql360)). **Stay tuned for the complete documents!**\n\n[24/6/30] Our paper [The Dawn of Natural Language to SQL: Are We Fully Ready?](https://arxiv.org/abs/2406.01265) has been accepted by VLDB'24.\n\n## :balloon:Features\n\n- **Easy-to-use Evaluation**: Command Line Usage / Python Code Usage.\n- **Integrated Metrics**: Execution Accuracy / Exact-Match Accuracy / Valid Efficiency Score / Question Variance Testing.\n- **Multi-angle Performance**: Fine-grained performance (JOIN, Sub-query, etc.) / Scenario-based (Business Intelligence, etc.)\n\n## :wrench:Installation\n\n```bash\npip install nl2sql360\n```\n\n## :rocket:Quick Start\n\n\u003cdetails\u003e\u003csummary\u003ePrepare Dataset\u003c/summary\u003e\n\nDownload NL2SQL dataset to `DATASET_DIR_PATH`. The directory structure should be like:\n```bash\nDATASET_DIR_PATH:\n├─database\n│  ├─academic\n│  │  ├─academic.sqlite\n│  ├─college\n│  │  ├─college.sqlite\n├─dev.json\n├─tables.json\n```\n\n- `database` directory contains multiple subdirectories, which include the corresponding `sqlite` database file.\n- `dev.json` is the samples file in JSON format, which at least contains three keys for `NL Question`, `Gold SQL`, `Databae Id`. You can also add the key for `Sample Complexity` for categorizing samples into different difficulty levels.\n- `tables.json` contains all database schema, following [Spider Preprocess Procedure](https://github.com/taoyds/spider/tree/master/preprocess). **You can also ignore this file if you do not want to evaluate Exact-Match Accuracy Metic.**\n- Note that the name for `database` directory, samples file `dev.json` and tables file `tables.json` can be changed.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eImport Dataset into NL2SQL360\u003c/summary\u003e\n\n- CLI Usage:\n\n  - Create / Modify the YAML configuration following [NL2SQL360/examples/cli_examples/dataset_spider.yaml](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/cli_examples/dataset_spider.yaml).\n\n  - Save the YAML file to the path `DATASET_YAML_PATH`. Then run the command line:\n\n    ```bash\n    nl2sql360-cli dataset DATASET_YAML_PATH\n    ```\n\n- Code Usage:\n\n  - Create / Modify Python File following [NL2SQL360/examples/py_examples/dataset_import.py](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/py_examples/dataset_import.py).\n  - Run the python file to import dataset.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eEvaluation NL2SQL Model\u003c/summary\u003e\n\n- CLI Usage:\n\n  - Create / Modify the YAML configuration following [NL2SQL360/examples/cli_examples/evaluation.yaml](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/cli_examples/evaluation.yaml).\n\n  - Save the YAML file to the path `DATASET_YAML_PATH`. Then run the command line:\n\n    ```bash\n    nl2sql360-cli evaluate DATASET_YAML_PATH\n    ```\n\n- Code Usage:\n\n  - Create / Modify Python File following [NL2SQL360/examples/py_examples/evaluation.py](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/py_examples/evaluation.py).\n  - Run the python file to evaluate the model.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eQuery Multi-angle Performance\u003c/summary\u003e\n\n- CLI Usage:\n\n  - Create / Modify the YAML configuration following [NL2SQL360/examples/cli_examples/report.yaml](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/cli_examples/report.yaml).\n\n  - Save the YAML file to the path `DATASET_YAML_PATH`. Then run the command line:\n\n    ```bash\n    nl2sql360-cli report DATASET_YAML_PATH\n    ```\n\n  - The generated report will be in `save_path` specified in the YAML file.\n\n- Code Usage:\n  - Create / Modify Python File following [NL2SQL360/examples/py_examples/report.py](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/py_examples/report.py).\n  - Run the python file to generate report.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eDelete History Cache\u003c/summary\u003e\n\n- CLI Usage:\n\n  - Create / Modify the YAML configuration following [NL2SQL360/examples/cli_examples/delete_history.yaml](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/cli_examples/delete_history.yaml).\n\n  - Save the YAML file to the path `DATASET_YAML_PATH`. Then run the command line:\n\n    ```bash\n    nl2sql360-cli delete DATASET_YAML_PATH\n    ```\n\n- Code Usage:\n\n  - Create / Modify Python File following [NL2SQL360/examples/py_examples/delete_history.py](https://github.com/HKUSTDial/NL2SQL360/blob/master/examples/py_examples/delete_history.py).\n  - Run the python file to delete dataset / evaluation cache.\n\n\u003c/details\u003e\n\n## :dart:Road Map\n\n:white_check_mark:Release **NL2SQL360** evaluation code.\n\n:white_check_mark:Release **NL2SQL360** experiments data.\n\n:white_check_mark:Release **NL2SQL360** Official Python Package.\n\n## :floppy_disk:Experiment Data\n\nWe have released all experiment data used in our paper.\n\n[Download Link](https://drive.google.com/drive/folders/1SDwY30H2r6XNYeS53wcZNocVFm0hVgpz?usp=sharing)\n\n## :pushpin:Citation\n\n```\n@misc{li2024dawn,\n      title={The Dawn of Natural Language to SQL: Are We Fully Ready?}, \n      author={Boyan Li and Yuyu Luo and Chengliang Chai and Guoliang Li and Nan Tang},\n      year={2024},\n      eprint={2406.01265},\n      archivePrefix={arXiv},\n      primaryClass={id='cs.DB' full_name='Databases' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers database management, datamining, and data processing. Roughly includes material in ACM Subject Classes E.2, E.5, H.0, H.2, and J.1.'}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHKUSTDial%2FNL2SQL360","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHKUSTDial%2FNL2SQL360","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHKUSTDial%2FNL2SQL360/lists"}