{"id":14181735,"url":"https://github.com/RUCKBReasoning/RESDSQL","last_synced_at":"2025-08-07T14:31:09.411Z","repository":{"id":133001605,"uuid":"600767872","full_name":"RUCKBReasoning/RESDSQL","owner":"RUCKBReasoning","description":"The Pytorch implementation of RESDSQL (AAAI 2023).","archived":false,"fork":false,"pushed_at":"2024-05-13T02:45:33.000Z","size":2394,"stargazers_count":233,"open_issues_count":13,"forks_count":58,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-08-18T11:13:38.280Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2302.05965","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RUCKBReasoning.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-12T14:44:22.000Z","updated_at":"2024-08-15T02:45:49.000Z","dependencies_parsed_at":"2024-04-19T03:32:53.370Z","dependency_job_id":"f617e451-c679-4ca7-adae-7e95a644b9a0","html_url":"https://github.com/RUCKBReasoning/RESDSQL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCKBReasoning%2FRESDSQL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCKBReasoning%2FRESDSQL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCKBReasoning%2FRESDSQL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RUCKBReasoning%2FRESDSQL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RUCKBReasoning","download_url":"https://codeload.github.com/RUCKBReasoning/RESDSQL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229045117,"owners_count":18011444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-18T11:04:11.774Z","updated_at":"2024-12-10T10:31:10.363Z","avatar_url":"https://github.com/RUCKBReasoning.png","language":"Python","funding_links":[],"categories":["💬 Classic Model"],"sub_categories":[],"readme":"\u003cp align=\"left\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"resdsql.png\" width=\"700\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n# RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL\nThis is the official implementation of the paper \"RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL\" (AAAI 2023).\n\nIf this repository could help you, please cite the following paper:\n```\n@inproceedings{li2022resdsql,\n  author = {Haoyang Li and Jing Zhang and Cuiping Li and Hong Chen},\n  title = \"RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL\",\n  booktitle = \"AAAI\",\n  year = \"2023\"\n}\n```\n\n`Update (2023.3.13):` We evaluated our method on a diagnostic evaluation benchmark, [Dr.Spider](https://github.com/awslabs/diagnostic-robustness-text-to-sql), which contains 17 test sets to measure the robustness of Text-to-SQL parsers under different perturbation perspectives.\n\n`Update (2023.5.19):` We added support for [CSpider](https://taolusi.github.io/CSpider-explorer/), a Chinese Text-to-SQL benchmark with Chinese questions, English database schema, and corresponding SQL queries.\n\n`Update (2023.11.1):` We are excited to present our text-to-SQL demo, available at https://github.com/RUCKBReasoning/text2sql-demo. This demo showcases the capabilities of our newly developed pre-trained language model CodeS, which has been specifically tailored for text-to-SQL tasks. Additionally, we have included comprehensive instructions on how to build the demo using your databases. We encourage you to experiment with it and explore its features! 🔥\n\n`Update (2024.4.12):` We are thrilled to announce that our latest work, CodeS, has been accepted by SIGMOD 2024. CodeS represents a significant advancement over RESDSQL, incorporating a more powerful language model. We also re-developed the schema filter to make it easy to use. For an in-depth look at CodeS, please consult our paper available at [CodeS-paper](https://arxiv.org/abs/2402.16347). Additionally, we have made the source code publicly available at [CodeS-code](https://github.com/RUCKBReasoning/codes) for community use and feedback.\n\n`Update (2024.4.19):` We are excited to announce the release of our newly developed schema filter, boasting 3 billion parameters and offering bilingual support for both Chinese and English. This tool is now available as an independent component and can be accessed at [text2sql-schema-filter](https://github.com/RUCKBReasoning/text2sql-schema-filter). If you're looking to enhance your text-to-SQL system with a schema filter, we encourage you to give it a try. \n\n## Overview\nWe introduce a new Text-to-SQL parser, **RESDSQL** (**R**anking-enhanced **E**ncoding plus a **S**keleton-aware **D**ecoding framework for Text-to-**SQL**), which attempts to decoulpe the schema linking and the skeleton parsing to reduce the difficulty of Text-to-SQL. More details can be found in our [paper](https://arxiv.org/abs/2302.05965). All experiments are conducted on a single NVIDIA A100 80G GPU.\n\n## Evaluation Results\nWe evaluate RESDSQL on six benchmarks: Spider, Spider-DK, Spider-Syn, Spider-Realistic, Dr.Spider, and CSpider. We adopt two metrics: Exact-set-Match accuracy (EM) and EXecution accuracy (EX). Let's look at the following numbers:\n\n**On Spider:**\n\n| Model | Dev EM | Dev EX | Test EM | Test EX |\n|-------|--------|--------|---------|---------|\n| RESDSQL-3B+NatSQL | **80.5%** | **84.1%** | **72.0%** | **79.9%** |\n| RESDSQL-3B | 78.0% | 81.8% | - | - |\n| RESDSQL-Large+NatSQL | 76.7% | 81.9% | - | - |\n| RESDSQL-Large | 75.8% | 80.1% | - | - |\n| RESDSQL-Base+NatSQL | 74.1% | 80.2% | - | - |\n| RESDSQL-Base | 71.7% | 77.9% | - | - |\n\n**On Spider-DK, Spider-Syn, and Spider-Realistic:**\n\n| Model | DK EM | DK EX | Syn EM | Syn EX | Realistic EM | Realistic EX |\n|-------|-------|-------|--------|--------|--------------|--------------|\n| RESDSQL-3B+NatSQL| 53.3% | 66.0% | 69.1% | 76.9% | 77.4% | 81.9% |\n\n**On Dr.Spider's perturbation sets:**\nFollowing Dr.Spider, we only report **EX** for each post-perturbation set and choose PICARD and CodeX as our baseline methods.\n\n| Perturbation set | PICARD | CodeX | RESDSQL-3B | RESDSQL-3B+NatSQL |\n|------------------|--------|-------|-------------------|-----|\n| DB-Schema-synonym | 56.5% | 62.0% | 63.3% | **68.3%** |\n| DB-Schema-abbreviation | 64.7% | 68.6% | 64.5% | **70.0%** |\n| DB-DBcontent-equivalence | 43.7% | **51.6%** | 40.3% | 40.1% |\n| NLQ-Keyword-synonym | 66.3% | 55.5% | 67.5% | **72.4%** |\n| NLQ-Keyword-carrier | 82.7% | 85.2% | **86.7%** | 83.5% |\n| NLQ-Column-synonym | 57.2% | 54.7% | 57.4% | **63.1%** |\n| NLQ-Column-carrier | 64.9% | 51.1% | **69.9%** | 63.9% |\n| NLQ-Column-attribute | 56.3% | 46.2% | 58.8% | **71.4%** |\n| NLQ-Column-value | 69.4%  | 71.4% | 73.4% | **76.6%** |\n| NLQ-Value-synonym | 53.0% | **59.9%** | 53.8% | 53.2% |\n| NLQ-Multitype | 57.1%  | 53.7% | 60.1% | **60.7%** |\n| NLQ-Others | 78.3% | 69.7% | 77.3% | **79.0%** |\n| SQL-Comparison | 68.0% | 66.9% | 70.2% | **82.0%** |\n| SQL-Sort-order | 74.5% | 57.8% | 79.7% | **85.4%** |\n| SQL-NonDB-number | 77.1% | **89.3%** | 83.2% | 85.5% |\n| SQL-DB-text | 65.1% | 72.4% | 67.8% | **74.3%** |\n| SQL-DB-number | 85.1% | 79.3% | 85.4% | **88.8%** |\n| Average | 65.9% | 64.4% | 68.2% | **71.7%** |\n\nNotice: We also employed the modified test suite script (see this [issue](https://github.com/awslabs/diagnostic-robustness-text-to-sql/issues/1)) to evaluate the model-generated results, but obtained the same numbers as above. Nevertheless, we suggest that further work should use their modified script to evaluate Dr.Spider.\n\n**On CSpider's development set:**\n| Model | EM | EXEC | \n| ----- | -- | ---- | \n| RESDSQL-3B+NatSQL | **66.3%** | **81.1%** | \n| RESDSQL-Large+NatSQL | 64.3% | 81.1% | \n| LGESQL + GTL + Electra + QT | 64.0% | - |\n| LGESQL + ELECTRA + QT | 64.5% | - |\n| RESDSQL-Base+NatSQL | 61.7% | 78.1% | \n\n\n## Prerequisites\nCreate a virtual anaconda environment:\n```sh\nconda create -n your_env_name python=3.8.5\n```\nActive it and install the cuda version Pytorch:\n```sh\nconda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch\n```\nInstall other required modules and tools:\n```sh\npip install -r requirements.txt\npip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz\npython nltk_downloader.py\n```\nCreate several folders:\n```sh\nmkdir eval_results\nmkdir models\nmkdir tensorboard_log\nmkdir third_party\nmkdir predictions\n```\nClone evaluation scripts:\n```sh\ncd third_party\ngit clone https://github.com/ElementAI/spider.git\ngit clone https://github.com/ElementAI/test-suite-sql-eval.git\nmv ./test-suite-sql-eval ./test_suite\ncd ..\n```\n\n## Prepare data\nDownload [data](https://drive.google.com/file/d/19tsgBGAxpagULSl9r85IFKIZb4kyBGGu/view?usp=sharing) **(including Spider, Spider-DK, Spider-Syn, Spider-Realistic, Dr.Spider, and CSpider)** and [database](https://drive.google.com/file/d/1s4ItreFlTa8rUdzwVRmUR2Q9AHnxbNjo/view?usp=share_link) and then unzip them:\n```sh\nunzip data.zip\nunzip database.zip\n```\nNotice: Dr.Spider has been preprocessed following the instructions on its Github page.\n\n## Inference\nAll evaluation results can be easily reproduced through our released scripts and checkpionts.\n### Step1: Prepare Checkpoints\nBecause RESDSQL is a two-stage algorithm, therefore, you should first download cross-encoder checkpoints. Here are links: \n| Cross-encoder Checkpoints | Google Drive | Baidu Netdisk |\n|----------|-----------|--------------|\n| text2natsql_schema_item_classifier| [Link](https://drive.google.com/file/d/1UWNj1ZADfKa1G5I4gBYCJeEQO6piMg4G/view?usp=share_link) | [Link](https://pan.baidu.com/s/15eGPMSx7K8oLV4hkjnCzaw) (pwd: 18w8) |\n| text2sql_schema_item_classifier | [Link](https://drive.google.com/file/d/1zHAhECq1uGPR9Rt1EDsTai1LbRx0jYIo/view?usp=share_link) | [Link](https://pan.baidu.com/s/1trSi8OBOcPo5NkZb_o-T4g) (pwd: dr62) |\n| xlm_roberta_text2natsql_schema_item_classifier (trained on CSpider) | - | [Link](https://pan.baidu.com/s/1oTkuqoU-RWr3QsNC9Y-RpA) (pwd: 3sdu) |\n\nThen, you should download T5 (for Spider) or mT5 (for CSpider) checkpoints: \n| T5/mT5 Checkpoints | Google Drive/OneDrive | Baidu Netdisk |\n|-------|-------|-------|\n| text2natsql-t5-3b | [OneDrive link](https://1drv.ms/u/s!Ak05bBUBFYiktcdziiE79xaeKtO6qg?e=e9424n) | [Link](https://pan.baidu.com/s/1ReHso0QgX5aT-hGySUnlrQ) (pwd: 4r98) |\n| text2sql-t5-3b | [Google Drive link](https://drive.google.com/file/d/1M-zVeB6TKrvcIzaH8vHBIKeWqPn95i11/view?usp=sharing) | [Link](https://pan.baidu.com/s/1mZxakfes4wRSEwnRW43i5A) (pwd: sc62) |\n| text2natsql-t5-large | [Google Drive link](https://drive.google.com/file/d/1ZwFsH24_qKC3xwYdedPi6T_8argguWHe/view?usp=sharing) | [Link](https://pan.baidu.com/s/18H8lgnv9gfXmUo_oO_CdOA) (pwd: 7iyq) |\n| text2sql-t5-large | [Google Drive link](https://drive.google.com/file/d/1-xwtKwfJZSrmJrU-_Xdkx1kPuZao7r7e/view?usp=sharing) | [Link](https://pan.baidu.com/s/1Mwg0OZZ48APEq9jPvQQNtw) (pwd: q58k) |\n| text2natsql-t5-base | [Google Drive link](https://drive.google.com/file/d/1QyfSfHHrxfIM5X9gKUYNr_0ZRVvb1suV/view?usp=share_link) | [Link](https://pan.baidu.com/s/1XegaZFvXuZ_jf3P-9YPQCQ) (pwd: pyxf) |\n| text2sql-t5-base | [Google Drive link](https://drive.google.com/file/d/1lqZ81f_fSZtg6BRcRw1-Ol-RJCcKRsmH/view?usp=sharing) | [Link](https://pan.baidu.com/s/1-6H7zStq0WCJHTjDuVspoQ) (pwd: wuek) |\n| text2natsql-mt5-xl-cspider (trained on CSpider) | - | [Link](https://pan.baidu.com/s/1tFkGOiw5ETB83-Ct3MuVXA) (pwd: y7ei) |\n| text2natsql-mt5-large-cspider (trained on CSpider) | - | [Link](https://pan.baidu.com/s/1LUjL-2nwNfUJhzI3cm7aEQ) (pwd: ydqk) |\n| text2natsql-mt5-base-cspider (trained on CSpider) | - | [Link](https://pan.baidu.com/s/1tbEUIBPUTA2Oz7K2lHT9oA) (pwd: d8b8) |\n\nThe checkpoints should be placed in the `models` folder. \n\nFor CSpider, we only provide the NatSQL version because its performance is better than SQL in our pre-experiments. To support CSpider, we replace roberta-large with xlm-roberta-large in the first stage and replace t5 with mt5 in the second stage.\n\n### Step2: Run Inference\nThe inference scripts are located in `scripts/inference`. \nConcretely, `infer_text2natsql.sh` is the inference script of RESDSQL-{Base, Large, 3B}+NatSQL, and `infer_text2sql.sh` is the inference script of RESDSQL-{Base, Large, 3B}. For example, you can run the inference of RESDSQL-3B+NatSQL on Spider's dev set via:\n```sh\nsh scripts/inference/infer_text2natsql.sh 3b spider\n```\nThe first argument (model scale) can be selected from `[base, large, 3b]` and the second argument (dataset name) can be selected from `[spider, spider-realistic, spider-syn, spider-dk, DB_schema_synonym, DB_schema_abbreviation, DB_DBcontent_equivalence, NLQ_keyword_synonym, NLQ_keyword_carrier, NLQ_column_synonym, NLQ_column_carrier, NLQ_column_attribute, NLQ_column_value, NLQ_value_synonym, NLQ_multitype, NLQ_others, SQL_comparison, SQL_sort_order, SQL_NonDB_number, SQL_DB_text, SQL_DB_number]`.\n\nThe predicted SQL queries are recorded in `predictions/{dataset_name}/{model_name}/pred.sql`.\n\n**Inference on CSpider's Dev Set (New Feature)**\nWe also provide inference scripts to run RESDSQL-{Base, Large, 3B}+NatSQL on CSpider's development set. Here is an example:\n```sh\nsh scripts/inference/infer_text2natsql_cspider.sh 3b\n```\nThe first argument (model scale) can be selected from `[base, large, 3b]`.\n\n## Training on Spider\nWe provide scripts in `scripts/train/text2natsql` and `scripts/train/text2sql` to train RESDSQL on Spider's training set and evaluate on Spider's dev set.\n\n**RESDSQL-{Base, Large, 3B}+NatSQL**\n```sh\n# Step1: preprocess dataset\nsh scripts/train/text2natsql/preprocess.sh\n# Step2: train cross-encoder\nsh scripts/train/text2natsql/train_text2natsql_schema_item_classifier.sh\n# Step3: prepare text-to-natsql training and development set for T5\nsh scripts/train/text2natsql/generate_text2natsql_dataset.sh\n# Step4: fine-tune T5-3B (RESDSQL-3B+NatSQL)\nsh scripts/train/text2natsql/train_text2natsql_t5_3b.sh\n# Step4: (or) fine-tune T5-Large (RESDSQL-Large+NatSQL)\nsh scripts/train/text2natsql/train_text2natsql_t5_large.sh\n# Step4: (or) fine-tune T5-Base (RESDSQL-Base+NatSQL)\nsh scripts/train/text2natsql/train_text2natsql_t5_base.sh\n```\n\n**RESDSQL-{Base, Large, 3B}**\n```sh\n# Step1: preprocess dataset\nsh scripts/train/text2sql/preprocess.sh\n# Step2: train cross-encoder\nsh scripts/train/text2sql/train_text2sql_schema_item_classifier.sh\n# Step3: prepare text-to-sql training and development set for T5\nsh scripts/train/text2sql/generate_text2sql_dataset.sh\n# Step4: fine-tune T5-3B (RESDSQL-3B)\nsh scripts/train/text2sql/train_text2sql_t5_3b.sh\n# Step4: (or) fine-tune T5-Large (RESDSQL-Large)\nsh scripts/train/text2sql/train_text2sql_t5_large.sh\n# Step4: (or) fine-tune T5-Base (RESDSQL-Base)\nsh scripts/train/text2sql/train_text2sql_t5_base.sh\n```\n\n**During training, the cross-encoder (i.e., the first stage) always keeps the best checkpoint, but T5 (i.e., the second stage) keeps all the intermediate checkpoints, because different test sets may achieve the best Text-to-SQL performance on different checkpoints**. Therefore, given a test set, we need to evaluate all the intermediate checkpoints and compare their performance to find the best checkpoint. The evaluation results of checkpoints are saved in `eval_results`.\n\nOur paper also report the performence of RESDSQL-3B+NatSQL (the most powerful version of RESDSQL) on Spider-DK, Spider-Syn, and Spider-Realistic. To obtain results on these datasets, we provide evaluation scripts in `scripts/evaluate_robustness`. Here is an example for Spider-DK:\n```sh\n# Step1: preprocess Spider-DK\nsh scripts/evaluate_robustness/preprocess_spider_dk.sh\n# Step2: Run evaluation on Spider-DK\nsh scripts/evaluate_robustness/evaluate_on_spider_dk.sh\n```\n\n## Training on CSpider\nWe additionally provide scripts in `scripts/train/cspider_text2natsql` and `scripts/train/cspider_text2sql` to train RESDSQL on CSpider's training set and evaluate on CSpider's dev set.\n\n**RESDSQL-{Base, Large, 3B}+NatSQL (CSpider version)**\n```sh\n# Step1: preprocess CSpider\nsh scripts/train/cspider_text2natsql/preprocess.sh\n# Step2: train cross-encoder\nsh scripts/train/cspider_text2natsql/train_text2natsql_schema_item_classifier.sh\n# Step3: prepare text-to-natsql training and development set for mT5\nsh scripts/train/cspider_text2natsql/generate_text2natsql_dataset.sh\n# Step4: fine-tune mT5-XL (RESDSQL-3B+NatSQL)\nsh scripts/train/cspider_text2natsql/train_text2natsql_mt5_xl.sh\n# Step4: (or) fine-tune mT5-Large (RESDSQL-Large+NatSQL)\nsh scripts/train/cspider_text2natsql/train_text2natsql_mt5_large.sh\n# Step4: (or) fine-tune mT5-Base (RESDSQL-Base+NatSQL)\nsh scripts/train/cspider_text2natsql/train_text2natsql_mt5_base.sh\n```\n\nIn order to train the NatSQL version on CSpider, we manually aligned and modified annotations of NatSQL. The aligned files are also released, see `NatSQL/NatSQLv1_6/train_cspider-natsql.json` and `NatSQL/NatSQLv1_6/dev_cspider-natsql.json`.\n\n**RESDSQL-{Base, Large, 3B} (CSpider version)**\n```sh\n# Step1: preprocess CSpider\nsh scripts/train/cspider_text2sql/preprocess.sh\n# Step2: train cross-encoder\nsh scripts/train/cspider_text2sql/train_text2sql_schema_item_classifier.sh\n# Step3: prepare text-to-sql training and development set for mT5\nsh scripts/train/cspider_text2sql/generate_text2sql_dataset.sh\n# Step4: fine-tune mT5-XL (RESDSQL-3B)\nsh scripts/train/cspider_text2sql/train_text2sql_mt5_xl.sh\n# Step4: (or) fine-tune mT5-Large (RESDSQL-Large)\nsh scripts/train/cspider_text2sql/train_text2sql_mt5_large.sh\n# Step4: (or) fine-tune mT5-Base (RESDSQL-Base)\nsh scripts/train/cspider_text2sql/train_text2sql_mt5_base.sh\n```\n\n\n## Acknowledgements\nWe would thanks to Hongjin Su and Tao Yu for their help in evaluating our method on Spider's test set. We would also thanks to PICARD ([paper](https://arxiv.org/abs/2109.05093), [code](https://github.com/ServiceNow/picard)), NatSQL ([paper](https://arxiv.org/abs/2109.05153), [code](https://github.com/ygan/NatSQL)), Spider ([paper](https://arxiv.org/abs/1809.08887), [dataset](https://yale-lily.github.io/spider)), Spider-DK ([paper](https://arxiv.org/abs/2109.05157), [dataset](https://github.com/ygan/Spider-DK)), Spider-Syn ([paper](https://arxiv.org/abs/2106.01065), [dataset](https://github.com/ygan/Spider-Syn)), Spider-Realistic ([paper](https://arxiv.org/abs/2010.12773), [dataset](https://doi.org/10.5281/zenodo.5205322)), Dr.Spider ([paper](https://openreview.net/pdf?id=Wc5bmZZU9cy), [dataset](https://github.com/awslabs/diagnostic-robustness-text-to-sql)), and CSpider ([paper](https://arxiv.org/abs/1909.13293), [dataset](https://taolusi.github.io/CSpider-explorer/)) for their interesting work and open-sourced code and dataset.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRUCKBReasoning%2FRESDSQL","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRUCKBReasoning%2FRESDSQL","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRUCKBReasoning%2FRESDSQL/lists"}