{"id":20473027,"url":"https://github.com/jianguoz/few-shot-intent-detection","last_synced_at":"2025-04-13T11:28:25.498Z","repository":{"id":37368568,"uuid":"373651784","full_name":"jianguoz/Few-Shot-Intent-Detection","owner":"jianguoz","description":"Few-Shot-Intent-Detection includes popular challenging intent detection datasets with/without OOS queries and state-of-the-art baselines and results.","archived":false,"fork":false,"pushed_at":"2023-07-19T05:22:52.000Z","size":1413,"stargazers_count":138,"open_issues_count":0,"forks_count":26,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-27T02:39:22.415Z","etag":null,"topics":["datasets","few-shot","intent-classification","intent-detection","libary"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jianguoz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-03T22:00:23.000Z","updated_at":"2025-03-26T07:32:20.000Z","dependencies_parsed_at":"2024-11-15T14:23:42.541Z","dependency_job_id":"a504d055-0f53-4acb-86f8-de98a967d242","html_url":"https://github.com/jianguoz/Few-Shot-Intent-Detection","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianguoz%2FFew-Shot-Intent-Detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianguoz%2FFew-Shot-Intent-Detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianguoz%2FFew-Shot-Intent-Detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianguoz%2FFew-Shot-Intent-Detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jianguoz","download_url":"https://codeload.github.com/jianguoz/Few-Shot-Intent-Detection/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248705150,"owners_count":21148488,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasets","few-shot","intent-classification","intent-detection","libary"],"created_at":"2024-11-15T14:23:08.280Z","updated_at":"2025-04-13T11:28:25.459Z","avatar_url":"https://github.com/jianguoz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Few-Shot-Intent-Detection\n\n## :bangbang: ❤️ ‼️ **07/18/2023: Check our latest updates on [DialogStudio](https://github.com/salesforce/DialogStudio)\n\n[DialogStudio](https://github.com/salesforce/DialogStudio) is a meticulously curated collection of dialogue datasets. These datasets are unified under a consistent format while retaining their original information. We incorporate domain-aware prompts and identify dataset licenses, making DialogStudio an exceptionally rich and diverse resource for dialogue research and model training.**\n\n\nFew-Shot-Intent-Detection is a repository designed for few-shot intent detection with/without Out-of-Scope (OOS) intents. It includes popular challenging intent detection datasets and baselines. For more details of the new released OOS datasets, please check our [paper](https://arxiv.org/abs/2106.04564).\n\n\n\n## Intent detection datasets\n\nWe process data based on previous published resources, all the data are in the same format as [DNNC](https://github.com/salesforce/DNNC-few-shot-intent). \n\n\n| Dataset      \t| Description  | #Train | #Valid | #Test \t|  Processed Data Link| \n|--------------\t|------\t|------\t|------\t|---------------\t|------\t|\n| [BANKING77](https://arxiv.org/abs/2003.04807)      \t| one banking domain with 77 intents  |8622|1540| 3080  \t|  [Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/BANKING77)                  \t|\n| [CLINC150](https://www.aclweb.org/anthology/D19-1131/)        | 10 domains and 150 intents |15000| 3000\t| 4500 \t| [Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/CLINC150)|                                              \t| Link\t|\n| [HWU64](https://arxiv.org/abs/1903.05566)        | personal assistant with 64 intents and several domains                                                 |8954| 1076\t| 1076 \t|  [Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/HWU64)\t|\n| [SNIPS](https://arxiv.org/pdf/1805.10190.pdf)        |snips voice platform with 7 intents   |13084| 700\t| 700 \t|  [Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/SNIPS)\t|\n| [ATIS](https://ieeexplore.ieee.org/document/5700816)        |airline travel information system   |4478| 500\t| 893 \t|  [Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/SNIPS)\t|\n\n\n\n## Intent detection datasets with OOS queries\n\n\nWhat is OOS queires:\n\n`OOD-OOS`: i.e., out-of-domain OOS. General out-of-scope queries which are not supported by the dialog systems, also called out-of-domain OOS. For instance, requesting an online NBA/TV show service in a banking system.\n\n`ID-OOS`: i.e., in-domain OOS. Out-of-scope queries which are more related to the in-scope intents, which makes the intent detection task more challenging. For instance, requesting a banking service that is not supported by the banking system.\n\n| Dataset      \t| Description  | #Train | #Valid | #Test \t|#OOD-OOS-Train |#OOD-OOS-Valid|#OOD-OOS-Test| #ID-OOS-Train |#ID-OOS-Valid|#ID-OOS-Test| Processed Data Link| \n|--------------\t|------\t|------\t|------\t|---------------\t|------\t|------\t|------\t|------\t|------\t|------|------\t|\n| [CLINC150](https://www.aclweb.org/anthology/D19-1131/)        | A dataset with general OOS-OOS queries |15000| 3000\t| 4500  |\t100| 100|1000| -|-|-|[Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/CLINC150)|\n| [CLINC-Single-Domain-OOS](https://arxiv.org/abs/2106.04564)        | Two domains with both general OOS-OOS queries and ID-OOS queries |500| 500\t| 500  |-\t| 200|1000| -|400|350|[Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/CLINC-Single-Domain-OOS)|                                             \n| [BANKING77-OOS](https://arxiv.org/abs/2106.04564)        | One banking domain with both general OOS-OOS queries and ID-OOS queries |5905| 1506\t| 2000  |-\t| 200|1000| 2062|530|1080|[Link](https://github.com/jianguoz/Few-Shot-Intent-Detection/tree/main/Datasets/BANKING77-OOS)|      \n\n\nData structure:\n\n```\nDatasets/\n├── BANKING77\n│   ├── train\n│   ├── train_10\n│   ├── train_5\n│   ├── valid\n│   └── test\n├── CLINC150\n│   ├── train\n│   ├── train_10\n│   ├── train_5\n│   ├── valid\n│   ├── test\n│   ├── oos\n│       ├──train\n│       ├──valid\n│       └──test\n├── HWU64\n│   ├── train\n│   ├── train_10\n│   ├── train_5\n│   ├── valid\n│   └── test\n├── SNIPS\n│   ├── train\n│   ├── valid\n│   └── test\n├── ATIS\n│   ├── train\n│   ├── valid\n│   └── test\n├── BANKING77-OOS\n│   ├── train\n│   ├── valid\n│   ├── test\n│   ├── id-oos\n│   │   ├──train\n│   │   ├──valid\n│   │   └──test\n│   ├── ood-oos\n│       ├──valid\n│       └──test\n├── CLINC-Single-Domain-OOS\n│   ├── banking\n│   │   ├── train\n│   │   ├── valid\n│   │   ├── test\n│   │   ├── id-oos\n│   │   │   ├──valid\n│   │   │   └──test\n│   │   ├── ood-oos\n│   │       ├──valid\n│   │       └──test\n│   ├── credit_cards\n│   │   ├── train\n│   │   ├── valid\n│   │   ├── test\n│   │   ├── id-oos\n│   │   │   ├──valid\n│   │   │   └──test\n│   │   ├── ood-oos\n│   │       ├──valid\n└── └──     └──test\n```\n\nBriefly describe the [BANKING77-OOS](https://arxiv.org/abs/2106.04564) dataset. \n\n*  A dataset with a single banking domain, includes both general Out-of-Scope (OOD-OOS) queries and In-Domain but Out-of-Scope (ID-OOS) queries, where ID-OOS queries are semantically similar intents/queries with in-scope intents.  BANKING77 originally includes 77 intents. BANKING77-OOS includes 50 in-scope intents in this dataset, and the ID-OOS queries are built up based on 27 held-out semantically similar in-scope intents.\n\nBriefly describe the [CLINC-Single-Domain-OOS](https://arxiv.org/abs/2106.04564) dataset. \n\n*  A dataset with two separate domains, i.e., the  \"Banking''  domain and the \"Credit cards''  domain with both general Out-of-Scope (OOD-OOS) queries and In-Domain but Out-of-Scope (ID-OOS) queries, where ID-OOS queries are semantically similar intents/queries with in-scope intents. Each domain in CLINC150 originally includes 15 intents. Each domain in the new dataset includes ten in-scope intents in this dataset, and the ID-OOS queries are built up based on five held-out semantically similar in-scope intents.\n\nBoth datasets can be used to conduct intent detection with and without OOD-OOS and ID-OOS queries\n\n\nYou can easily load the processed data:\n```python\nclass IntentExample:\n    def __init__(self, text, label, do_lower_case):\n        self.original_text = text\n        self.text = text\n        self.label = label\n\n        if do_lower_case:\n            self.text = self.text.lower()\n        \ndef load_intent_examples(file_path, do_lower_case=True):\n    examples = []\n\n    with open('{}/seq.in'.format(file_path), 'r', encoding=\"utf-8\") as f_text, open('{}/label'.format(file_path), 'r', encoding=\"utf-8\") as f_label:\n        for text, label in zip(f_text, f_label):\n            e = IntentExample(text.strip(), label.strip(), do_lower_case)\n            examples.append(e)\n\n    return examples\n```\n\nMore details can check [code for load data and do random sampling for few-shot learning](https://github.com/salesforce/DNNC-few-shot-intent/blob/master/train_classifier.py#L127).\n\n## State-of-the art models and baselines\n\n\n**[DNNC](https://www.aclweb.org/anthology/2020.emnlp-main.411/)**\n\nDownload pre-trained RoBERTa NLI checkpoint: \n```bash\nwget https://storage.googleapis.com/sfr-dnnc-few-shot-intent/roberta_nli.zip\n```\nAccess to public code: [Link](https://github.com/salesforce/DNNC-few-shot-intent)\n\n**[CONVERT](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/)**\n\nDownload pre-trained checkpoint: \n```bash\nwget https://github.com/connorbrinton/polyai-models/releases/download/v1.0/model.tar.gz\n```\n\nAccess to public code:\n```bash\nwget https://github.com/connorbrinton/polyai-models/archive/refs/tags/v1.0.zip\n```\n\n\n**[CONVBERT](https://arxiv.org/abs/2009.13570)** \n\nDownload pre-trained checkpoints: \n\nStep-1: install [AWS CL2](https://aws.amazon.com/cli/): e.g., install [MacOS PKG](https://awscli.amazonaws.com/AWSCLIV2.pkg)\n\nStep-2: \n```bash\naws s3 cp s3://dialoglue/ --no-sign-request `Your_folder_name` --recursive\n```\nThen the checkpoints are downloaded into  `Your_folder_name`\n\n## Few-shot intent detection baselines/leaderboard:\n\n**5-shot learning**\n\n| Model      \t| BANKING77  | CLICN150 | HWU64 | \n|--------------\t|------\t|------\t|------\t|\n|[RoBERTa+Classifier](https://www.aclweb.org/anthology/2020.emnlp-main.411/) (EMNLP 2020) | 74.04 | 87.99 | 75.56 |\n|[USE](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/) (ACL 2020 NLP4ConvAI)| 76.29 | 87.82 | 77.79 |\n|[CONVERT](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/) (ACL 2020 NLP4ConvAI)| 75.32 | 89.22 | 76.95|\n|[USE+CONVERT](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/) (ACL 2020 NLP4ConvAI)      | 77.75 | 90.49 | 80.01 | \n|[CONVBERT+MLM+Example+Observers](https://arxiv.org/abs/2010.08684)  (NAACL 2021)     | - | - | - |\n|[DNNC](https://www.aclweb.org/anthology/2020.emnlp-main.411/) (EMNLP 2020)              | 80.40 | 91.02 | 80.46 | \n|[CPFT](https://arxiv.org/pdf/2109.06349.pdf) (EMNLP 2021) |80.86| 92.34 | 82.03|\n|[ICDA](https://arxiv.org/abs/2302.05096) (EACL 2023) |84.01| 92.62 | 82.45|\n\n\n**10-shot learning**\n\n| Model      \t| BANKING77  | CLICN150 | HWU64 | \n|--------------\t|------\t|------\t|------\t|\n|[RoBERTa+Classifier](https://www.aclweb.org/anthology/2020.emnlp-main.411/) (EMNLP 2020) | 84.27 | 91.55 | 82.90 |\n|[USE](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/) (ACL 2020 NLP4ConvAI)| 84.23 | 90.85 | 83.75 |\n|[CONVERT](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/)(ACL 2020 NLP4ConvAI) | 83.32 | 92.62 | 82.65|\n|[USE+CONVERT](https://www.aclweb.org/anthology/2020.nlp4convai-1.5/) (ACL 2020 NLP4ConvAI)       | 85.19 | 93.26 | 85.83 | \n|[CONVBERT](https://arxiv.org/abs/2009.13570) (ArXiv 2020)| 83.63 | 92.10 | 83.77 |\n|[CONVBERT+MLM](https://arxiv.org/abs/2009.13570)  (ArXiv 2020)     | 83.99 | 92.75 | 84.52 |\n|[CONVBERT+MLM+Example+Observers](https://arxiv.org/abs/2010.08684) (NAACL 2021) | 85.95 | 93.97 | 86.28 |\n|[DNNC](https://www.aclweb.org/anthology/2020.emnlp-main.411/) (EMNLP 2020)              | 86.71 | 93.76 | 84.72 |\n|[CPFT](https://arxiv.org/pdf/2109.06349.pdf) (EMNLP 2021) |87.20| 94.18 | 87.13|\n|[ICDA](https://arxiv.org/abs/2302.05096) (EACL 2023) |89.79| 94.84 | 87.41|\n\n`Note:` the 5-shot learning results of RoBERTa+Classifier, DNNC and CPFT, and the 10-shot learning results of all the models are reported by the paper authors. \n\n## Citation\n\nPlease cite our paper if you use above resources in your work:\n\n```bibtex\n@article{zhang2020discriminative,\n  title={Discriminative nearest neighbor few-shot intent detection by transferring natural language inference},\n  author={Zhang, Jian-Guo and Hashimoto, Kazuma and Liu, Wenhao and Wu, Chien-Sheng and Wan, Yao and Yu, Philip S and Socher, Richard and Xiong, Caiming},\n  journal={EMNLP},\n  pages={5064--5082},\n  year={2020}\n}\n@article{zhang2021few,\n  title={Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning},\n  author={Zhang, Jianguo and Bui, Trung and Yoon, Seunghyun and Chen, Xiang and Liu, Zhiwei and Xia, Congying and Tran, Quan Hung and Chang, Walter and Yu, Philip},\n  journal={EMNLP},\n  year={2021}\n}\n@article{zhang2022pretrained,\n  title={Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection},\n  author={Zhang, Jian-Guo and Hashimoto, Kazuma and Wan, Yao and Liu, Zhiwei and Liu, Ye and Xiong, Caiming and Yu, Philip S},\n  journal={The 4th Workshop on NLP for Conversational AI, ACL 2022},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjianguoz%2Ffew-shot-intent-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjianguoz%2Ffew-shot-intent-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjianguoz%2Ffew-shot-intent-detection/lists"}