{"id":20064482,"url":"https://github.com/xlang-ai/icl-selective-annotation","last_synced_at":"2025-10-13T06:07:10.485Z","repository":{"id":58758293,"uuid":"532432979","full_name":"xlang-ai/icl-selective-annotation","owner":"xlang-ai","description":"[ICLR 2023] Code for our paper \"Selective Annotation Makes Language Models Better Few-Shot Learners\"","archived":false,"fork":false,"pushed_at":"2023-07-15T17:30:14.000Z","size":34033,"stargazers_count":97,"open_issues_count":1,"forks_count":14,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-28T00:48:36.681Z","etag":null,"topics":["active-learning","in-context-learning","language-model","natural-language-processing","nlp","sample-selection"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xlang-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-04T03:57:23.000Z","updated_at":"2024-04-28T00:48:38.976Z","dependencies_parsed_at":"2024-04-28T00:58:46.069Z","dependency_job_id":null,"html_url":"https://github.com/xlang-ai/icl-selective-annotation","commit_stats":null,"previous_names":["xlang-ai/icl-selective-annotation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2Ficl-selective-annotation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2Ficl-selective-annotation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2Ficl-selective-annotation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2Ficl-selective-annotation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xlang-ai","download_url":"https://codeload.github.com/xlang-ai/icl-selective-annotation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224458570,"owners_count":17314685,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["active-learning","in-context-learning","language-model","natural-language-processing","nlp","sample-selection"],"created_at":"2024-11-13T13:46:27.473Z","updated_at":"2025-10-13T06:07:10.394Z","avatar_url":"https://github.com/xlang-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Selective Annotation Makes Language Models Better Few-Shot Learners\n\nCode for paper [Selective Annotation Makes Language Models Better Few-Shot Learners](http://arxiv.org/abs/2209.01975)\n\n\u003cimg src=\"pics/pipeline.png\" align=\"middle\" width=\"75%\"\u003e\n\nMany recent approaches to natural language tasks are built on the remarkable\nabilities of large language models. Large language models can perform in-context\nlearning, where they learn a new task from a few task demonstrations, without\nany parameter updates. This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. Departing from\nrecent in-context learning methods, we formulate an annotation-efficient, two-step\nframework: **selective annotation** that chooses a pool of examples to annotate from\nunlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time. Based on this framework, we propose an\nunsupervised, graph-based selective annotation method, **vote-k**, to select diverse,\nrepresentative examples to annotate. Extensive experiments on 10 datasets (covering classification, commonsense reasoning, dialogue, and text/code generation)\ndemonstrate that our selective annotation method improves the task performance\nby a large margin. On average, vote-k achieves a **12.9%/11.4% relative gain** under\nan annotation budget of 18/100, as compared to randomly selecting examples to\nannotate. Compared to state-of-the-art supervised finetuning approaches, it yields\nsimilar performance with **10-100× less annotation cost** across 10 tasks. We further\nanalyze the effectiveness of our framework in various scenarios: language models\nwith varying sizes, alternative selective annotation methods, and cases where there\nis a test data domain shift. We hope that our studies will serve as a basis for data\nannotations as large language models are increasingly applied to new tasks\n\n## Cloning this repo\nRun the following command to clone this repo\n```\ngit clone https://github.com/HKUNLP/icl-selective-annotation\n```\n\n## Dependencies\nTo establish the environment, run this code in the shell:\n```\nconda env create -f selective_annotation.yml\nconda activate selective_annotation\ncd transformers\npip install -e .\n```\nThat will create the environment selective_annotation we used.\n\n## Usage\n\n### Environment setup\n\nActivate the environment by running\n```\nconda activate selective_annotation\n```\n\n### End-to-end pipeline: selection, inference, evaluation\nGPT-J as the in-context learning model, DBpedia as the task, and vote-k as the selective annotation method (1 GPU, 40GB memory)\n```\npython main.py --task_name dbpedia_14 --selective_annotation_method votek --model_cache_dir models --data_cache_dir datasets --output_dir outputs\n```\n\n## Citation\nIf you find our work helpful, please cite us\n```\n@article{Selective_Annotation,\n      title={Selective Annotation Makes Language Models Better Few-Shot Learners}, \n      author={Hongjin Su and Jungo Kasai and Chen Henry Wu and Weijia Shi and Tianlu Wang and Jiayi Xin and Rui Zhang and Mari Ostendorf and Luke Zettlemoyer and Noah A. Smith and Tao Yu},\n      journal={ArXiv},\n      year={2022},\n}\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlang-ai%2Ficl-selective-annotation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxlang-ai%2Ficl-selective-annotation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlang-ai%2Ficl-selective-annotation/lists"}