{"id":28795629,"url":"https://github.com/ukplab/on-emergence","last_synced_at":"2025-06-25T09:09:00.527Z","repository":{"id":192063894,"uuid":"682539640","full_name":"UKPLab/on-emergence","owner":"UKPLab","description":"Codes and files for the paper Are Emergent Abilities in Large Language Models just In-Context Learning","archived":false,"fork":false,"pushed_at":"2025-01-09T09:51:38.000Z","size":271,"stargazers_count":33,"open_issues_count":1,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-18T03:09:40.973Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2309.01809","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-sa-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UKPLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-24T11:46:35.000Z","updated_at":"2025-01-09T23:20:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"1f9d33ae-23b7-4405-b86b-d6a449d391a7","html_url":"https://github.com/UKPLab/on-emergence","commit_stats":null,"previous_names":["ukplab/on-emergence"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/UKPLab/on-emergence","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UKPLab%2Fon-emergence","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UKPLab%2Fon-emergence/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UKPLab%2Fon-emergence/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UKPLab%2Fon-emergence/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UKPLab","download_url":"https://codeload.github.com/UKPLab/on-emergence/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UKPLab%2Fon-emergence/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261841974,"owners_count":23217918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-18T03:09:40.288Z","updated_at":"2025-06-25T09:09:00.506Z","avatar_url":"https://github.com/UKPLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Are Emergent Abilities in Large Language Models just In-Context Learning?\n\n\u003e **Abstract:** Large language models have exhibited *emergent* abilities, demonstrating exceptional performance across diverse tasks for which they were not explicitly trained, including those that require complex reasoning abilities. The emergence of such abilities carries profound implications for the future direction of research in NLP, especially as the deployment of such models becomes more prevalent. However, one key challenge is that the evaluation of these abilities is often confounded by competencies that arise in models through alternative prompting techniques, such as in-context learning and instruction following, which also emerge as the models are scaled up. In this study, we provide the first comprehensive examination of these emergent abilities while accounting for various potentially biasing factors that can influence the evaluation of models. We conduct rigorous tests on a set of 18 models, encompassing a parameter range from 60 million to 175 billion parameters, across a comprehensive set of 22 tasks. Through an extensive series of over 1,000 experiments, we provide compelling evidence that emergent abilities can primarily be ascribed to in-context learning. We find no evidence for the emergence of reasoning abilities, thus providing valuable insights into the underlying mechanisms driving the observed abilities and thus alleviating safety concerns regarding their use.\n\nContact persons: Sheng Lu, Irina Bigoulaeva, Harish Tayyar Madabushi\n\nhttps://www.ukp.tu-darmstadt.de/\n\nhttps://www.tu-darmstadt.de/\n\n## Evaluation scores\nSee [evaluation_scores](https://github.com/UKPLab/on-emergence/blob/main/evaluation_scores.csv) for the evaluation scores of our experiments.\n\n## Output files\nAll of the output files associated to our experiments can be found [here](https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/3931). We name each file in a *date_time* fashion (e.g., ```results-20230524_123238.json```), which is also the ```run_id``` of an experiment that can be found in [evaluation_scores](https://github.com/UKPLab/on-emergence/blob/main/evaluation_scores.csv).\n\n## Citation\nPlease use the following citation:\n\n```\n@inproceedings{lu-etal-2024-emergent,\n    title = \"Are Emergent Abilities in Large Language Models just In-Context Learning?\",\n    author = \"Lu, Sheng  and\n      Bigoulaeva, Irina  and\n      Sachdeva, Rachneet  and\n      Tayyar Madabushi, Harish  and\n      Gurevych, Iryna\",\n    editor = \"Ku, Lun-Wei  and\n      Martins, Andre  and\n      Srikumar, Vivek\",\n    booktitle = \"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\",\n    month = aug,\n    year = \"2024\",\n    address = \"Bangkok, Thailand\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.acl-long.279/\",\n    doi = \"10.18653/v1/2024.acl-long.279\",\n    pages = \"5098--5139\",\n    abstract = \"Large language models, comprising billions of parameters and pre-trained on extensive web-scale corpora, have been claimed to acquire certain capabilities without having been specifically trained on them. These capabilities, referred to as {\\textquotedblleft}emergent abilities,{\\textquotedblright} have been a driving force in discussions regarding the potentials and risks of language models. A key challenge in evaluating emergent abilities is that they are confounded by model competencies that arise through alternative prompting techniques, including in-context learning, which is the ability of models to complete a task based on a few examples. We present a novel theory that explains emergent abilities, taking into account their potential confounding factors, and rigorously substantiate this theory through over 1000 experiments. Our findings suggest that purported emergent abilities are not truly emergent, but result from a combination of in-context learning, model memory, and linguistic knowledge. Our work is a foundational step in explaining language model performance, providing a template for their efficient use and clarifying the paradox of their ability to excel in some instances while faltering in others. Thus, we demonstrate that their capabilities should not be overestimated.\"\n}\n```\n\n## License\n\nThis work is licensed under a\n[Creative Commons Attribution-ShareAlike 4.0 International License][cc-by-sa].\n\n[![CC BY-SA 4.0][cc-by-sa-image]][cc-by-sa]\n\n[cc-by-sa]: http://creativecommons.org/licenses/by-sa/4.0/\n[cc-by-sa-image]: https://licensebuttons.net/l/by-sa/4.0/88x31.png\n[cc-by-sa-shield]: https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fukplab%2Fon-emergence","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fukplab%2Fon-emergence","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fukplab%2Fon-emergence/lists"}