{"id":19932220,"url":"https://github.com/amazon-science/contraclm","last_synced_at":"2025-06-28T17:34:36.887Z","repository":{"id":179766468,"uuid":"663759286","full_name":"amazon-science/ContraCLM","owner":"amazon-science","description":"[ACL 2023] Code for ContraCLM: Contrastive Learning For Causal Language Model","archived":false,"fork":false,"pushed_at":"2023-12-20T21:00:39.000Z","size":131,"stargazers_count":33,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-03T11:35:47.766Z","etag":null,"topics":["contrastive-learning","generative-ai","gpt-2","llm","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-07-08T03:27:51.000Z","updated_at":"2024-12-18T06:04:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"112a46d9-769d-4c38-80a4-bc63cbbab75f","html_url":"https://github.com/amazon-science/ContraCLM","commit_stats":null,"previous_names":["amazon-science/contraclm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amazon-science/ContraCLM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FContraCLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FContraCLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FContraCLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FContraCLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/ContraCLM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2FContraCLM/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262469596,"owners_count":23316345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contrastive-learning","generative-ai","gpt-2","llm","nlp"],"created_at":"2024-11-12T23:09:24.930Z","updated_at":"2025-06-28T17:34:36.882Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ContraCLM: Contrastive Learning for Causal Language Model\n\nThis repository contains code for the ACL 2023 paper, [ContraCLM: Contrastive Learning for Causal Language Model](https://arxiv.org/abs/2210.01185). \n\nWork done by: Nihal Jain*, Dejiao Zhang*, Wasi Uddin Ahmad*, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Xiaofei Ma, Bing Xiang. (* \u003cem\u003eindicates equal contribution\u003c/em\u003e). \n\n## Updates\n\n* [07-08-2023] Initial release of the code.\n\n## Quick Links\n\n* [Overview](#overview)\n* [Setup](#setup)\n   * [Environment](#environment)\n   * [Datasets](#datasets)\n* [Pretraining](#pretraining)\n   * [GPT2](#pretain-gpt2-on-nl-data)\n   * [CodeGen](#pretrain-codegen-350m-mono-on-pl-data)\n* [Evaluation](#evaluation)\n* [Citation](#citation)\n\n## Overview\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"static/llms_contraclm.png\" width=500\u003e\u003c/img\u003e\n\u003c/p\u003e\n\nWe present ContraCLM, a novel contrastive learning framework which operates at both the token-level and sequence-level. ContraCLM enhances  the discrimination of representations from a decoder-only language model and bridges the gap with encoder-only models, making causal language models better suited for tasks beyond language generation. We encourage you to check out our [paper](https://arxiv.org/abs/2210.01185) for more details.\n\n## Setup\n\nThe setup involves installing the necessary dependencies in an environment and placing the datasets in the requisite directory.\n\n### Environment\n\nRun these commands to create a new conda environment and install the required packages for this repository.\n\n```bash\n# create a new conda environment with python \u003e= 3.8\nconda create -n contraclm python=3.8.12\n\n# install dependencies within the environment\nconda activate contraclm\npip install -r requirements.txt\n```\n\n### Datasets \u0026 Preprocessing\n\nSee \u003ca href=\"preprocess/\"\u003ehere\u003c/a\u003e.\n\n## Pretraining\n\nIn this section, we show how to use this repository to pretrain (i) `GPT2` on Natural Language (NL) data, and (ii) `CodeGen-350M-Mono` on Programming Language (PL) data.\n\n### Common Instructions\n\n1. This section assumes that you have the train and validation data stored at `TRAIN_DIR` and `VALID_DIR` respectively, and are within an environment with all the above dependencies installed (see [Setup](#setup)).\n\n2. You can get an overview of all the flags associated with pretraining by running:\n```bash\npython pl_trainer.py --help\n```\n\n### Pretain `GPT2` on NL Data\n\n#### Usage\n\n```bash runscripts/run_wikitext.sh```\n   1. For quickly testing the code and debug, suggesting run the code with MLE loss only \n   by setting ```CL_Config=$(eval echo ${options[1]})``` within the script. \n   2. All other opotions involves CL loss at either token-level or sequence-level. \n\n### Pretrain `CodeGen-350M-Mono` on PL Data\n\n#### Usage\n\n1. Configure the variables at the top of `runscripts/run_code.sh`. There are lots of options but only the dropout options are explained here (others are self-explanatory):\n\n   * `dropout_p`: The dropout probability value used in `torch.nn.Dropout`\n \n   * `dropout_layers`: If \u003e 0, this will activate the last `dropout_layers` with probability `dropout_p`\n\n   * `functional_dropout`: If specified, will use a functional dropout layer on top of the token representations output from the CodeGen model\n\n 2. Set the variable `CL` according to desired model configuration. Make sure the paths to `TRAIN_DIR, VALID_DIR` are set as desired.\n\n 3. Run the command: `bash runscripts/run_code.sh`\n\n## Evaluation\n\nSee the relevant task-specific directories [here](evaluation/).\n\n## Citation\n\nIf you use our code in your research, please cite our work as:\n\n```\n@inproceedings{jain-etal-2023-contraclm,\n    title = \"{C}ontra{CLM}: Contrastive Learning For Causal Language Model\",\n    author = \"Jain, Nihal  and\n      Zhang, Dejiao  and\n      Ahmad, Wasi Uddin  and\n      Wang, Zijian  and\n      Nan, Feng  and\n      Li, Xiaopeng  and\n      Tan, Ming  and\n      Nallapati, Ramesh  and\n      Ray, Baishakhi  and\n      Bhatia, Parminder  and\n      Ma, Xiaofei  and\n      Xiang, Bing\",\n    booktitle = \"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\",\n    month = jul,\n    year = \"2023\",\n    address = \"Toronto, Canada\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2023.acl-long.355\",\n    pages = \"6436--6459\"\n}\n```\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fcontraclm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Fcontraclm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fcontraclm/lists"}