{"id":29799619,"url":"https://github.com/kyle1668/llm-tta","last_synced_at":"2025-07-28T08:10:24.936Z","repository":{"id":155763098,"uuid":"624690305","full_name":"Kyle1668/LLM-TTA","owner":"Kyle1668","description":"Code for the paper: Improving Black-box Robustness with In-Context Rewriting","archived":false,"fork":false,"pushed_at":"2024-05-23T17:07:23.000Z","size":25877,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-02T20:19:09.539Z","etag":null,"topics":["augmentation","nlp","research-paper","test-time","tta"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2402.08225","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kyle1668.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-07T03:07:18.000Z","updated_at":"2025-01-30T02:35:55.000Z","dependencies_parsed_at":"2023-11-28T21:28:37.387Z","dependency_job_id":"ac1c7451-e444-4b7a-ac14-82b763e69183","html_url":"https://github.com/Kyle1668/LLM-TTA","commit_stats":null,"previous_names":["kyle1668/in-context-domain-transfer-improves-out-of-domain-robustness","kyle1668/llm-tta"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Kyle1668/LLM-TTA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyle1668%2FLLM-TTA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyle1668%2FLLM-TTA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyle1668%2FLLM-TTA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyle1668%2FLLM-TTA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kyle1668","download_url":"https://codeload.github.com/Kyle1668/LLM-TTA/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyle1668%2FLLM-TTA/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267482004,"owners_count":24094508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["augmentation","nlp","research-paper","test-time","tta"],"created_at":"2025-07-28T08:10:14.496Z","updated_at":"2025-07-28T08:10:24.925Z","avatar_url":"https://github.com/Kyle1668.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Improving Black-box Robustness with In-Context Rewriting\n\nThis repo contains the code to replicate the results in [our paper](https://arxiv.org/abs/2402.08225) and extend our study to additional models and datasets. We use Make to abstract most of the commands to replicate our results. Don't hestiate to reach out to Kyle O'Brien with questions.\n\n## Installation\n\nWe've tested our experiments using Python 3.10. Conda is recommended. \n\n```bash\nconda create -n llm-tta python=3.10\nconda activate llm-tta\n```\n\nYou can install the pip packages and datasets using the following command.\n\n```bash\nmake install_depends\n```\n\nGenerating augmentation is slow. We use a caching mechanism to leverage previously generated augmentations for each test input to speed up experiments. You can fill the cache using the following command. The cache wil otherwise automatically begin populating otherwise. \n\n```bash\nmake download_rewrites_cache\n```\n\nAnd clear the cache with:\n```bash\nmake clear_rewrites_cache\n```\n\n## Run Experiments\n\nYou can iterate through the main results using the following commands.\n\n```bash\nmake main_results_async # split across all GPUs (Recommended)\nmake main_results_sync # Single-GPU setup\n```\n\n`Makefile` contains multiple other preset experiment configurations and shows examples of various command line arguments.\n\n# Acknowledgments\n\nWe are grateful to EleutherAI for permitting access to their compute resources for initial experiments. The welcome and open research community on the EleutherAI Discord was especially helpful for the literature review, debugging PyTorch issues, and information necessary to conduct the parameter count ablation experiment (Appendix A.2). In particular, we would like to thank Stella Biderman, Nora Belrose, and Hailey Schoelkopf.\n\nLydia O’Brien provided copy editing and feedback on figure design and engaged in extensive discussions that\nshaped the direction of this project.\n\nM. Ghassemi’s work is supported in part by Quanta Computing and the Gordon and Betty Moore Foundation. The research of J. Mendez-Mendez is funded by an MIT-IBM Distinguished Postdoctoral Fellowship.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyle1668%2Fllm-tta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyle1668%2Fllm-tta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyle1668%2Fllm-tta/lists"}