{"id":13654794,"url":"https://github.com/hendrycks/apps","last_synced_at":"2025-04-05T00:09:47.586Z","repository":{"id":41098400,"uuid":"361262700","full_name":"hendrycks/apps","owner":"hendrycks","description":"APPS: Automated Programming Progress Standard (NeurIPS 2021)","archived":false,"fork":false,"pushed_at":"2024-06-19T06:32:49.000Z","size":45,"stargazers_count":450,"open_issues_count":4,"forks_count":58,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-28T23:08:48.196Z","etag":null,"topics":["code-generation","program-synthesis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hendrycks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-24T20:43:37.000Z","updated_at":"2025-03-26T15:45:45.000Z","dependencies_parsed_at":"2024-08-02T03:11:53.674Z","dependency_job_id":null,"html_url":"https://github.com/hendrycks/apps","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Fapps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Fapps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Fapps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hendrycks%2Fapps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hendrycks","download_url":"https://codeload.github.com/hendrycks/apps/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266565,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","program-synthesis"],"created_at":"2024-08-02T03:00:47.293Z","updated_at":"2025-04-05T00:09:47.572Z","avatar_url":"https://github.com/hendrycks.png","language":"Python","readme":"# Measuring Coding Challenge Competence With APPS\nThis is the repository for [Measuring Coding Challenge Competence With APPS](https://arxiv.org/pdf/2105.09938) by\n[Dan Hendrycks\\*](https://danhendrycks.com/), [Steven Basart\\*](https://stevenbas.art), [Saurav Kadavath](http://www.sauravkadavath.com), Mantas Mazeika, [Akul Arora](https://github.com/akulaarora), Ethan Guo, [Collin Burns](http://collinpburns.com), Samir Puranik, [Horace He](http://horace.io), [Dawn Song](https://people.eecs.berkeley.edu/~dawnsong/), and [Jacob Steinhardt](https://www.stat.berkeley.edu/~jsteinhardt/).\n\nDownload the [**APPS dataset here**](https://people.eecs.berkeley.edu/~hendrycks/APPS.tar.gz). (~1.3GB)\n\nThis repository contains both training and evaluation code.\n\nFine-tuned GPT-2 1.5B and GPT-Neo 2.7B weights are available [here](https://drive.google.com/file/d/1XW1Od9L-5l9zXl1HUCyER5pS9zQTbIvU/view?usp=sharing).\n\nFor other benchmarks of enormous Transformers, see a dataset which tests ability in [competition math](https://github.com/hendrycks/math), a dataset which tests knowledge of [ethics](https://github.com/hendrycks/ethics), and [a dataset spanning 50+ academic subjects](https://github.com/hendrycks/test).\n\n## How to Use\n\nThe training instructions are specified in [train/README](train/README.md) and similarly the evaluation instructions are specified in [eval/README](eval/README.md).\n\n### Hugging Face\n\nThe dataset is also available in [Hugging Face datasets](https://huggingface.co/datasets/codeparrot/apps) under apps.\n\n## Citation\n\nIf you find this useful in your research, please consider citing\n\n    @article{hendrycksapps2021,\n      title={Measuring Coding Challenge Competence With APPS},\n      author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt},\n      journal={NeurIPS},\n      year={2021}\n    }\n","funding_links":[],"categories":["Python","Dataset and Benchmark","Anthropomorphic-Taxonomy"],"sub_categories":["Papers (This list is a bit outdated, need to update)","Typical Professional Quotient (PQ)-Professional Expertise evaluation benchmarks"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrycks%2Fapps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhendrycks%2Fapps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhendrycks%2Fapps/lists"}