{"id":26103664,"url":"https://github.com/aleph-alpha/benchpress-hackathon","last_synced_at":"2025-04-12T18:08:39.087Z","repository":{"id":264336070,"uuid":"891649140","full_name":"Aleph-Alpha/benchpress-hackathon","owner":"Aleph-Alpha","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-23T13:50:47.000Z","size":353,"stargazers_count":1,"open_issues_count":1,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-12T18:07:46.685Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-20T17:51:22.000Z","updated_at":"2024-11-23T15:28:49.000Z","dependencies_parsed_at":"2024-11-23T14:42:44.372Z","dependency_job_id":null,"html_url":"https://github.com/Aleph-Alpha/benchpress-hackathon","commit_stats":null,"previous_names":["aleph-alpha/benchpress-hackathon"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fbenchpress-hackathon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fbenchpress-hackathon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fbenchpress-hackathon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fbenchpress-hackathon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/benchpress-hackathon/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610340,"owners_count":21132921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-09T20:41:58.874Z","updated_at":"2025-04-12T18:08:39.057Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchpress Hackathon\n\nWelcome to the Benchpress Hackathon. The goal is to build an agent that excels in solving hard coding problems.\n\n## Installation\n\nTo install the dependencies Python 3.10 is required.\nInstall the dependencies in an isolated environment:\n\n```\npip install -r requirements.txt\n```\n\n## Usage\n\nThe challenge comes with a Jupyter notebook for your implementation and various utilities.\nWe provide a development set and a validation set you can use to develop your solution.\nThe development set is for testing your code and consists of 300 problems with a varying number of test cases.\nYou are free to use all data provided with a problem, a sample has the following structure:\n\n```python\n{\n    # Unique identifier for the problem in the APPS dataset.\n    \"problem_id\": 4424,\n    # The problem statement\n    \"question\": \"Given three integers ...\",\n    # The expected function name and the input/output examples\n    # representing test cases.\n    \"input_output\": {\n        \"fn_name\": \"expression_matter\",\n        \"inputs\": [ ... ],\n        \"outputs\": [ ... ]\n    },\n    \"url\": \"https://www.codewars.com/kata/5ae62fcf252e66d44d00008e\",\n    \"difficulty\": \"introductory\",\n    # The starter code for the problem.\n    \"starter_code\": \"def expression_matter(a, b, c):\\n\\t\"\n}\n```\n\nThe validation set is consists of 100 problems, and includes an additional key `test_cases` which is used to score your solution with the provided scoring function.\n\n```python\n{\n    ...\n    \"test_cases\": {\n        \"fn_name\": \"expression_matter\",\n        \"inputs\": [ ... ],\n        \"outputs\": [ ... ]\n    },\n    ...\n}\n```\n\n### Loading Problems\n\nUse the `load_sample` function to load a problem from the development or validation set.\n\n```python\nfrom utilities import load_sample\n\nproblem = load_sample(index=0, dataset_path=\"./data/dev\")\n```\n\n### Generating Code\n\nUse the `aleph_alpha_client` to generate code.\nMake sure your `AA_TOKEN` is set.\n\n```python\nfrom aleph_alpha_client import Client, CompletionRequest, Prompt\n\nclient = Client(AA_TOKEN)\n\nrequest = CompletionRequest(\n    prompt=Prompt.from_text(\"Your prompt.\"),\n    maximum_tokens=256,\n)\n\n# API reference for the client:\n# https://aleph-alpha-client.readthedocs.io/en/latest/\nresponse = client.complete(request, model=MODEL)\n```\n\n### Running Tests\n\nUse the `run_test_cases` function to run the generated code against the test cases.\nThe function returns a dictionary with the test results, including the expected output, the generated output, a boolean indicating whether the test passed and a traceback in case of an error.\n\n```python\nfrom utilities import run_test_cases\n\ntest_results = run_test_cases(\n    problem=problem, \n    generation=response.completions[0].completion, \n    timeout=10,\n)\n```\n\n### Scoring\n\nUse the `score` function to score your solution on the validation set.\nIt expects a function that takes a problem and a client and returns a generation.\n\n```python\nfrom utilities import score\n\npassed_problems, passed_test_cases = score(\n    generation_func=generate_code, \n    client=client,\n    dataset_path=\"./data/val\", \n    length=50,\n)\n```\n\n## Evaluation\n\nThe evaluation is done by running the generated code against a set of test cases. \nYou can evaluate your function on the provided validation set by running the scoring function in the notebook.\n\nIf you want to test your solution, you can send it to us via email and we will run it against the test set.\n\n## Attribution\n\nThe dataset is based on the [APPS dataset](https://github.com/hendrycks/apps), which is licensed under the MIT license. \nParts of the code are adapted from APPS.\n\n## Caution: Important Notice\n\n- It is forbidden to use any of the other samples besides the provided ones from the APPS dataset!\n- It is forbidden to use any other LLM besides the two provided ones (llama3.1-8b. llama3.1-70b)\n- Please be gentle and do not waste compute. \n  - Set max tokens\n  - only have a max of ten requests in flight\n  - use llama3.1-8b for debugging of your scripts (rather than 70b)\n  - Please refer also to https://docs.aleph-alpha.com/docs/best-practices/\n- Your solution is tested against a secret test set, which contains 200 additional samples. Keep the global timeout of 10 minutes in mind\n\n\nHave fun!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fbenchpress-hackathon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleph-alpha%2Fbenchpress-hackathon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fbenchpress-hackathon/lists"}