{"id":23932660,"url":"https://github.com/StonyBrookNLP/appworld-leaderboard","last_synced_at":"2025-09-11T15:32:04.407Z","repository":{"id":250641101,"uuid":"832318601","full_name":"StonyBrookNLP/appworld-leaderboard","owner":"StonyBrookNLP","description":"🌍 Leaderboard Repository for \"AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agent\", ACL2024","archived":false,"fork":false,"pushed_at":"2024-12-02T05:06:01.000Z","size":130,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-12-02T06:19:10.011Z","etag":null,"topics":["acl-2024","ai","ai-agents","ai-apis","ai-assistants","ai-environment","ai-planning","autonomous-agents","coding-agents","function-calling","interactive-coding","llm","llm-agents","nlp-datasets","nlp-machine","tool-usage"],"latest_commit_sha":null,"homepage":"https://appworld.dev/leaderboard","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StonyBrookNLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-22T19:22:00.000Z","updated_at":"2024-12-02T05:06:05.000Z","dependencies_parsed_at":"2024-12-05T18:15:23.542Z","dependency_job_id":null,"html_url":"https://github.com/StonyBrookNLP/appworld-leaderboard","commit_stats":null,"previous_names":["stonybrooknlp/appworld-leaderboard"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fappworld-leaderboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fappworld-leaderboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fappworld-leaderboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fappworld-leaderboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StonyBrookNLP","download_url":"https://codeload.github.com/StonyBrookNLP/appworld-leaderboard/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232657375,"owners_count":18556830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acl-2024","ai","ai-agents","ai-apis","ai-assistants","ai-environment","ai-planning","autonomous-agents","coding-agents","function-calling","interactive-coding","llm","llm-agents","nlp-datasets","nlp-machine","tool-usage"],"created_at":"2025-01-06T00:29:16.025Z","updated_at":"2025-01-06T00:29:17.649Z","avatar_url":"https://github.com/StonyBrookNLP.png","language":"Python","funding_links":[],"categories":["Building"],"sub_categories":["Benchmarks"],"readme":"# AppWorld Leaderboard\n\nThis is the leaderboard repository of the benchmark proposed in [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://appworld.dev/) (ACL 2024).\n\nThe project's main repository is [here](https://github.com/stonybrooknlp/appworld), and leaderboard UI is [here](https://appworld.dev/leaderboard). This repository stores bundled (encrypted) experiment outputs from participanting models (including our baselines) and the raw [leaderboard data JSON](/experiments/outputs/_leaderboard.json) which is dynamically rendered in the UI. You can use this repository to:\n\n1. Download and locally view experiment outputs from other participanting methods.\n2. Submit your own agent's experiment outputs to be included on the leaderboard via a PR.\n\nFor both cases, you first need to\n\n1. Install `appworld`: `pip install appworld \u0026\u0026 appworld install`.\n2. Install [Git LFS](https://git-lfs.com/) and clone the `appworld-leaderboard` repository.\n\n## Submit Your Agent's Outputs\n\n**First**, pack your agent's `test_normal` and `test_challenge` experiment outputs:\n\n\u003cdetails\u003e\n\u003csummary\u003e::Click:: Experiment outputs refresher\u003c/summary\u003e\n\n---\n\nYour experiment outputs are located in `./experiments/outputs/{experiment_name}` relative to the `APPWORLD_ROOT`, which as we discussed earlier, defaults to `.`, but can be configured by passing `APPWORLD_ROOT` environment variable or `--root` in CLI.\n\n---\n\n\u003c/details\u003e\n\nFor the leaderboard, experiment names must be alphanumeric with optional hyphens and underscores, and they must end with the dataset name, i.e., `_test_normal` or `_test_challenge`, e.g., `react_gpt4o_test_normal`. You should have two experiment outputs, one for each dataset. The prefix portion of their names must be the same, e.g., `react_gpt4o_test_normal` and `react_gpt4o_test_challenge`. Rename the directory accordingly if necessary.\n\nNow, `pack` the two experiments, individually, with the following commands, and same metadata.\n\n```bash\nappworld pack {test_normal_experiment_name|test_challenge_experiment_name} \\\n    # The method name used in the experiment:\n    --method_name METHOD_NAME \\\n    # A brief additional note about the method:\n    --method_tooltip METHOD_TOOLTIP \\\n    # The LLM name used in the experiment:\n    --llm_name LLM_NAME \\\n    # A brief additional note about the LLM\n    --llm_tooltip \\\n    # URL to find more information about this submission\n    -url URL\n# Example:\n# appworld pack react_gpt4o_test_normal \\\n#   --method_name react\n#   --method_tooltip 'Reason + Act'\n#   --llm_name 'GPT4-o'\n#   --url 'https://appworld.dev/'\n```\n\nThe pack command compresses and encrypts the experiment outputs in `leaderboard.bundle` files within your experiment output directories. E.g., `./experiments/outputs/react_gpt4o_test_normal/leaderboard.bundle`.\n\n\u003e [!Caution]\n\u003e Do NOT put your experiment outptus in an unencrypted format publicly accessible on the internet.\n\n**Next**, Copy the two bundle files at the following locations relative to `appworld-leaderboard` repo's root directory.\n\n```bash\n./experiments/outputs/{test_normal_experiment_name}/leaderboard.bundle\n./experiments/outputs/{test_challenge_experiment_name}/leaderboard.bundle\n```\n\nThen create a PR with these two files and post a comment as follows. Here, `EXPERIMENT_NAME_PREFIX` refers to the experiment name without the suffix of `_test_normal` and `_test_challenge` for the two splits.\n\n```bash\n# Change python and appworld version as desired. NOTE: Make sure there is not white space in the comment before the command starts.\n/add-to-leaderboard --python {PYTHON_VERSION} --appworld {APPWORLD_VERSION} {EXPERIMENT_NAME_PREFIX}\n```\n\nThis will start an automatic GitHub workflow which you can follow along in the GitHb Actions tab. If it's successful, it'll post a comment with your leaderboard entry in the PR, and update the leaderboard file, which you will be able to see in the PR diff as well. Verify the details in the comment. If you want to update the submission, just push updated bundle files and post the command comment again. Once you're happy with the result, assign the PR to me ([Harsh](https://github.com/harshTrivedi/)). I will take a look ASAP to ensure that no new changes are made after the last evaluation, and merge it.\n\nNote that you can also submit multiple agents submissions in the same PR. In this case add more bundle files to the PR and add more space-delimited experiment name prefixes to the PR command comment.\n\n\u003e [!IMPORTANT]\n\u003e **Reminder:** We track experiment outputs in encrypted `.bundle` files to reduce the risk of it becoming part of the training corpora of LLMs. So please do NOT post it here (or anywhere else publicly on the interent) in unencrypted or uncompressed format. See [license](https://github.com/stonybrooknlp/appworld/tree/main?tab=readme-ov-file#lock_with_ink_pen-license).\n\n\n## View Existing Experiment Outputs\n\n`CD` into the root of this repository. The experiment outputs are available in this format:\n\n```bash\nexperiments/outputs/{experiment_name}/leaderboard.bundle\n```\n\nTo unpack the bundle, run the following based on the experiment name you want to open.\n\n```bash\nappworld unpack {experiment_name}\n```\n\nThe bundle will be unpacked in the directory:\n\n```bash\nexperiments/outputs/{experiment_name}/\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FStonyBrookNLP%2Fappworld-leaderboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FStonyBrookNLP%2Fappworld-leaderboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FStonyBrookNLP%2Fappworld-leaderboard/lists"}