{"id":51531649,"url":"https://github.com/eth-sri/autobaxbuilder","last_synced_at":"2026-07-09T03:01:15.490Z","repository":{"id":355778854,"uuid":"1119435405","full_name":"eth-sri/AutoBaxBuilder","owner":"eth-sri","description":"Reproduction package for AutoBaxBuilder, a framework to build new BaxBench tasks from scratch","archived":false,"fork":false,"pushed_at":"2026-05-20T21:49:03.000Z","size":7399,"stargazers_count":12,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-21T03:36:28.579Z","etag":null,"topics":["backend-service","baxbench","llms","rest-api","security","vulnerabilities"],"latest_commit_sha":null,"homepage":"https://baxbench.com/autobaxbuilder/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eth-sri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-19T09:12:30.000Z","updated_at":"2026-05-20T21:49:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/eth-sri/AutoBaxBuilder","commit_stats":null,"previous_names":["eth-sri/autobaxbuilder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eth-sri/AutoBaxBuilder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eth-sri%2FAutoBaxBuilder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eth-sri%2FAutoBaxBuilder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eth-sri%2FAutoBaxBuilder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eth-sri%2FAutoBaxBuilder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eth-sri","download_url":"https://codeload.github.com/eth-sri/AutoBaxBuilder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eth-sri%2FAutoBaxBuilder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35284883,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend-service","baxbench","llms","rest-api","security","vulnerabilities"],"created_at":"2026-07-09T03:01:14.172Z","updated_at":"2026-07-09T03:01:15.479Z","avatar_url":"https://github.com/eth-sri.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AutoBaxBuilder\n\n## Overview\n\nWe present AutoBaxBuilder, an automated framework that generates code security benchmark tasks from scratch, reducing manual effort by ~12× while matching or outperforming expert tests and exploits.\n\n- Paper: [AutoBaxBuilder: Bootstrapping Code Security Benchmarking](https://arxiv.org/abs/2512.21132)\n- Website: [baxbench.com/autobaxbuilder](https://baxbench.com/autobaxbuilder)\n- Dataset: [HuggingFace](https://huggingface.co/datasets/eth-sri/autobaxbench)\n\n## Setup\n\nWe recommend using the **Docker-based workflow** for reproducibility. A native Conda-based setup is also provided for convenience.\n\n### Option 1: Docker\n\nBuild the Docker image, ensuring user and Docker group IDs are aligned with the host for reproducibility and correct permissions:\n\n```bash\ndocker build \\\n  --build-arg UID=$(id -u) \\\n  --build-arg GID=$(id -g) \\\n  --build-arg DOCKER_GID=$(getent group docker | cut -d: -f3) \\\n  -t autobaxbuilder .\n```\n\nShould `getent` not exist on your system, fall back to `--build-arg DOCKER_GID=$(stat -c '%g' /var/run/docker.sock)` instead.\n\nLoad environment variables in a `.env` file:\n\n```bash\nexport OPENAI_API_KEY=\"\u003cyour_API_key\u003e\"\nexport TOGETHER_API_KEY=\"\u003cyour_API_key\u003e\"\nexport ANTHROPIC_API_KEY=\"\u003cyour_API_key\u003e\"\nexport OPENROUTER_API_KEY=\"\u003cyour_API_key\u003e\"\n```\n\nRun an interactive shell inside the container, mounting the current directory and loading the environment variables:\n\n```bash\ndocker run \\\n  --network host \\\n  --env-file /path/to/env \\\n  -it \\\n  --memory=\"4g\" \\\n  --rm \\\n  -v /var/run/docker.sock:/var/run/docker.sock \\\n  -v $(pwd):/app \\\n  autobaxbuilder /bin/bash\n```\n\nIf `--network host` is not available in your system configuration, use `--add-host=localhost:host-gateway` instead.\n\n### Option 2: Native (Conda)\n\nThe project uses the conda package manager. Please install it from [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html). If you would like to have a lightweight installation, consider [Miniconda](https://docs.anaconda.com/miniconda/).\n\nThen, the environment can be installed by running the following command:\n\n```\nconda env create -n autobaxbuilder -f env.yaml\n```\n\nActivate the environment:\n\n```\nconda activate autobaxbuilder\n```\n\nThen, install the module:\n```\npip install -e .\n```\n\nOptional system dependencies (generally not needed, only required by some BaxBench scenarios):\n- **imagemagick** - For image conversion scenarios\n- **ffmpeg** - For video processing scenarios  \n- **poppler-utils** - For PDF text extraction (provides `pdftotext`)\n- **nodejs** - For TypeScript compilation scenarios\n- **g++** and **make** - For C++ compilation scenarios\n\n```bash\nsudo apt install imagemagick ffmpeg poppler-utils nodejs g++ make\n```\n\nOptionally, set up the pre-commit hooks:\n\n```\npre-commit install\n```\n\n\n## Generating New Scenarios with AutoBaxBuilder\n\nEnsure the docker daemon is running. Then, either use the convenience script (recommended) or run the stages manually.\n\n### Using the Convenience Script\nFrom the repository root, `autobaxbuilder.sh` runs the full generation pipeline for one or more scenarios. Usage: `./autobaxbuilder.sh [OPTIONS]`\n\n```bash\nOptions: \n  -n, --n-scenarios N    Total scenarios to generate                    (default: 1)\n  -P, --parallel N       Parallel workers                               (default: 1)\n  -d, --difficulty N     Number of endpoints: 1=easy 3=medium 5=hard    (default: 3)\n  -p, --path PATH        Artifacts base directory                       (default: \u003cscript_dir\u003e/artifacts)\n  -v, --debug            Enable debug output                            (default: off)\n```\n\nThe script generates scenarios, tests, and exploits, then copies the latest generated scenario `.py` file into the artifacts directory.\n\n### Manually\nRun AutoBaxBuilder from the repository root with `python src/main.py`. It runs in 3 modes, specified by the 3 flags:\n\n#### Modes\n- `--generate_scenarios`: Generate scenarios\n- `--generate_tests`: Generate tests (requires `--scenario` parameter)\n- `--generate_exploits`: Generate exploits (requires `--scenario` parameter)\n\n#### Required Parameters\n- `--scenario`: Scenario name (required when using `--generate_tests` or `--generate_exploits`)\n\n#### Optional Parameters\n- `--difficulty N`: Number of endpoints of the scenario\n- `--N_RETRIES N`: Number recovery steps in agentic retry loops\n- `--N_SOL_STEPS N`: Maximum steps for solution iteration\n- `--N_TEST_STEPS N`: Maximum steps for test iteration\n- `--N_SEC_STEPS N`: Maximum steps for security iteration\n- `--debug`: Debug mode (print additional information)\n- `--path PATH`: Artifact path\n\n#### Examples\n```bash\n# Generate scenarios\npython src/main.py --generate_scenarios\n\n# Generate tests for a specific scenario\npython src/main.py --generate_tests --scenario FooBarScenario\n\n# Generate exploits for a specific scenario\npython src/main.py --generate_exploits --scenario FooBarScenario\n\n# Run in debug mode\npython src/main.py --generate_tests --debug\n\n# Save artifacts in a different directory\npython src/main.py --generate_scenarios --path /path/to/artifacts/\n```\n\n#### Artifact Overview\n`--generate_scenarios` produces a new folder in the artifacts directory, corresponding to a novel scenario. `--generate_tests` takes the scenario name of a scenario produced with `--generate_scenarios`, on the basis of which it generates functional tests. `--generate_exploits` also takes a scenario name and builds on top of the previous steps to develop security tests. In each step, artifacts are created. For an exemplary scenario named `FooBarScenario`, these are structured as follows:\n- `FooBarScenario.json`: Initial scenario specification from `--generate_scenarios`\n- `FooBarScenario_iu{t}`: Scenario specification (JSON and Py) after t steps of test iteration\n- `FooBarScenario_iw{t}`: Scenario specification (JSON and Py) after t steps of security iteration\n- `FooBarScenario_implementations_it{t}`: Solutions after t steps of solution iteration\n- `FooBarScenario_implementations_iu{t}`: Solutions after t steps of test iteration\n- `FooBarScenario_implementations_iw{t}`: Solutions after t steps of security iteration\n- `FooBarScenario_results_{it/iu/iw}{t}`: Results (JSON and iteration matrix as png) of running the tests against the solutions in each intermediate step.\n- `FooBarScenario_tasklist.json`: Stored solution code paths (implementation detail)\n- `token_usage.txt` and `verdicts.txt`: Diagnostic logs\n\n### Evaluating Generated Scenarios\n\nThe `.py` artifacts the pipeline produces can directly be used as novel scenarios in the BaxBench framework. Refer to the instructions in the evaluation section below on how to generate, test and evaluate solutions for a scenario.\n\n## Evaluating AutoBaxBuilder\n\nThe AutoBaxBench scenarios are included with and without CWE-400 in `src/scenarios/with_cwe_400` and `src/scenarios/without_cwe_400` respectively. The scenarios of our alternative model ablation are in `src/scenarios/alt_model_ablation`. When generating your own scenarios, these are stored in the `artifacts/` directory by default. The latest artifact is the latest `artifacts/scenario_name/*_iw*.py` file. To run BaxBench with these scenarios and reproduce our evaluation results, follow the following steps.\n\n### 1. Clone the BaxBench Repository\n\n```bash\ngit clone git@github.com:logic-star-ai/baxbench.git\ncd baxbench\n```\n\n### 2. Set Up BaxBench Environment\n\nFollow the setup instructions in the [BaxBench repository](https://github.com/logic-star-ai/baxbench) to install dependencies and configure the environment. Alternatively, you can simply use the AutoBaxBuilder conda or docker setup from above, which already includes all required dependencies.\n\n### 3. Copy AutoBaxBuilder Scenario Files\n\nCopy the scenario files from AutoBaxBuilder to the BaxBench scenarios directory, adjusting paths as needed:\n\n```bash\n# Copy AutoBaxBuilder generated scenario artifact\ncp \"$(ls -v /path/to/autobaxbuilder/artifacts/scenario_name/*_iw*.py | tail -n 1)\" /path/to/baxbench/src/scenarios/\n\n# Copy AutoBaxBench scenario artifact\ncp /path/to/autobaxbuilder/src/scenarios/without_cwe_400/scenario_name.py /path/to/baxbench/src/scenarios/\n```\n\n### 4. Register Scenarios in BaxBench\n\nUpdate the `src/scenarios/__init__.py` file in BaxBench to include the new scenario(s). Add import statements for each scenario you copied and include it in the `all_scenarios` variable.\n\n### 5. Run BaxBench Evaluation\n\nNow you can run the full BaxBench evaluation pipeline on the generated scenarios:\n\n```bash\n# Generate solutions for a scenario\npython src/main.py \\\n  --models gpt-4o \\\n  --mode generate \\\n  --scenarios scenario_name\n\n# Test the generated solutions\npython src/main.py \\\n  --models gpt-4o \\\n  --mode test \\\n  --scenarios scenario_name\n\n# Evaluate the results\npython src/main.py \\\n  --models gpt-4o \\\n  --mode evaluate \\\n  --scenarios scenario_name\n```\n\nRefer to the [BaxBench repository](https://github.com/logic-star-ai/baxbench) for more details on how to generate, test, and evaluate solutions for scenarios.\n\n## Citation\n\nIf you find AutoBaxBuilder to be helpful in your research, please use the following citation\n\n```bibtex\n@article{vonarx2025autobaxbuilderbootstrappingcodesecurity,\n      title={AutoBaxBuilder: Bootstrapping Code Security Benchmarking}, \n      author={Tobias von Arx and Niels Mündler and Mark Vero and Maximilian Baader and Martin Vechev},\n      year={2025},\n      eprint={2512.21132},\n      archivePrefix={arXiv},\n}\n```\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feth-sri%2Fautobaxbuilder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feth-sri%2Fautobaxbuilder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feth-sri%2Fautobaxbuilder/lists"}