https://github.com/aielte-research/hacksynth
LLM Agent and Evaluation Framework for Autonomous Penetration Testing
https://github.com/aielte-research/hacksynth
ai autonomous-pentesting ctf ctf-tools cybersecurity llms penetration-testing
Last synced: 5 months ago
JSON representation
LLM Agent and Evaluation Framework for Autonomous Penetration Testing
- Host: GitHub
- URL: https://github.com/aielte-research/hacksynth
- Owner: aielte-research
- License: agpl-3.0
- Created: 2024-11-27T16:02:49.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-06-24T10:20:29.000Z (12 months ago)
- Last Synced: 2025-09-09T01:25:31.930Z (10 months ago)
- Topics: ai, autonomous-pentesting, ctf, ctf-tools, cybersecurity, llms, penetration-testing
- Language: Python
- Homepage:
- Size: 1.92 MB
- Stars: 221
- Watchers: 3
- Forks: 38
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
The paper can be found on [arXiv](https://arxiv.org/abs/2412.01778).
## Introduction
We introduce HackSynth, a novel Large Language Model (LLM)-based agent capable of autonomous penetration testing.
HackSynth's dual-module architecture includes a Planner and a Summarizer, which enable it to generate commands and process feedback iteratively.
To benchmark HackSynth, we propose two new Capture The Flag (CTF)-based benchmark sets utilizing the popular platforms PicoCTF and OverTheWire.
These benchmarks include two hundred challenges across diverse domains and difficulties, providing a standardized framework for evaluating LLM-based penetration testing agents.
## Using the repository
- You will have to create a Hugging Face and a Neptune.ai account
- Copy your API keys to the `.env` file, and set the desired CUDA devices, based on the `.env_example`
- [Set up the PicoCTF benchmark](picoctf_bench/README.md)
- [Set up the OverTheWire benchmark](overthewire_bench/README.md)
- Start the HackSynth Agent
- Install the environment:
```
python -m venv cyber_venv
source cyber_venv/bin/activate
pip install -r requirements.txt
```
- Start the benchmark with the following:
```
python run_bench.py -b benchmark.json -c config.json
```
The `benchmark.json` should be one of the generated `benchmark_solved.json` files, or an equivalently structured file.
The configuration files used by us for the measurements in the paper are also available in the configs folder.
## How to Cite
If you use this code in your work or research, please cite the corresponding paper:
```bibtex
@misc{muzsai2024hacksynthllmagentevaluation,
title={HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing},
author={Lajos Muzsai and David Imolai and András Lukács},
year={2024},
eprint={2412.01778},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2412.01778},
}
```
## Contributors
- Lajos Muzsai (muzsailajos@protonmail.com)
- David Imolai (david@imol.ai)
- András Lukács (andras.lukacs@ttk.elte.hu)
> 🔍 Also see our related project on reinforcement learning for cryptographic CTFs: [HackSynth-GRPO](https://github.com/aielte-research/HackSynth-GRPO)
## License
The project uses the GNU AGPLv3 license.