{"id":13572114,"url":"https://github.com/ethz-spylab/satml-llm-ctf","last_synced_at":"2025-04-04T09:31:45.514Z","repository":{"id":207658928,"uuid":"718695662","full_name":"ethz-spylab/satml-llm-ctf","owner":"ethz-spylab","description":"Code used to run the platform for the LLM CTF colocated with SaTML 2024","archived":false,"fork":false,"pushed_at":"2024-03-20T15:00:18.000Z","size":1094,"stargazers_count":25,"open_issues_count":0,"forks_count":6,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-11-05T05:33:52.565Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://ctf.spylab.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ethz-spylab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-14T16:01:39.000Z","updated_at":"2024-10-20T01:57:32.000Z","dependencies_parsed_at":"2023-11-17T00:11:39.386Z","dependency_job_id":"cb2f133e-2f83-442f-809d-22ceabe6a077","html_url":"https://github.com/ethz-spylab/satml-llm-ctf","commit_stats":null,"previous_names":["ethz-spylab/satml-llms-ctf-issues","ethz-spylab/satml-llm-ctf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-spylab%2Fsatml-llm-ctf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-spylab%2Fsatml-llm-ctf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-spylab%2Fsatml-llm-ctf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-spylab%2Fsatml-llm-ctf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ethz-spylab","download_url":"https://codeload.github.com/ethz-spylab/satml-llm-ctf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247153464,"owners_count":20892670,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T14:01:13.766Z","updated_at":"2025-04-04T09:31:40.502Z","avatar_url":"https://github.com/ethz-spylab.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# LLM CTF\n\nThis is the code used to run the [2024 SaTML LLM CTF](https://ctf.spylab.ai/). The code was developed from scratch by:\n\n- [Edoardo Debenedetti](https://github.com/dedeswim)\n- [Daniel Paleka](https://github.com/dpaleka)\n- [Javier Rando](https://github.com/javirandor)\n- [@nustom](https://github.com/nustom) (hired with the support of [BERI](https://existence.org/))\n\nThe app is a [FastAPI](https://fastapi.tiangolo.com/) web server, with a [MongoDB](https://www.mongodb.com) database,\nand a [Redis](https://redis.io/) cache. The web server is served by [Uvicorn](https://www.uvicorn.org/). Everything runs\nin [Docker](https://www.docker.com/) and `docker compose`.\n\nNote that the platform was developed while the competition was running and the exact specifics were being detailed,\nso not all design decisions were optimal.\n\nWe ran the application on a single Google Cloud VM with 64GB of RAM and 32 vCPUs. This was enough for most of the\ncompetition, but the most heated phases were running a bit too slow.\n\nSome potential improvements that could be done to the platform (PRs welcome!) are:\n\n- [ ] Move to a relational DB, as DB operations turned out to be more relational than we expected when we first started\nthe project.\n- [ ] Write **real** tests for the code. Currently, we have some form of [integration tests](tests/basic_api_test.py)\nthat test the API, but we don't have any unit tests.\n- [ ] Make the whole repo more templetable, so that it can be used as a starting point for other CTFs and similar projects.\n- [ ] Simplify the slight mess in [`app/schemas`](app/schemas). Currently, there is some redundancy in the schema classes.\n- [ ] Move from `docker compose` to `kubernetes` or something similar for better scalability and reliability.\n- [ ] Use [`rye`](https://rye-up.com) to manage the Python project.\n\n## Setting up the environment\n\n1. Create a `.env` file with the same content as `.env.example`, and change the values as needed.\n2. Create a `.env.prod` file with the same content as `.env.example`, and change the values as needed.\n3. Create a `.secrets` folder with the same content as `secrets.example`, and change the values as instructed in each file.\n\n## How to (re)start the application\n\n```\ndocker compose --env-file .env.prod -f compose.prod.yml up --build -d\n```\n\nor\n\n```\ndocker compose --env-file .env.prod -f compose.prod.yml up --build -d web\n```\n\nTo only start the web service container. If the container(s) are already running, then they will be re-built and re-started.\n\n### Development\n\n```bash\ndocker compose up --build -d web\n```\n\n### Production\n\n```bash\ndocker compose --env-file .env.prod -f compose.prod.yml up --build -d web\n```\n\n### Stopping\n\n```bash\ndocker compose down\n```\n\nUse `web` if you want to start the app, otherwise don't specify to start everything\n\n### Checking the logs\n\n```bash\ndocker compose logs -f\n```\n\nThe `-f` flag behaves like in `cat`.\n\n## Linting and code style\n\nLint with\n\n```bash\nruff check --fix .\n```\n\nFormat with\n\n```bash\nruff format .\n```\n\nCheck types with\n\n```bash\nmypy .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethz-spylab%2Fsatml-llm-ctf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fethz-spylab%2Fsatml-llm-ctf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethz-spylab%2Fsatml-llm-ctf/lists"}