{"id":28562649,"url":"https://github.com/rustls/rustls-bench-app","last_synced_at":"2025-06-10T12:10:48.771Z","repository":{"id":204858079,"uuid":"712477995","full_name":"rustls/rustls-bench-app","owner":"rustls","description":null,"archived":false,"fork":false,"pushed_at":"2024-09-23T14:42:06.000Z","size":276,"stargazers_count":13,"open_issues_count":3,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-08T00:08:38.040Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rustls.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-10-31T14:47:53.000Z","updated_at":"2024-09-23T14:07:16.000Z","dependencies_parsed_at":"2023-12-06T13:29:39.074Z","dependency_job_id":"5d88b88b-1a98-4113-94c2-a1421cfafc87","html_url":"https://github.com/rustls/rustls-bench-app","commit_stats":null,"previous_names":["aochagavia/rustls-bench-app","rustls/rustls-bench-app"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustls%2Frustls-bench-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustls%2Frustls-bench-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustls%2Frustls-bench-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustls%2Frustls-bench-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rustls","download_url":"https://codeload.github.com/rustls/rustls-bench-app/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustls%2Frustls-bench-app/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259072991,"owners_count":22801090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-10T12:10:47.707Z","updated_at":"2025-06-10T12:10:48.762Z","avatar_url":"https://github.com/rustls.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rustls bench runner\n\nThis repository contains rustls' bench runner software, meant to be deployed to a bare-metal server\n(though nothing prevents deploying to a VPS for testing). There are two main directories, each with\na readme providing more information:\n\n- `ansible`: ansible scripts to set up the server and manage deployments.\n- `ci-bench-runner`: a Rust binary that runs benchmarks and reports results based on GitHub activity.\n\n## Configuration\n\n#### GitHub app\n\nThe bench runner needs an associated [GitHub app](https://docs.github.com/en/apps) to receive event\nnotifications and react to them. The app should be private, to ensure it only gets installed on the\nofficial rustls repository. If you are creating the GitHub app from scratch (e.g. for testing\npurposes), ensure it requires the following _repository permissions_:\n\n- Commit statuses: read and write.\n- Contents: read-only.\n- Issues: read and write.\n- Metadata: read-only.\n- Pull requests: read and write.\n\nThe app should also require subscriptions to the following events (don't forget to configure the\nwebhook's URL _and_ a secret to ensure event authenticity):\n\n- Issue comment.\n- Pull request.\n- Pull request review.\n- Push.\n\nMake sure to set the GitHub webook URL for your application with the correct\n`/webhooks/github` route, e.g.:\n\n- `https://my-domain.example.com/webhooks/github`\n\n#### Hardware settings\n\nTo reduce noise, the server should be configured to disable dynamic frequency scaling (also known as\nTurbo Boost) and hyper threading. This needs to be done at the BIOS / UEFI level.\n\n#### Bencher.dev project\n\nThe bench runner saves the results from `main` to a local database, but it can also push them to\n[Bencher.dev] for visualization. For that, you will need to:\n\n- Set up a project\n- Create a testbed in the project\n- Create an API token (tokens aren't scoped, so if you are going to share them with others you might\n  want to create a unique Bencher.dev account just for this project)\n\n## Features\n\nThe following features are supported:\n\n- Run the benchmarks on every push to `main` and store the results.\n- Calculate a per-benchmark significance threshold based on the result history for the `main` branch.\n- Run the benchmarks on pull requests, comparing the results against the pull request's base branch.\n  For security, comparison bench runs are only triggered in the following scenarios:\n  - A PR is created or updated and the head branch lives in the rustls repository.\n  - A maintainer leaves a GitHub review approving the PR.\n  - A maintainer posts a comment to the PR including `@rustls-benchmarking bench` as part of the\n    body. This can be used as a fallback mechanism when the triggers mentioned above are not enough.\n- Report comparison results in a comment to the relevant PR, reusing the same comment when new\n  results are available.\n- Pause event processing by creating a file called `pause` in the application's working directory.\n- Show information about the application through the `/info` endpoint. Includes the hash of the\n  deployed commit, the id of the active job (if any) and whether event processing is currently\n  enabled.\n\nInteresting ideas for later:\n\n- Compile the benchmarks using different C toolchains, like GCC and LLVM (we are currently using the\n  system's GCC).\n\n## Nix Support\n\nSince this repository requires both Ansible, and Rust, a [Nix flake] is provided\nto offer a self-contained developer environment.\n\nTo use:\n1. [Install Nix]\n2. `nix develop`\n3. You're all set! Use `cargo` and `ansible` as needed.\n\nYou may also install [Direnv] and `direnv allow` this directory to have the Nix\ndeveloper environment load/unload automatically when you enter/leave the\ndirectory.\n\n[Nix flake]: https://zero-to-nix.com/concepts/flakes\n[Install Nix]: https://nixos.org/\n[Direnv]: https://direnv.net/\n[Bencher.dev]: https://bencher.dev/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustls%2Frustls-bench-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frustls%2Frustls-bench-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustls%2Frustls-bench-app/lists"}