{"id":29303550,"url":"https://github.com/scverse/benchmark","last_synced_at":"2026-03-01T08:34:47.317Z","repository":{"id":223282237,"uuid":"759806546","full_name":"scverse/benchmark","owner":"scverse","description":"Code for running benchmarks","archived":false,"fork":false,"pushed_at":"2025-08-12T10:06:22.000Z","size":259,"stargazers_count":3,"open_issues_count":8,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-12T10:29:29.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scverse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-19T11:22:00.000Z","updated_at":"2025-08-12T10:06:24.000Z","dependencies_parsed_at":"2024-05-07T16:27:13.139Z","dependency_job_id":"3126f4bd-e793-44d7-b2b6-bd1b4627afbc","html_url":"https://github.com/scverse/benchmark","commit_stats":null,"previous_names":["scverse/benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/scverse/benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scverse%2Fbenchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scverse%2Fbenchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scverse%2Fbenchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scverse%2Fbenchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scverse","download_url":"https://codeload.github.com/scverse/benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scverse%2Fbenchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29965406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T06:55:38.174Z","status":"ssl_error","status_checked_at":"2026-03-01T06:53:04.810Z","response_time":124,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-07T01:09:00.440Z","updated_at":"2026-03-01T08:34:47.301Z","avatar_url":"https://github.com/scverse.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# benchmark machine\n\n## Development plan:\n\n1. [MVP](https://github.com/scverse/benchmark/milestone/1):\n\n   Queue and runs exist on the benchmark server, which handles everything\n\n2. [Dedicated queue](https://github.com/scverse/benchmark/milestone/2)\n\n   To avoid a web server running in the background, have a VM or so that maintains the queue\n\n3. [Cancellation](https://github.com/scverse/benchmark/milestone/3)\n\n   Improve UX by allowing cancellation and other niceties\n\n## Usage\n\nEventually, we’ll just have a project-wide webhook like this. For now, if you want to test:\n\n1. Add a [asv config][] to your project (either the project root or a \u003csamp\u003ebenchmarks\u003c/samp\u003e directory)\n2. Add a webhook to your scverse project with these [webhook settings][], i.e.\n   - URL and secret from [the app’s 1password][]\n   - Content type: \u003csamp\u003eapplication/json\u003c/samp\u003e\n   - Let me select individual events → **Pull Requests**\n3. Add a label \u003ckbd\u003ebenchmark\u003c/kbd\u003e to a PR authored by a trusted user.\n4. Watch [scverse-benchmarks][] add and update a comment with the PR’s performance impact.\n\n[asv config]: https://asv.readthedocs.io/en/v0.6.1/using.html\n[webhook settings]: https://github.com/scverse/benchmark/settings/hooks/464592128\n[the app’s 1password]: https://start.1password.com/open/i?a=4HRQALMH4VEWFDAIQ53YNCNIG4\u0026v=cc6gruiwkh6shmg4mmszacgmrq\u0026i=qz7yf6i5w52lk4gp3cnmkqodfm\u0026h=scverse.1password.com\n[scverse-benchmarks]: https://github.com/apps/scverse-benchmark\n\n## MVP Setup\n\nAll these currently assume you have a \u003csamp\u003e\u0026lt;user\u003e\u003c/samp\u003e login with sudo rights on the \u003csamp\u003escvbench\u003c/samp\u003e server.\n\n### Debugging\n\n- Use Rocky Linux’ cockpit (e.g. run `ssh -L 9090:\u003cip\u003e:9090 \u003cyourname\u003e@\u003cip\u003e` and then go to http://localhost:9090)\n- Use `journalctl -u benchmark -f` on the server to tail the logs of the service.\n- Check GitHub’s page for [Hook deliveries][].\n\n[Hook deliveries]: https://github.com/scverse/benchmark/settings/hooks/464592128?tab=deliveries\n\n### One-time server setup\n1. As the \u003csamp\u003ebenchmarker\u003c/samp\u003e user, install micromamba, then:\n\n   ```shell\n   micromamba create -n asv -c conda-forge conda mamba virtualenv asv\n   micromamba run -n asv asv machine --yes\n   ```\n\n   (use `micromamba activate asv` to make `asv` available in your PATH)\n\n2. Update `LoadCredentialEncrypted` lines in \u003csamp\u003ebenchmark.service\u003c/samp\u003e using\n\n   ```shell\n   sudo systemd-creds encrypt --name=webhook_secret secret.txt -\n   sudo systemd-creds encrypt --name=app_key app-key.pem -\n   shred secret.txt app-key.pem\n   ```\n\n3. Copy the \u003csamp\u003ebenchmark.service\u003c/samp\u003e file to the system, enable and start the service:\n\n   ```console\n   $ rsync benchmark.service \u003cuser\u003e@scvbench:\n   $ ssh \u003cuser\u003e@scvbench\n   scvbench$ sudo mv benchmark.service /etc/systemd/system/\n   scvbench$ sudo systemctl enable --now benchmark\n   ```\n\nFurther steps:\n1. Setup chrony (\u003csamp\u003e/etc/chrony.conf\u003c/samp\u003e) to use internal servers\n\n   ```ini\n   server 146.107.1.13 trust\n   server 146.107.5.10\n   ```\n\n2. [Performance setup](https://github.com/scverse/benchmark/issues/1)\n\n### Deployment\n1. Make changes in \u003csamp\u003e\u0026lt;branch\u003e\u003c/samp\u003e (either \u003csamp\u003emain\u003c/samp\u003e or a PR branch) and wait until CI finishes.\n2. Run `nu scripts/deploy.nu \u003cbranch\u003e --user=\u003cuser\u003e`.\n3. Trigger a run,\n   e.g. remove and re-add the \u003ckbd\u003ebenchmark\u003c/kbd\u003e label in [PR 11][].\n\n[PR 11]: https://github.com/scverse/benchmark/pull/11\n\n## Development\n\nFor local development:\n\n1. Start the server locally\n2. use `scripts/test.nu` to send a payload (check the script for examples for both steps)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscverse%2Fbenchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscverse%2Fbenchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscverse%2Fbenchmark/lists"}