{"id":22083381,"url":"https://github.com/cubed-dev/cubed-benchmarks","last_synced_at":"2026-02-17T20:33:24.664Z","repository":{"id":225227219,"uuid":"765402588","full_name":"cubed-dev/cubed-benchmarks","owner":"cubed-dev","description":"Automated benchmark suite for testing the performance of cubed","archived":false,"fork":false,"pushed_at":"2025-03-17T10:27:06.000Z","size":46,"stargazers_count":2,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-24T19:37:08.363Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cubed-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-29T21:10:28.000Z","updated_at":"2024-10-30T14:48:22.000Z","dependencies_parsed_at":"2025-07-24T16:36:34.532Z","dependency_job_id":"f3ae5223-8fba-4d38-84df-92eefdc58dfa","html_url":"https://github.com/cubed-dev/cubed-benchmarks","commit_stats":null,"previous_names":["cubed-dev/cubed-benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cubed-dev/cubed-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cubed-dev%2Fcubed-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cubed-dev%2Fcubed-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cubed-dev%2Fcubed-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cubed-dev%2Fcubed-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cubed-dev","download_url":"https://codeload.github.com/cubed-dev/cubed-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cubed-dev%2Fcubed-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29557318,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T20:14:27.083Z","status":"ssl_error","status_checked_at":"2026-02-17T20:14:26.018Z","response_time":100,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T00:13:05.540Z","updated_at":"2026-02-17T20:33:19.652Z","avatar_url":"https://github.com/cubed-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cubed Benchmarks\n\nSet of Cubed benchmarks to run at scale on various executors.\n\n## Test Locally (for developers)\n\nThe `cubed benchmarks` test suite can be run locally with the following steps:\n\n1. Create a conda environment using `mamba env create -n test-env -f ci/environment.yml`\n2. Activate the environment with `conda activate test-env`\n3. Add test packages with `mamba env update -f ci/environment-test.yml`\n4. Run tests with `CUBED_CONFIG=tests/configs/local_single-threaded.yaml pytest`\n\n## Benchmarking\n\nThe `cubed-benchmarks` test suite contains a series of pytest fixtures which enable\nbenchmarking metrics to be collected and stored for historical and regression analysis.\nBy default, these metrics are not collected and stored, but they can be enabled\nby including the `--benchmark` flag in your pytest invocation.\n\nFrom a high level, here is how the benchmarking works:\n\n* Data from individual test runs are collected and stored in a local sqlite database.\n  The schema for this database is stored in `benchmark_schema.py`\n* The local sqlite databases are appended to a global historical record, stored in S3.\n* The historical data can be analyzed using any of a number of tools.\n\n### Running the benchmarks locally\n\nYou can collect benchmarking data by running pytest with the `--benchmark` flag.\nThis will create a local `benchmark.db` sqlite file in the root of the repository.\nIf you run a test suite multiple times with benchmarking,\nthe data will be appended to the database.\n\nYou can compare with historical data by downloading the global database from S3 first:\n\n```shell\naws s3 cp s3://cubed-runtime-ci/benchmarks/benchmark.db ./benchmark.db\nexport CUBED_CONFIG=...\npytest --benchmark\n```\n\n### Running the benchmarks locally using the `processes` executor\n\nThe `processes` executor uses all the cores on the local machine to run. These\nbenchmarks can be run locally on a machine with multiple cores (they are not\nrun in CI since CI hosts typically don't have many cores).\n\n```shell\nexport CUBED_CONFIG=tests/configs/local_processes.yaml\npytest --benchmark\n```\n\n### Inspecting the benchmarks database\n\nThere are many ways of doing this, but here is a simple one using DuckDB:\n\n```\n\u003e duckdb benchmark.db\nselect name, call_outcome, start, duration, name_prefix from test_run;\n```\n\n\n## Acknowledgements\n\nThis repository is heavily inspired by [`Coiled Benchmarks`](https://github.com/coiled/benchmarks/), with many sections of code being lifted directly.\n\n## License\n\n[Apache-2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcubed-dev%2Fcubed-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcubed-dev%2Fcubed-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcubed-dev%2Fcubed-benchmarks/lists"}