{"id":13750679,"url":"https://github.com/ordo-one/package-benchmark","last_synced_at":"2026-05-11T06:27:49.490Z","repository":{"id":60276067,"uuid":"527583250","full_name":"ordo-one/package-benchmark","owner":"ordo-one","description":"Swift benchmark runner with many performance metrics and great CI support","archived":false,"fork":false,"pushed_at":"2024-05-22T08:28:36.000Z","size":1080,"stargazers_count":267,"open_issues_count":18,"forks_count":20,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-22T15:43:40.864Z","etag":null,"topics":["benchmark","benchmark-framework","benchmarks","ordo-core-package","performance","performance-metrics","server-side-swift","swift","swift-package","swift-package-manager-plugin"],"latest_commit_sha":null,"homepage":"","language":"Swift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ordo-one.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-22T13:44:27.000Z","updated_at":"2024-08-13T15:01:49.695Z","dependencies_parsed_at":"2023-11-06T09:26:27.550Z","dependency_job_id":"dd39ee4c-366d-40eb-ad46-a6cdc072156a","html_url":"https://github.com/ordo-one/package-benchmark","commit_stats":{"total_commits":90,"total_committers":7,"mean_commits":"12.857142857142858","dds":0.2222222222222222,"last_synced_commit":"f7d347b96e855b744bad417a463ab873b69c7a99"},"previous_names":[],"tags_count":86,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ordo-one%2Fpackage-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ordo-one%2Fpackage-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ordo-one%2Fpackage-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ordo-one%2Fpackage-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ordo-one","download_url":"https://codeload.github.com/ordo-one/package-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224869035,"owners_count":17383306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmark-framework","benchmarks","ordo-core-package","performance","performance-metrics","server-side-swift","swift","swift-package","swift-package-manager-plugin"],"created_at":"2024-08-03T08:00:44.149Z","updated_at":"2025-10-21T10:52:09.926Z","avatar_url":"https://github.com/ordo-one.png","language":"Swift","funding_links":[],"categories":["Swift","Frameworks with plugins"],"sub_categories":[],"readme":"[![](https://img.shields.io/endpoint?url=https%3A%2F%2Fswiftpackageindex.com%2Fapi%2Fpackages%2Fordo-one%2Fpackage-benchmark%2Fbadge%3Ftype%3Dswift-versions)](https://swiftpackageindex.com/ordo-one/package-benchmark)\n[![](https://img.shields.io/endpoint?url=https%3A%2F%2Fswiftpackageindex.com%2Fapi%2Fpackages%2Fordo-one%2Fpackage-benchmark%2Fbadge%3Ftype%3Dplatforms)](https://swiftpackageindex.com/ordo-one/package-benchmark)\n[![codecov](https://codecov.io/gh/ordo-one/package-benchmark/branch/main/graph/badge.svg?token=hXHmhEG1iF)](https://codecov.io/gh/ordo-one/package-benchmark)\u003cbr\u003e\n[![Swift address sanitizer](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-sanitizer-address.yml/badge.svg)](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-sanitizer-address.yml)\n[![Swift thread sanitizer](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-sanitizer-thread.yml/badge.svg)](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-sanitizer-thread.yml)\n[![Swift Linux build](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-linux-build.yml/badge.svg)](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-linux-build.yml)\n[![Swift macOS build](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-macos-build.yml/badge.svg)](https://github.com/ordo-one/package-benchmark/actions/workflows/swift-macos-build.yml)\n\n# Benchmark \n\nThe Benchmark package allows you to easily create sophisticated Swift performance benchmarks for a wide variety of metrics.\n\nThis README provides a quick overview of the Benchmark package. For more detailed information, please [see the documentation](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark).\n\n## Overview\n\nPerformance is a key feature for many apps and frameworks. Benchmark helps make it easy to measure and track [many different metrics](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/metrics) that affects performance, such as CPU usage, ARC traffic, memory/malloc usage and use of operating system resources such as threads and system calls, as well as completely custom metric counters.\n\nBenchmark works on both macOS and Linux and supports several key workflows for performance measurements:\n\n* **[Automated Pull Request performance regression checks](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/comparingbenchmarksci)** by comparing the performance metrics of a pull request with the main branch and having the PR workflow check fail if there is a regression according to absolute or relative threshold tolerances specified per benchmark\n* **[Manual comparison of multiple performance baselines](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/creatingandcomparingbaselines)** for iterative or A/B performance work by an individual developer\n* **[Export of benchmark results in several formats](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/exportingbenchmarks)** for analysis or visualization\n\nBenchmark provides a quick way for measuring and validating of performance metrics, while other more specialized tools such as Instruments, DTrace, Heaptrack, Leaks, Sample and more can be used for attributing performance problems or for finding root causes for any deviations found.\n\nBenchmark is suitable for both smaller ad-hoc benchmarks focusing on execution time and more extensive benchmarks that care about several additional metrics such as memory allocations, syscalls, thread usage, context switches, ARC traffic, and more. Using [Histogram](https://github.com/ordo-one/package-histogram) it’s especially suitable for capturing latency statistics for large number of samples.\n\n## Documentation\n\nDocumentation on how to use Benchmark in your Swift package can be [viewed online](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark) or inside Xcode using `Build Documentation`. \n\nAdditionally the command plugin provides help information if you run `swift package benchmark help` from the command line.\n\n## Adding dependencies and getting started\n\nThere are just a few steps required to get started benchmarking:\n1. Add a dependency to the Benchmark project\n2. Add benchmark executable targets with `swift package benchmark init`\n3. Add the snippet or code you want to benchmark\n4. Run `swift package benchmark`\n\nThe steps in some detail:\n### Step 1: Add a package dependency to Package.swift\nTo add the dependency on Benchmark, add the dependency to your package:\n```swift\n.package(url: \"https://github.com/ordo-one/package-benchmark\", .upToNextMajor(from: \"1.4.0\")),\n```\n\n### Step 2: Add benchmark exectuable targets using `benchmark init`\nThe absolutely easiest way to add new benchmark executable targets to your project is by using:\n```bash\nswift package --allow-writing-to-package-directory benchmark init MyNewBenchmarkTarget\n```\n\nThis will perform the following steps for you:\n* Create a Benchmarks/MyNewBenchmarkTarget directory\n* Create a Benchmarks/MyNewBenchmarkTarget/MyNewBenchmarkTarget.swift benchmark target with the required boilerplate\n* Add a new executable target for the benchmark to the end of your Package.swift file\n\nThe init command validates that the name you specify isn’t used by any existing target and will not overwrite any existing file with that name in the Benchmarks/ location. \n\nAfter you’ve created the new target, you can directly run it with e.g.:\n```bash\nswift package benchmark --target MyNewBenchmarkTarget\n```\n\n### Step 2 (optional approach): Add benchmark exectuable targets manually\nAlternatively if you don't want the plugin to modify your project directory, you can do the same steps manually:\nCreate an executable target in Package.swift for each benchmark suite you want to measure.\nThe source for all benchmarks must reside in a directory named `Benchmarks` in the root of your swift package.\nThe benchmark plugin uses this directory combined with the executable target information to automatically discover and run your benchmarks.\nFor each executable target, include dependencies on both `Benchmark` (supporting framework) and `BenchmarkPlugin` (boilerplate generator) from package-benchmark.\nThe following example shows an benchmark suite named `My-Benchmark` with the required dependency on `Benchmark` and the source files for the benchmark that reside in the directory `Benchmarks/My-Benchmark`:\n```swift\n.executableTarget(\n      name: \"My-Benchmark\",\n      dependencies: [\n          .product(name: \"Benchmark\", package: \"package-benchmark\"),\n      ],\n      path: \"Benchmarks/My-Benchmark\",\n      plugins: [\n          .plugin(name: \"BenchmarkPlugin\", package: \"package-benchmark\"),\n      ]\n),\n```\n\n## Step 3: Writing benchmarks\nThere are [documentation available](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/writingbenchmarks) as well as a [a sample project](https://github.com/ordo-one/package-benchmark-samples) using various aspects of this package in practice.\n\n## Sample benchmark code\n```swift\nimport Benchmark\n\nlet benchmarks : @Sendable () -\u003e Void = {\n    Benchmark(\"Minimal benchmark\") { benchmark in\n      // measure something here\n    }\n\n    func defaultCounter() -\u003e Int {\n        10\n    }\n    \n    func dummyCounter(_ count: Int) {\n        for index in 0 ..\u003c count {\n            blackHole(index)\n        }\n    }\n    \n    Benchmark(\"All metrics, full concurrency, async\",\n              configuration: .init(metrics: BenchmarkMetric.all,\n                                   maxDuration: .seconds(10))) { benchmark in\n        let _ = await withTaskGroup(of: Void.self, returning: Void.self, body: { taskGroup in\n            for _ in 0..\u003c80  {\n                taskGroup.addTask {\n                    dummyCounter(defaultCounter()*1000)\n                }\n            }\n            for await _ in taskGroup {\n            }\n        })\n    }     \n}\n```\n\n### Step 4: Running benchmarks\nTo execute all defined benchmarks, simply run:\n\n```swift package benchmark```\n\nPlease see the [documentation](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/runningbenchmarks) for more detail on all options.\n\n### Sample output benchmark run\n\u003cimg width=\"1005\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225311258-1247f8e9-c1fd-4598-a4b8-2b41a9b9a8e7.png\"\u003e\n\n### Sample output benchmark grouped by metric \n\u003cimg width=\"1089\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225281786-411530de-25c2-47b5-b99f-0d7bac3209a7.png\"\u003e\n\n### Sample output delta comparison\n\u003cimg width=\"1173\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225282373-e7ba9fa1-1a2a-4028-b053-9f3aa82361b0.png\"\u003e\n\n### Sample output threshold deviation check\n\u003cimg width=\"956\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225282982-95c9c641-9455-4df2-81bc-6aee43721223.png\"\u003e\n\n### Sample usage of YouPlot\nInstall [YouPlot](https://github.com/red-data-tools/YouPlot)\n\n```bash\nswift package benchmark run --filter InternalUTCClock-now --metric wallClock --format histogramPercentiles --path stdout --no-progress | uplot lineplot -H\n```\n\n\u003cimg width=\"523\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225284254-c1349494-2323-4460-b18a-7bc2896b5dc4.png\"\u003e\n\n### JMH Visualization\n\nUsing [jmh.morethan.io](https://jmh.morethan.io)\n\n\u003cimg width=\"1262\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225313246-4369da1f-0890-4856-8fd8-b28d56d842aa.png\"\u003e\n\n\u003cimg width=\"1482\" alt=\"image\" src=\"https://user-images.githubusercontent.com/8501048/225313559-33014755-797f-4ddf-b536-24c1a618f271.png\"\u003e\n\n## Swift 6 support\nThe package supports Swift 6.0 benchmark targets as well as Swift 5.10 targets (for Swift 5.9 support, need to use version 1.28.0 exactly).\n\nFor Swift 6 targets, use the following signature for the benchmarks closure:\n```swift\nlet benchmarks : @Sendable () -\u003e Void = {\n```\n\n## Output\n\nThe default text output from Benchmark is oriented around [the five-number summary](https://en.wikipedia.org/wiki/Five-number_summary) percentiles, plus the last decile (`p90`) and the last percentile (`p99`) - it's thus a variation of a [seven-figure summary](https://en.wikipedia.org/wiki/Seven-number_summary) with the focus on the 'bad' end of results (as those are what we typically care about addressing).\nWe've found that focusing on percentiles rather than average or standard deviations, is more useful for a wider range of benchmark measurements and gives a deeper understanding of the results.\nPercentiles allows for a consistent way of expressing benchmark results of both throughput and latency measurements (which typically do **not** have a standardized distribution, being almost always multi-modal in nature).\nThis multi-modal nature of the latency measurements leads to the common statistical measures of mean and standard deviation being potentially misleading.\n\nPlease see the [documentation for further details on percentiles](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/aboutpercentiles).\n\n## API and file format stability\nThe API will be deemed stable as of `1.0.0` and follows semantical versioning for future releases. \n\nThe export file formats that are externally defined (e.g. JMH or HDR Histogram formats) will follow the upstream definitions if they change, but have been quite stable for several years. \n\nThe Histogram codable representation is not stable and may change if the Histogram implementation changes.\n\nThe benchmark internal baseline representation (stored in `.benchmarkBaselines`) is not stable and is not viewed as public API and may break over time.\n\nFor those wanting to save benchmark data over time, it's recommended to export data in e.g. HDR Histogram representations (percentiles, average, stddev etc) or simply post processing the histogramSamples format (which is raw data) to your desired representation.\n\nPR:s for additional standardized formats are welcome, as the export formats are the intended stable interface for saving such data.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fordo-one%2Fpackage-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fordo-one%2Fpackage-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fordo-one%2Fpackage-benchmark/lists"}