{"id":50315331,"url":"https://github.com/paulegradie/sailfish","last_synced_at":"2026-06-17T07:01:21.311Z","repository":{"id":38101080,"uuid":"458722167","full_name":"paulegradie/Sailfish","owner":"paulegradie","description":"Sailfish - a production-friendly performance benchmark runner for .NET","archived":false,"fork":false,"pushed_at":"2026-06-10T11:09:22.000Z","size":15825,"stargazers_count":12,"open_issues_count":2,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-10T13:07:34.103Z","etag":null,"topics":["benchmark","performance","test-automation","testing","testing-tools"],"latest_commit_sha":null,"homepage":"https://paulgradie.com/Sailfish/","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulegradie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-02-13T05:47:47.000Z","updated_at":"2026-06-10T10:18:22.000Z","dependencies_parsed_at":"2024-06-22T15:42:02.436Z","dependency_job_id":"f15d037c-a9e7-4b23-9ec3-1912957c3261","html_url":"https://github.com/paulegradie/Sailfish","commit_stats":null,"previous_names":[],"tags_count":109,"template":false,"template_full_name":null,"purl":"pkg:github/paulegradie/Sailfish","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulegradie%2FSailfish","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulegradie%2FSailfish/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulegradie%2FSailfish/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulegradie%2FSailfish/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulegradie","download_url":"https://codeload.github.com/paulegradie/Sailfish/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulegradie%2FSailfish/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34437451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","performance","test-automation","testing","testing-tools"],"created_at":"2026-05-29T00:00:31.159Z","updated_at":"2026-06-17T07:01:21.254Z","avatar_url":"https://github.com/paulegradie.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sailfish\n\n[![Build Pipeline v4.0](https://github.com/paulegradie/Sailfish/actions/workflows/build-v4.0.yml/badge.svg)](https://github.com/paulegradie/Sailfish/actions/workflows/build-v4.0.yml)\n![NuGet](https://img.shields.io/nuget/dt/Sailfish)\n[![codecov](https://codecov.io/gh/paulegradie/Sailfish/graph/badge.svg?token=UN17VRVD0N)](https://codecov.io/gh/paulegradie/Sailfish)\n\nSailfish is a .NET performance testing framework that makes it easy to write, run, and analyze performance tests with statistical rigor.\n\n- Documentation: https://paulgradie.com/Sailfish/\n- NuGet: https://www.nuget.org/packages/Sailfish/\n\n---\n\n## Table of Contents\n- [Features](#features)\n- [Quick Start](#quick-start)\n- [Algorithm Comparisons](#algorithm-comparisons)\n- [Adaptive Sampling](#adaptive-sampling)\n- [Installation](#installation)\n- [IDE setup](#ide-setup)\n- [Outputs and Reporting](#outputs-and-reporting)\n- [Used By](#used-by)\n- [Contributing](#contributing)\n- [License](#license)\n\n---\n\n## Features\n- Statistical rigor: outlier detection, multiple tests, significance analysis\n- **Method comparisons by default** — every `[SailfishMethod]` in a `[Sailfish]` class is automatically compared; opt out per class with `[Sailfish(DisableComparison = true)]`\n- Multiple outputs: test logs, Markdown, CSV\n- Easy CI/CD integration\n- Historical analysis with SailDiff\n- Timer calibration with 0–100 Jitter Score; shows in Markdown header, manifest, and Environment Health\n- Seeded run order (opt‑in): Deterministic ordering across classes, methods, and variable sets when a seed is provided; seed appears in Markdown header and manifest\n\n\n- Highly configurable\n\n## Quick Start\n\n```csharp\n[Sailfish]\npublic class MyPerformanceTest\n{\n    [SailfishMethod]\n    public void TestMethod()\n    {\n        // Your code to test\n        Thread.Sleep(10);\n    }\n}\n```\n\n## Algorithm Comparisons\nEvery `[SailfishMethod]` on a `[Sailfish]` class is automatically compared against its siblings — no extra attributes required. Pick a baseline with `IsBaseline = true` for an N−1 baseline-vs-contender report; without one you get the full N×N matrix. Both modes report ratios with 95% confidence intervals and BH-FDR–adjusted q-values.\n\n```csharp\n[WriteToMarkdown]  // Generate consolidated markdown output\n[WriteToCsv]       // Generate consolidated CSV output\n[Sailfish(SampleSize = 100)]\npublic class SortBenchmarks\n{\n    [SailfishMethod(IsBaseline = true)]\n    public void QuickSort() { /* implementation */ }\n\n    [SailfishMethod]\n    public void BubbleSort() { /* implementation */ }\n\n    [SailfishMethod]\n    public void MergeSort() { /* implementation */ }\n}\n```\n\nIf a class isn't really doing a comparison (a smoke test that times unrelated operations), set `[Sailfish(DisableComparison = true)]` and the methods run individually with no comparison output. For classes that need multiple distinct comparisons in one place, set `ComparisonGroup = \"...\"` on individual `[SailfishMethod]`s.\n\nSee [Method Comparisons](https://paulgradie.com/Sailfish/docs/1/method-comparisons) for the full feature, including explicit `ComparisonGroup` for multi-group classes and the SF1300/1301/1302 build-time analyzers.\n\n## Adaptive Sampling\nAdaptive sampling stops collecting samples once results are statistically stable (defaults shown):\n- Coefficient of Variation (CV) ≤ 5%\n- Relative Confidence Interval width ≤ 20% at 95% confidence\n\nAttribute-based (per-class):\n```csharp\n[Sailfish(UseAdaptiveSampling = true, TargetCoefficientOfVariation = 0.05, MaximumSampleSize = 1000)]\npublic class StableTiming\n{\n    [SailfishMethod]\n    public async Task Work() =\u003e await Task.Delay(10);\n}\n```\n\nGlobal configuration (all tests in a run):\n```csharp\nvar runSettings = RunSettingsBuilder.CreateBuilder()\n    .WithGlobalAdaptiveSampling(targetCoefficientOfVariation: 0.05, maximumSampleSize: 500)\n    // other options like .WithGlobalSampleSize(...), .DisableOverheadEstimation(), etc.\n    .Build();\n```\n\nNote: Global settings act as defaults/overrides and can be combined with attribute settings per class.\n\n\n## Outlier Handling (defaults + opt‑in)\nBy default, Sailfish removes both upper and lower outliers (Tukey fences) to preserve historical behavior.\n\nTo opt into configurable outlier handling, set the flag on the `[Sailfish]` attribute (per class) or on `RunSettingsBuilder` (globally).\n\nPer class:\n\n````csharp\n[Sailfish(\n    UseConfigurableOutlierDetection = true,\n    OutlierStrategy = OutlierStrategy.RemoveUpper // or RemoveLower, RemoveAll, DontRemove, Adaptive\n)]\npublic class MyTest { }\n````\n\nGlobally (overrides per-class values):\n\n````csharp\nvar run = RunSettingsBuilder.CreateBuilder()\n    .WithGlobalOutlierHandling(useConfigurable: true, strategy: OutlierStrategy.RemoveUpper)\n    .Build();\n````\n\n- Legacy default (no behavior change): `UseConfigurableOutlierDetection = false` → `RemoveAll`\n- Strategies:\n  - `RemoveUpper`: remove only upper-fence outliers\n  - `RemoveLower`: remove only lower-fence outliers\n  - `RemoveAll`: remove both sides (legacy behavior)\n  - `DontRemove`: keep all data points, still reporting detected outliers\n  - `Adaptive`: choose based on which side(s) have detected outliers\n\n## Reproducible Run Order (Seed)\nSet a seed to make run order deterministic across test classes, methods, and variable sets:\n\n````csharp\nvar run = RunSettingsBuilder.CreateBuilder()\n    .WithSeed(12345) // deterministic ordering across classes, methods, and variable sets\n    .Build();\n````\n\n- Legacy fallback: `.WithArg(\"seed\", \"12345\")` is still honored if `Seed` is null\n- The seed is surfaced in the Markdown header and in the Reproducibility Manifest\n\n## Installation\n\nRequires **.NET 9** or **.NET 10**.\n\nInstall via NuGet:\n\n```bash\n# For a class-library / programmatic runner\ndotnet add package Sailfish\n\n# For tests that should be discovered by Visual Studio / Rider Test Explorer\ndotnet add package Sailfish.TestAdapter\n```\n\n## IDE setup\n\n### Visual Studio\nWorks out of the box. Reference `Sailfish.TestAdapter` (NuGet or `ProjectReference`) and Test Explorer will discover tests after a build.\n\n### JetBrains Rider\nRider needs a one-time setting flipped so it engages VSTest discovery for projects that use Sailfish (or any third-party VSTest adapter without a built-in ReSharper provider).\n\n1. `Settings → Build, Execution, Deployment → Unit Testing → VSTest`\n2. Confirm **Enable VSTest adapters support** is on.\n3. Under **Projects with unit tests**, click `+` and add a file mask. Use `*` to opt every project in the solution in (recommended), or a specific name like `*PerformanceTests*` to narrow.\n4. Save → `Build → Rebuild Solution`.\n\nSailfish tests then appear in the Unit Tests tool window with gutter play-buttons, and parameterized variants nest under their parent `SailfishMethod` (via the `TestCase.Hierarchy` properties the adapter emits).\n\n**Why the mask is needed**: ReSharper has built-in providers for xUnit / NUnit / MSTest only. For other VSTest adapters, it relies on the project being explicitly opted in to VSTest discovery via this mask — without it, ReSharper treats the project as non-test and greys out the run command. See issue [#98](https://github.com/paulegradie/Sailfish/issues/98) for background.\n\n## Outputs and Reporting\n- Per-group method comparisons (N−1 baseline mode or full N×N matrix)\n- BH-FDR–adjusted q-values and 95% ratio confidence intervals\n- Consolidated Markdown and CSV outputs\n- Improved / Slower / Similar labels at α = 0.05\n\nSee the full documentation for output details and examples: https://paulgradie.com/Sailfish/\n\n## Used By\n\u003cp\u003e\n  \u003cimg src=\"./assets/OctopusDeploy-logo-RGB.svg\" alt=\"Octopus Deploy\" width=\"320\" /\u003e\n\u003c/p\u003e\n\u003cp\u003e\n  \u003cimg src=\"./assets/empower.svg\" alt=\"Empower\" width=\"320\" /\u003e\n\u003c/p\u003e\n\n## Contributing\nContributions are welcome! Feel free to open issues and pull requests.\n\n## License\nSailfish is released under the [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulegradie%2Fsailfish","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulegradie%2Fsailfish","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulegradie%2Fsailfish/lists"}