{"id":22822469,"url":"https://github.com/switcherapi/feature-flag-benchmark","last_synced_at":"2025-03-30T23:41:38.466Z","repository":{"id":103554754,"uuid":"519084831","full_name":"switcherapi/feature-flag-benchmark","owner":"switcherapi","description":"Evaluate performance of 5 different Feature Flag SDKs","archived":false,"fork":false,"pushed_at":"2024-12-13T01:38:00.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-02-06T05:14:41.107Z","etag":null,"topics":["feature-flags","feature-toggle","switcher-api"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/switcherapi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-29T04:56:43.000Z","updated_at":"2024-12-13T01:38:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"a387b0de-cbf7-48d1-9002-1945ce7a8be9","html_url":"https://github.com/switcherapi/feature-flag-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/switcherapi%2Ffeature-flag-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/switcherapi%2Ffeature-flag-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/switcherapi%2Ffeature-flag-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/switcherapi%2Ffeature-flag-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/switcherapi","download_url":"https://codeload.github.com/switcherapi/feature-flag-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246395573,"owners_count":20770240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["feature-flags","feature-toggle","switcher-api"],"created_at":"2024-12-12T16:12:03.967Z","updated_at":"2025-03-30T23:41:38.449Z","avatar_url":"https://github.com/switcherapi.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\nThis benchmark compares 5 major Feature Flag SDKs against Switcher Client SDK for Java.\u003cbr\u003e\nThe tests included in this benchmark are focused only in performance and does not reflect nor measure any other SDK capabilities.\u003cbr\u003e\nThe intent is to gain more knowledge and define a baseline to improve Switcher Client SDK overall performance.\n\n## Run\nBuild jar and run the benchmark\n```\nmvn clean verify\njava -jar target/benchmarks.jar\n```\n\n## Tests explained\n\n### [Async] testSwitcherRemoteThrottle, testHarness and testOptimizely\nUses in-memory cache by preloading or using async calls to remote service.\u003cbr\u003e\n\n### [Local] testSwitcherLocal, testSplit and testUnleash\nIn-memory configuration loaded from file.\u003cbr\u003e\n\u003cp\u003e* Split SDK does not provide free access to fetch data from remote service - in this test the SDK is returning a default value.\u003c/p\u003e\n\n### [Remote] testSwitcherRemote\nThese tests are 100% remote.\u003cbr\u003e\n\n## Operations /s in 5s\n| Benchmark                                       | Type   | Mode  | Score           | Units |\n|:------------------------------------------------|:-------|:------|:----------------|:------|\n| Simple Boolean (static feature flag)            | Local  | thrpt | 585,493,539.045 | ops/s |\n| FeatureFlagBenchmark.testSwitcherRemoteThrottle | Async  | thrpt | 34,327,818.480  | ops/s |\n| FeatureFlagBenchmark.testSwitcherLocal          | Local  | thrpt | 17,861,430.894  | ops/s |\n| FeatureFlagBenchmark.testHarness                | Async  | thrpt | 5,993,383.681   | ops/s |\n| FeatureFlagBenchmark.testSplit                  | Local  | thrpt | 5,176,741.297   | ops/s |\n| FeatureFlagBenchmark.testUnleash                | Local  | thrpt | 2,674,586.707   | ops/s |\n| FeatureFlagBenchmark.testOptimizely             | Async  | thrpt | 480,875.500     | ops/s |\n| FeatureFlagBenchmark.testTogglz                 | Local  | thrpt | 419,103.750     | ops/s |\n| FeatureFlagBenchmark.testSwitcherRemote         | Remote | thrpt | 187.247         | ops/s |\n\n### Switcher Client SDK:\n - when Async, can be up to 6x faster than the other SDKs.\n - when Local, can be up to 3x faster than the other SDKs.\n - 100% remote can only be enabled with Switcher Client SDK.\n\n## Average calls/ns in 5s\n| Benchmark                                       | Mode | Score         | Units |\n|:------------------------------------------------|:-----|:--------------|:------|\n| FeatureFlagBenchmark.testSwitcherRemoteThrottle | avgt | 31,661        | ns/op |\n| FeatureFlagBenchmark.testSwitcherLocal          | avgt | 63.852        | ns/op |\n| FeatureFlagBenchmark.testHarness                | avgt | 360.610       | ns/op |\n| FeatureFlagBenchmark.testSplit                  | avgt | 430.571       | ns/op |\n| FeatureFlagBenchmark.testUnleash                | avgt | 800.930       | ns/op |\n| FeatureFlagBenchmark.testOptimizely             | avgt | 3,844.586     | ns/op |\n| FeatureFlagBenchmark.testTogglz                 | avgt | 4,568.257     | ns/op |\n| FeatureFlagBenchmark.testSwitcherRemote         | avgt | 5,699,296.355 | ns/op |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswitcherapi%2Ffeature-flag-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswitcherapi%2Ffeature-flag-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswitcherapi%2Ffeature-flag-benchmark/lists"}