{"id":16716270,"url":"https://github.com/rayluo/perf_baseline","last_synced_at":"2026-06-04T22:30:53.855Z","repository":{"id":192890837,"uuid":"680618485","full_name":"rayluo/perf_baseline","owner":"rayluo","description":"  The perf_baseline is a performance regression detection tool for Python projects. It uses timeit to automatically time your function, records the result as a baseline into a file, and compares subsequent test results against the initial baseline to detect performance regression based on your specified threshold.","archived":false,"fork":false,"pushed_at":"2024-05-13T07:04:47.000Z","size":11,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-23T07:18:10.609Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rayluo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-19T20:35:29.000Z","updated_at":"2024-05-13T07:04:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"fcb76884-5d8d-4f1c-88b3-8d66ebb7487d","html_url":"https://github.com/rayluo/perf_baseline","commit_stats":null,"previous_names":["rayluo/perf_baseline"],"tags_count":1,"template":false,"template_full_name":"rayluo/python-project-template","purl":"pkg:github/rayluo/perf_baseline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayluo%2Fperf_baseline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayluo%2Fperf_baseline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayluo%2Fperf_baseline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayluo%2Fperf_baseline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rayluo","download_url":"https://codeload.github.com/rayluo/perf_baseline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayluo%2Fperf_baseline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33923173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T21:12:36.799Z","updated_at":"2026-06-04T22:30:53.840Z","avatar_url":"https://github.com/rayluo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Perf Baseline\n\nThe ``perf_baseline`` is a performance regression detection tool for Python projects.\nIt uses ``timeit`` to automatically time your function,\nrecords the result as a baseline into a file,\nand compares subsequent test results against the initial baseline\nto detect performance regression based on your specified threshold.\n(We do not compare against the second-last performance test result,\ntherefore your performance won't suffer from gradual decline.)\n\n\n## Installation\n\n`pip install perf_baseline`\n\n## Usage 1: Benchmark functions with parameters\n\nLet's say your project contains some important functions like this:\n\n```python\ndef add(a, b):\n    return a + b\n\ndef sub(a, b):\n    return a - b\n```\n\nYou can guard against potential performance regression by:\n\n```python\nfrom perf_baseline import Baseline\n\nbaseline = Baseline(\n    \"my_baseline.bin\",  # Performance test result will be saved into this file\n    threshold=1.8,  # Subsequent tests will raise exception if a new test is more than 1.8x slower than the baseline\n)\n\ndef test_add_should_not_have_regression():\n    baseline.set_or_compare(\"add(2, 3)\", name=\"my addition implementation\", globals={\"add\": add})\n\ndef test_sub_should_not_have_regression():\n    baseline.set_or_compare(\"sub(5, 4)\", globals={\"sub\": sub})\n    # Note: When absent, name defaults to the current test function's name,\n    #       which is \"test_sub_should_not_have_regression\" in this case\n```\n\nThat is it.\n\nNow you can run ``pytest`` to test it, or ``pytest --log-cli-level INFO`` to see some logs.\nThe test case will pass if there is no performance regression, or raise ``RegressionError`` otherwise.\n\nUnder the hood, ``perf_baseline`` stores the *initial* test results into the file you specified.\nEach file may contain multiple data points, differentiated by their unique names.\nIf the ``name`` parameter is omitted, the current test case's name will be used instead.\n\nSubsequent tests will automatically be compared with the initial baseline,\nand error out when performance regression is detected.\n\nIf you want to reset the baseline, simply delete that baseline file and then start afresh.\n\n\n## Usage 2: Test a parameter-less callable\n\nIn real world projects, your test subject may require nontrivial setup,\nsuch as prepopulating some test data,\nand those initialization time shall be excluded from benchmark.\nHow do we achieve that?\n\nA common solution is creating a wrapper class,\nwhich has an expensive constructor and a (typically parameter-less) action method.\nAnd then you can have ``Baseline`` to check that action method.\nFor example:\n\n```python\nclass TestDriver:\n    def __init__(self, size):\n        self.data = list(range(size))\n        import random\n        random.shuffle(self.data)\n    def run(self):\n        self.data.sort()  # Some implementation to sort the self.data in-place\n\nbaseline = Baseline(\"my_baseline.bin\", threshold=2.0)\n\ndef test_my_sort_implementation():\n    driver = TestDriver(1000*1000)\n    baseline.set_or_compare(\n        driver.run,  # A parameter-less callable can be tested as-is, without setting globals\n    )\n\n    # Alternatively, you may also combine the above two lines into this one-liner\n    baseline.set_or_compare(TestDriver(1000*1000).run)  # The driver initialization is still done only once\n```\n\n\n## Do NOT commit the baseline file into Git\n\nAdd your baseline filename into your ``.gitignore``, so that you won't accidentally commit it.\n\n```\nmy_baseline.bin\n```\n\nWhy?\n\nThe idea is that a performance baseline (such as 123456 ops/sec) is\nonly meaningful and consistent when running on the *same* computer.\nSwitching to a different computer, it will have a different baseline.\n\nBy not committing a baseline into the source code repo,\neach maintainer of your project (and each of their computers)\nwill have their own baseline created by the first run.\n\nThis way, you won't need to use a large threshold across different computers\n(it is impossible to specify a constant threshold that works on different computers anyway).\nPer-computer baselines all self-calibrate to match the performance of each computer.\n\n\n## How to run this in Github Action?\n\n``perf_baseline`` relies on an *updatable* baseline file,\nwhich shall be writable when a new data point (represented by a new ``name``) occurs,\nand remain read-only when an old data point (represented by same ``name``) already exists.\n\nAs of this writing, [Github's Cache Action](https://github.com/marketplace/actions/cache)\nsupports the updatable usage via some hack, inspired by\n[this hint](https://github.com/actions/toolkit/issues/505#issuecomment-1650290249).\nUse the following snippet, and modify its ``path`` and ``hashFiles(...)`` to match your filenames.\n\n```yaml\n    - name: Setup an updatable cache for Performance Baselines\n      uses: actions/cache@v3\n      with:\n        path: my_baseline.bin\n        key: ${{ runner.os }}-performance-${{ hashFiles('tests/test_benchmark.py') }}\n        restore-keys: ${{ runner.os }}-performance-\n    - name: Now you can run test cases powered by perf_baseline\n      ...\n```\n\n\n## How to choose an appropriate threshold?\n\nThe performance of an implementation always fluctuates a little.\nYou need an appropriate threshold to detect performance regression.\nAn excessive threshold will not catch a performance regression.\nAn inadequate threshold will yield many false positives.\nYou shall run some preliminary tests to see the normal range of fluctuation.\n\n1. Start with a bigger threshold, such as 10 (which means tolerating 10x slower runs).\n2. Run your ``perf-baseline``-powered test cases multiple times,\n   with logs enabled (i.e. ``pytest --log-cli-level INFO``),\n   and focus on the lines with ``Actual/Baseline = ..../.... = 1.234 (VS threshold 10)``,\n   take notes on that ratio number (1.234 in the example above).\n3. After you have a bunch of ratio samples, uses the highest/lowest plus a sensible margin as the threshold.\n\n\u003c!--\nTODO: Shall perf_baseline automate this process in next version, so that the users do not have to?\nA threshold can probably be detected in one run, by examining its stdev.\nNote that a threshold auto-detected on one machine may not be suitable for another,\nso, human decision is probably still necessary to choose a bigger threshold to sustain fluctuation.\n--\u003e\n\nFYI, benchmarks running on bare metal are usually consistent within 10% to 20% (i.e. threshold=1.2).\nThe agents/runners of Github Actions seem to fluctuate a lot, possibly caused by\n[\"noisy neighbor\" effect](https://en.wikipedia.org/wiki/Cloud_computing_issues#Performance_interference_and_noisy_neighbors).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayluo%2Fperf_baseline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frayluo%2Fperf_baseline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayluo%2Fperf_baseline/lists"}