{"id":13723979,"url":"https://github.com/knqyf263/cob","last_synced_at":"2025-05-16T03:05:39.129Z","repository":{"id":48563592,"uuid":"233287637","full_name":"knqyf263/cob","owner":"knqyf263","description":"Continuous Benchmark for Go Project","archived":false,"fork":false,"pushed_at":"2024-10-14T06:22:48.000Z","size":281,"stargazers_count":384,"open_issues_count":8,"forks_count":26,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-10-14T20:36:15.650Z","etag":null,"topics":["benchmark","ci","go","golang","test","testing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/knqyf263.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-11T19:43:56.000Z","updated_at":"2024-10-14T06:22:07.000Z","dependencies_parsed_at":"2023-10-20T21:02:27.474Z","dependency_job_id":"6757a89a-e440-4bbf-a44b-bbd206adbf04","html_url":"https://github.com/knqyf263/cob","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fcob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fcob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fcob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fcob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/knqyf263","download_url":"https://codeload.github.com/knqyf263/cob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254459088,"owners_count":22074605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","ci","go","golang","test","testing"],"created_at":"2024-08-03T01:01:47.959Z","updated_at":"2025-05-16T03:05:34.115Z","avatar_url":"https://github.com/knqyf263.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003cimg src=\"img/logo.png\" width=\"350\"\u003e\n\n[![GitHub release](https://img.shields.io/github/release/knqyf263/cob.svg)](https://github.com/knqyf263/cob/releases/latest)\n![](https://github.com/knqyf263/cob/workflows/Go/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/knqyf263/cob)](https://goreportcard.com/report/github.com/knqyf263/cob)\n[![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat)](https://github.com/knqyf263/cob/blob/master/LICENSE)\n\n\n\n# Abstract\n`cob` compares benchmarks between the latest commit (HEAD) and the previous commit (HEAD{@1}). The program will fail if the change in score is worse than the threshold. This tools is suitable for CI/CD to detect a regression of a performance automatically.\n\n\u003cimg src=\"img/usage.png\" width=\"700\"\u003e\n\n`cob` runs `go test -bench` before and after commit internally, so it depends on `go` command.\n\n**CAUTION: Note that `git reset` is executed when you run `cob`. You should commit all changes before running `cob`.**\n\n# Table of Contents\n\u003c!-- TOC --\u003e\n- [Abstract](#abstract)\n- [Continuous Integration (CI)](#continuous-integration-ci)\n  - [GitHub Actions](#github-actions)\n  - [Travis CI](#travis-ci)\n  - [CircleCI](#circleci)\n- [Example](#example)\n  - [Run only those benchmarks matching a regular expression](#run-only-those-benchmarks-matching-a-regular-expression)\n  - [Show only benchmarks with worse score](#show-only-benchmarks-with-worse-score)\n  - [Specify a threshold](#specify-a-threshold)\n  - [Specify a base commit compared with HEAD](#specify-a-base-commit-compared-with-head)\n  - [Compare only memory allocation](#compare-only-memory-allocation)\n- [Usage](#usage)\n- [Q\u0026A](#qa)\n  - [A result of benchmarks is unstable](#a-result-of-benchmarks-is-unstable)\n\n# Continuous Integration (CI)\n\nSee [cob-example](https://github.com/knqyf263/cob-example) for details.\n\n## GitHub Actions\n\n```\nname: Bench\non: [push, pull_request]\njobs:\n  test:\n    name: Bench\n    runs-on: ubuntu-latest\n    steps:\n\n    - name: Set up Go 1.13\n      uses: actions/setup-go@v1\n      with:\n        go-version: 1.13\n      id: go\n\n    - name: Check out code into the Go module directory\n      uses: actions/checkout@v1\n\n    - name: Install cob\n      run: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin\n\n    - name: Run Benchmark\n      run: cob\n```\n\n## Travis CI\n\n```\ndist: bionic\nlanguage: go\ngo:\n  - 1.13.x\n\nbefore_script:\n  - curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin\n\nscript:\n  - cob\n```\n\n## CircleCI\n\n```\nversion: 2\njobs:\n  bench:\n    docker:\n      - image: circleci/golang:1.13\n    steps:\n      - checkout\n      - run:\n          name: Install cob\n          command: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin\n      - run:\n          name: Run cob\n          command: cob\nworkflows:\n  version: 2\n  build-workflow:\n    jobs:\n      - bench\n```\n\n\n# Example\n\n## Override a command to measure benchmarks\n\nTo measure benchmarks by `make bench`, you can use `-bench-cmd` and `-bench-args` options.\n\n```\n$ cob -bench-cmd make -bench-args bench\n```\n\n## Run only those benchmarks matching a regular expression\n\n```\n$ cob -bench-args \"test -bench Append -benchmem ./...\"\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eResult\u003c/summary\u003e\n\n```\n2020/01/12 17:32:30 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}\n2020/01/12 17:32:32 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD\n\nResult\n======\n\n+-----------------------------+----------+---------------+-------------------+\n|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |\n+-----------------------------+----------+---------------+-------------------+\n| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      117 B/op     |\n+                             +----------+---------------+-------------------+\n|                             | HEAD@{1} |  115.00 ns/op |      23 B/op      |\n+-----------------------------+----------+---------------+-------------------+\n\nComparison\n==========\n\n+-----------------------------+---------+-------------------+\n|            Name             | NsPerOp | AllocedBytesPerOp |\n+-----------------------------+---------+-------------------+\n| BenchmarkAppend_Allocate-16 | 55.65%  |      408.70%      |\n+-----------------------------+---------+-------------------+\n```\n\n\u003c/details\u003e\n\n## Show only benchmarks with worse score\n\n```\n$ cob -only-degression\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eResult\u003c/summary\u003e\n\n```\n2020/01/12 17:48:35 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}\n2020/01/12 17:48:38 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD\n\nComparison\n==========\n\n+-----------------------------+---------+-------------------+\n|            Name             | NsPerOp | AllocedBytesPerOp |\n+-----------------------------+---------+-------------------+\n| BenchmarkAppend_Allocate-16 | 52.34%  |      347.83%      |\n+-----------------------------+---------+-------------------+\n\n2020/01/12 17:48:39 This commit makes benchmarks worse\n```\n\n\u003c/details\u003e\n\n## Specify a threshold\n\nThe following option means the program fails if a benchmark score gets worse than 50%.\n\n```\n$ cob -threshold 0.5 ./...\n```\n\n## Specify a base commit compared with HEAD\nBy default, `cob` uses `HEAD~1`. If you compare benchmarks with different commit, you can use `--base` option.\n\n```\n$ cob --base origin/master ./...\n```\n\n## Compare only memory allocation\nYou can use `-compare` option.\n\n```\n$ cob -compare B/op\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eResult\u003c/summary\u003e\n\n```\n2020/01/15 14:46:31 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD~1\n2020/01/15 14:46:33 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD\n\nResult\n======\n\n+-----------------------------+----------+---------------+-------------------+\n|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |\n+-----------------------------+----------+---------------+-------------------+\n| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      121 B/op     |\n+                             +----------+---------------+-------------------+\n|                             | HEAD@{1} |  104.00 ns/op |      23 B/op      |\n+-----------------------------+----------+---------------+-------------------+\n|      BenchmarkCall-16       |   HEAD   |   0.50 ns/op  |       0 B/op      |\n+                             +----------+---------------+                   +\n|                             | HEAD@{1} |   0.49 ns/op  |                   |\n+-----------------------------+----------+---------------+-------------------+\n\nComparison\n==========\n\n+-----------------------------+---------+-------------------+\n|            Name             | NsPerOp | AllocedBytesPerOp |\n+-----------------------------+---------+-------------------+\n| BenchmarkAppend_Allocate-16 |    -    |      426.09%      |\n+-----------------------------+---------+-------------------+\n|      BenchmarkCall-16       |    -    |       0.00%       |\n+-----------------------------+---------+-------------------+\n\n2020/01/15 14:46:35 This commit makes benchmarks worse\n```\n\n\u003c/details\u003e\n\n## Skip running cob\nIf your commit message contains `[skip cob]`, cob is skipped.\n\n```\n$ git add README.md\n$ git commit -m \"[skip cob] update README.md\"\n$ cob\n2020/04/19 12:46:57 [skip cob] is detected, so the benchmark is skipped\n```\n\n\n# Usage\n\n```\nNAME:\n   cob - Continuous Benchmark for Go project\n\nUSAGE:\n   cob [global options] command [command options] [arguments...]\n\nCOMMANDS:\n   help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --only-degression   Show only benchmarks with worse score (default: false)\n   --threshold value   The program fails if the benchmark gets worse than the threshold (default: 0.2)\n   --base value        Specify a base commit compared with HEAD (default: \"HEAD~1\")\n   --compare value     Which score to compare (default: \"ns/op,B/op\")\n   --bench-cmd value   Specify a command to measure benchmarks (default: \"go\")\n   --bench-args value  Specify arguments passed to -cmd (default: \"test -run '^$' -bench . -benchmem ./...\")\n   --help, -h          show help (default: false)\n```\n\n# Q\u0026A\n\n## Benchmarks with the same name\n\nSpecify a package name.\n\n```\n$ cob -bench-args \"test -bench . -benchmem ./foo\" \n$ cob -bench-args \"test -bench . -benchmem ./bar\" \n```\n\n## A result of benchmarks is unstable\n\nYou can specify `-benchtime`.\n\n```\n$ cob -bench-args \"test -bench . -benchmem -benchtime 10s ./...\" \n```\n\n# License\n\nThis repository is available under the [MIT](https://github.com/knqyf263/cob/blob/master/LICENSE)\n\n# Author\n\n[Teppei Fukuda](https://github.com/knqyf263) (knqyf263)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknqyf263%2Fcob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknqyf263%2Fcob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknqyf263%2Fcob/lists"}