{"id":18776581,"url":"https://github.com/onsi/ginkgo_perf","last_synced_at":"2025-12-15T16:30:14.258Z","repository":{"id":139821806,"uuid":"257679556","full_name":"onsi/ginkgo_perf","owner":"onsi","description":"A repo exploring Ginkgo's performance vs go test","archived":false,"fork":false,"pushed_at":"2020-04-21T20:59:35.000Z","size":357,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-08T17:40:57.123Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onsi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-21T18:20:20.000Z","updated_at":"2024-06-19T12:27:43.188Z","dependencies_parsed_at":null,"dependency_job_id":"881456af-f498-4162-9ca3-51b12d65b661","html_url":"https://github.com/onsi/ginkgo_perf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fginkgo_perf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fginkgo_perf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fginkgo_perf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onsi%2Fginkgo_perf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onsi","download_url":"https://codeload.github.com/onsi/ginkgo_perf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239684291,"owners_count":19680097,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T19:46:46.855Z","updated_at":"2025-12-15T16:30:12.193Z","avatar_url":"https://github.com/onsi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ginkgo-perf\n\nTrying to get to the bottom of `ginkgo` vs `go test` performance.\n\n## Usage\n\n`go run .` will run a few different scenarios and save the test run timings in `test_runs`.  If a scenario is already saved in `test_runs` it is skipped.\n`go run . force` will forcibly rerun all the scenarios regardless of what is stored in `test_runs`\n\nEither command spits out a summary of the various scenarios.\n\n`python visualize.py` generates a plot\n\n## Finding (To Date)\n\nThere are four scenarios, each runs the tests stored in `sample`:\n\n- `go test`: simply runs `go test sample`\n- `go test compile`: mimics Ginkgo's behavior of compiling the test first (`go test -c -output=\"test\"`) then running the test (`./test`).\n- `ginkgo_cli_base`: runs the default Ginkgo cli, which first compiles the test and stores it in a temp directory then runs the test.\n- `ginkgo_cli_no_tmp`: runs a modified Ginkgo cli, which first compiles the test and stores it in the package directory then runs the test.\n\n### Run-time Statistics\n\nFor each scenario (where possible) we capture the Compile Time, Run Time, and Total Time.  The following are stats from a 100-iteration run on my macbook pro:\n\n| Scenario | Metric | Min | Max | Mean | Median | Std Dev |\n| --- | --- | --- | --- | --- | --- | --- |\n| `go test` | Run Time | 0.9271 | 1.0830 | 0.9998s | 0.9999s | 0.0186s |\n| `go test` | Total Time | 0.9271 | 1.0830 | 0.9998s | 0.9999s | 0.0186s |\n| `go test compile` | Compile Time | 0.8978s | 1.1267 | 0.9642s | 0.9564s | 0.0421s |\n| `go test compile` | Run Time | 0.1264s | 0.2085 | 0.1353s | 0.1333s | 0.0105s |\n| `go test compile` | Total Time | 1.0339s | 1.2637 | 1.1007s | 1.0903s | 0.0438s |\n| `ginkgo_cli_base` | Compile Time | 0.8865s | 1.0358 | 0.9345s | 0.9306s | 0.0283s |\n| `ginkgo_cli_base` | Run Time | 0.1752s | 0.9746 | 0.4585s | 0.1866s | 0.3380s |\n| `ginkgo_cli_base` | Total Time | 1.0782s | 1.9955 | 1.4081s | 1.1674s | 0.3380s |\n| `ginkgo_cli_no_tmp` | Compile Time | 0.9008s | 1.1703 | 0.9794s | 0.9646s | 0.0568s |\n| `ginkgo_cli_no_tmp` | Run Time | 0.1262s | 0.1754 | 0.1370s | 0.1339s | 0.0104s |\n| `ginkgo_cli_no_tmp` | Total Time | 1.0468s | 1.3352 | 1.1308s | 1.1161s | 0.0636s |\n\n\nAnd here's the plot:\n\n![timings plot](https://github.com/onsi/ginkgo_perf/blob/master/timings.png)\n\n### What it means:\n\n- `go test` is 100ms faster than `go test compile`\n- `ginkgo_cli_no_tmp` is equivalent to `go test compile`\n- `ginkgo_cli_base` - i.e. the default ginkgo behavior that compiles the test into a tmp directory - sees notable latency in the Run Time - of about 500ms on average with a wide variance.\n\nSo.  Apparently.  Don't try to run binaries from a temp directory (at least on MacOS) as that introduces latency with a long-tail distribution (see the plot).\n\n### Applying this to the Abacus issue\n\nThis performance exploration started - in part - due to [this issue](https://github.com/onsi/ginkgo/issues/644).  On my laptop vanilla ginkgo takes ~4.1s but the no-temp-dir flavor of ginkgo takes ~3.3s.\n\n### Applying this to `ginkgo`\n\nRunning ginkgo's unit tests shows a reduction in run time from ~16.3s to ~12.7s.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonsi%2Fginkgo_perf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonsi%2Fginkgo_perf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonsi%2Fginkgo_perf/lists"}