{"id":15010701,"url":"https://github.com/jcolechanged/josh.benchmarking","last_synced_at":"2026-03-13T20:12:58.490Z","repository":{"id":49354037,"uuid":"514995140","full_name":"jColeChanged/josh.benchmarking","owner":"jColeChanged","description":"A lein plugin to help library authors accrete benchmarking datasets.","archived":false,"fork":false,"pushed_at":"2022-11-25T06:45:43.000Z","size":50,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-01-26T14:05:15.053Z","etag":null,"topics":["benchmarking","clojure","leiningen","plugin"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jColeChanged.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-18T01:13:41.000Z","updated_at":"2022-07-18T04:44:58.000Z","dependencies_parsed_at":"2023-01-23T10:01:02.804Z","dependency_job_id":null,"html_url":"https://github.com/jColeChanged/josh.benchmarking","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jColeChanged%2Fjosh.benchmarking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jColeChanged%2Fjosh.benchmarking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jColeChanged%2Fjosh.benchmarking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jColeChanged%2Fjosh.benchmarking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jColeChanged","download_url":"https://codeload.github.com/jColeChanged/josh.benchmarking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243291159,"owners_count":20267855,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","clojure","leiningen","plugin"],"created_at":"2024-09-24T19:35:24.986Z","updated_at":"2025-12-24T20:42:50.826Z","avatar_url":"https://github.com/jColeChanged.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# josh.benchmarking\r\n\r\nThis is a lein plugin which persists benchmarking results generated by `criterium` whenever `lein benchmark` is run.\r\nBy default it emits messages which explain the benchmark results with historical context via CLI messages.\r\nBy using [josh.benchmaking.visualization](https://github.com/jcolechanged/josh.benchmarking.visualization) as a supplement\r\nmuch richer understanding of historical performance can be achieved.\r\n\r\nEarly alpha software. Anyone using this should be aware that dramatic changes to API are likely.\r\n\r\nA library author ought to know the impact that their changes have on their library's performance, different libraries which tackle the same problem ought to be able to share common benchmarks, and the community at large ought to have insight into the performance of each release so its members can understand whether the philosophy of a library with regard to performance aligns with their own performance needs.\r\n\r\nInstead benchmarks are a thing of niche GitHub repos, contentious debate in blog posts, and very often can be found nestled inside `(comment)` blocks. There is a lot of wonderful things to say about REPL-driven development, for example, it gives fast feedback. However, ad-hoc solutions where you have to expend effort to generate value aren't ideal. It would be better to automate benchmarking such that we produce [technical income][ti].\r\n\r\nThis is a library to help library authors do that automation. It is not itself a benchmarking tool, but is scaffolding for keeping track of historical benchmarking runs. Instead of running `criterium` directly benchmark functions are setup like so:\r\n\r\n```clojure\r\n(ns benchmarks.library.core\r\n  (:require [josh.benchmarking.core :refer [defbenchmark]]))\r\n\r\n;; Perhaps if I have benchmarks proving my library is the \r\n;; fastest someone will think this is worth using.\r\n\r\n;; Competiing library used 1100 for sleeping benchmark.\r\n;; Very important that we use the same number as them \r\n;; to ensure comparison across libraries.\r\n(def shared-sleep-length 1100)\r\n(def sneaky-sneaky (- shared-sleep-length 100))\r\n\r\n(def ^{:is-benchmark true} \r\n  fast-sleeping-benchmark\r\n  (fn [] (Thread/sleep sneaky-sneaky)))\r\n```\r\n\r\nNow, with your obviously fair benchmark - you wouldn't try to be \r\nmore popular than you *deserve* now would you - you run your benchmarks:\r\n\r\n```\r\nlein benchmark\r\n```\r\n\r\nThe benchmark will run and as part of running the benchmarks the results will be \r\nshoved into a dataset compatible with the `tech.ml.dataset` API. If no configuration \r\nis provided then a human readable file format will be used instead.\r\n\r\nOn future runs the datasets of past runs will be used to give insight into performance \r\nby showing local relative performance - how things have changed since the last few commits.\r\n\r\nUnlike `criterium` which throws away the results we keep the results around. This means we \r\ncan offer some useful features - like visualizing the benchmark results.\r\n\r\nThese results are emitted as events which you can react to either individually or at the end of all benchmarks running. \r\nThese event handlers are your opportunity to inject your own idea of what should happen in response to performance changes into your library - for example, by failing a CI/CD pipeline if there is a notable regression.\r\n\r\nI encourage you to make read access to your performance dataset publicly available via a link on your GitHub page. I also encourage you to adopt benchmarks that you fail - for example, many libraries fail miserably when presented with more data than can fit in memory, but others don't. Surfacing these capabilities via benchmarks can prove \r\nyour claim to operate at the largest scales.\r\n\r\n## Usage\r\n\r\nFIXME: Use this for user-level plugins:\r\n\r\nPut `[josh.benchmarking \"0.1.0-SNAPSHOT\"]` into the `:plugins` vector of your `:user`\r\nprofile.\r\n\r\nFIXME: Use this for project-level plugins:\r\n\r\nPut `[josh.benchmarking \"0.1.0-SNAPSHOT\"]` into the `:plugins` vector of your project.clj.\r\n\r\nFIXME: and add an example usage that actually makes sense:\r\n\r\n    $ lein benchmark\r\n\r\n## License\r\n\r\nCopyright © 2022 Joshua Cole\r\n\r\nThis program and the accompanying materials are made available under the\r\nterms of the Eclipse Public License 2.0 which is available at\r\nhttp://www.eclipse.org/legal/epl-2.0.\r\n\r\nThis Source Code may also be made available under the following Secondary\r\nLicenses when the conditions for such availability set forth in the Eclipse\r\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\r\nthe Free Software Foundation, either version 2 of the License, or (at your\r\noption) any later version, with the GNU Classpath Exception which is available\r\nat https://www.gnu.org/software/classpath/license.html.\r\n\r\n# References \r\n\r\n[ti]: https://joshuacol.es/2020/03/06/modeling-technical-income.html\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcolechanged%2Fjosh.benchmarking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcolechanged%2Fjosh.benchmarking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcolechanged%2Fjosh.benchmarking/lists"}