{"id":26784854,"url":"https://github.com/sagar-viradiya/auto-benchmark","last_synced_at":"2025-04-19T17:06:47.976Z","repository":{"id":189422095,"uuid":"667014303","full_name":"sagar-viradiya/auto-benchmark","owner":"sagar-viradiya","description":"A Gradle plugin to automate macro-benchmark on baseline profile.","archived":false,"fork":false,"pushed_at":"2024-01-02T20:48:19.000Z","size":316,"stargazers_count":63,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-01-03T11:38:35.230Z","etag":null,"topics":["android","baseline-profiles","macro-benchmark","performance"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sagar-viradiya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-07-16T11:02:30.000Z","updated_at":"2024-01-03T11:38:35.231Z","dependencies_parsed_at":"2024-01-02T11:34:12.467Z","dependency_job_id":"a0c28964-ec68-44f9-a62b-fcd3052ed455","html_url":"https://github.com/sagar-viradiya/auto-benchmark","commit_stats":null,"previous_names":["sagar-viradiya/auto-benchmark"],"tags_count":2,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar-viradiya%2Fauto-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar-viradiya%2Fauto-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar-viradiya%2Fauto-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sagar-viradiya%2Fauto-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sagar-viradiya","download_url":"https://codeload.github.com/sagar-viradiya/auto-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246174178,"owners_count":20735406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","baseline-profiles","macro-benchmark","performance"],"created_at":"2025-03-29T10:32:43.613Z","updated_at":"2025-03-29T10:33:10.137Z","avatar_url":"https://github.com/sagar-viradiya.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Autobenchmark\n[![Gradle Plugin](https://img.shields.io/gradle-plugin-portal/v/io.github.sagar-viradiya.autobenchmark?color=%233cafc6\u0026label=Plugin\u0026logo=gradle\u0026style=flat-square)](https://plugins.gradle.org/plugin/io.github.sagar-viradiya.autobenchmark)\n\nA Gradle plugin to automate macro-benchmark on baseline profile. \nRun your macro-benchmark tests for profile verification on Firbase test lab and verify benchmark result JSON.\nWith this plugin it is possible to integrate baseline profile verification in your CI pipeline to fully automate baseline profile. \n\nFor generating profile you can leverage the official [baseline profile gradle plugin](https://developer.android.com/topic/performance/baselineprofiles/create-baselineprofile#baseline-profile-gradle-plugin). This plugin closes the loop of automating baseline profile end to end by providing automation on profile verification.\n\nFor more information on benchmarking baseline profile visit official [Android guide](https://developer.android.com/topic/performance/baselineprofiles/measure-baselineprofile).\n\n## Applying plugin\n\n### Using plugin DSL\nApply plugin to app module `build.gradle.kts` file.\n```kotlin\nplugins {\n  id(\"io.github.sagar-viradiya.autobenchmark\") version \"1.0.0.alpha02\"\n}\n```\n\n### Using legacy plugin application\n\nAdd this to top project level `build.gradle.kts`\n```kotlin\nbuildscript {\n  repositories {\n    maven {\n      url = uri(\"https://plugins.gradle.org/m2/\")\n    }\n  }\n  dependencies {\n    classpath(\"io.github.sagar-viradiya:autobenchmark:1.0.0.alpha02\")\n  }\n}\n```\nAnd then apply plugin to app module `build.gradle.kts` file.\n\n```kotlin\napply(plugin = \"io.github.sagar-viradiya.autobenchmark\")\n```\n\n## ⚙️ Configuring plugin\n\nSince it is recommended to run macro-benchmark on a physical device this plugin supports running tests on Firebase test lab through [Fladle](https://runningcode.github.io/fladle/). \nFollowing are the mandatory parameters that you need to configure.\n\n```kotlin\nautoBenchmark {\n    // A relative file path from the root to app apk with baseline profile\n    appApkFilePath.set(\"/sample/build/outputs/apk/benchmark/sample-benchmark.apk\")\n    // A relative file path from the root to benchmark apk\n    benchmarkApkFilePath.set(\"/benchmark/build/outputs/apk/benchmark/benchmark-benchmark.apk\")\n    // Service account JSON file path to authenticate GCloud\n    serviceAccountJsonFilePath.set(\"../../.config/gcloud/application_default_credentials.json\")\n    // Physical device configuration map to run benchmark\n    physicalDevices.set(mapOf(\n        \"model\" to \"redfin\", \"version\" to \"30\"\n    ))\n    // Tolerance percentage for improvement below which verification will fail\n    tolerancePercentage.set(10f)\n}\n```\n\n### 🔐 Authenticate G-Cloud to run tests on Firebase test lab\n\nTo run tests on Firebase test lab and download JSON result of profile verification, you need to authenticate to G-Cloud.\n\n#### Authenticating on local machine\n\n1. Run `./gradlew flankAuth`\n2. Sign in to web browser.\n\nThis will store credentials in ~/.flank directory\n\n#### Authenticating on CI\n\nYou will need service account JSON file to setup authentication on CI. Follow the [test lab docs](https://firebase.google.com/docs/test-lab/android/continuous) to create a service account.\nBase64 encode this file on your local machine and set this as environment variable on CI as GCLOUD_KEY. \n\n```shell\nbase64 -i \"$HOME/.config/gcloud/application_default_credentials.json\" | pbcopy\n```\n\nThen on CI decode the JSON.\n\n```shell\nGCLOUD_DIR=\"$HOME/.config/gcloud/\"\nmkdir -p \"$GCLOUD_DIR\"\necho \"$GCLOUD_KEY\" | base64 --decode \u003e \"$GCLOUD_DIR/application_default_credentials.json\"\n```\nPlease refer to GitHub action setup to know how this is being done.\n\nFor more info refer [this](https://flank.github.io/flank/#authenticate-with-a-service-account) flank guide for authentication as this plugin internally uses Fladle and Flank\n\n## 📈 Running benchmark tests and baseline profile verification\n\nTo run benchmark and verify result run following gradle task.\n\n```shell\n./gradlew [your_app]:runBenchmarkAndVerifyProfile\n```\n\nFirst, this will run all benchmark tests on Firebase test lab, and it will download benchmark result JSON from G-Cloud.\nThis JSON file will be analysed next to compare 'No compilation median startup time' with 'baseline profile median startup time'.\nIf the improvement percentage is below provided `tolerancePercentage` then gradle task will fail.\n\nFor example, if no compilation median startup is 233 ms and baseline profile median startup\nis 206 ms then startup time is improved by ~11%. If the provided tolerance percentage is \n10 then task will successfully complete. \n\nTask will also print improvement percentage, no compilation startup time median, and baseline profile startup time median.\n\n```shell\n\u003e Task :sample:runBenchmarkAndVerifyProfile\nNo compilation median : 233.609398\nBaseline profile median : 206.635229\nImprovement percentage : 11.546700\n\nBUILD SUCCESSFUL in 2m 35s\n9 actionable tasks: 3 executed, 6 up-to-date\n```\n\n\u003e Note : One important requirement for benchmark result JSON parsing to work properly \n\u003e is to have 'NoCompilation' in the test name targeting no compilation mode, and 'BaselineProfile' in the test name targeting baseline profile mode.\n\n## Sample\nPlease refer to sample app to see the plugin setup. Before you issue `./gradlew :sample:runBenchmarkAndVerifyProfile` you need to, \nbuild sample apk and benchmark apk first.\n\nRun following commands to build apks\n\n```shell\n./gradlew :sample:assembleBenchmark\n```\n\n```shell\n./gradlew :benchmark:assembleBenchmark\n```\n\nAlso make sure to have service account JSON file in the configured path (`serviceAccountJsonFilePath`)\n\nFor verifying profile on CI, please refer to GitHub action setup.\n\n## Contribution\nUnfortunately it is not ready to accept any contribution\nyet. Once this is stable enough contribution guidelines will be updated here. Meanwhile, feel free to open an [issue](https://github.com/sagar-viradiya/auto-benchmark/issues) for feature requests and improvements.\n\n## License\n\n    MIT License\n    \n    Copyright (c) 2023 Autobenchmark Contributors\n    \n    Permission is hereby granted, free of charge, to any person obtaining a copy \n    of this software and associated documentation files (the \"Software\"), to deal\n    in the Software without restriction, including without limitation the rights\n    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n    copies of the Software, and to permit persons to whom the Software is\n    furnished to do so, subject to the following conditions:\n    \n    The above copyright notice and this permission notice shall be included in all\n    copies or substantial portions of the Software.\n    \n    THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n    SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsagar-viradiya%2Fauto-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsagar-viradiya%2Fauto-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsagar-viradiya%2Fauto-benchmark/lists"}