https://github.com/block/francis
A Kotlin-based CLI for A/B performance testing on Android
https://github.com/block/francis
Last synced: 2 months ago
JSON representation
A Kotlin-based CLI for A/B performance testing on Android
- Host: GitHub
- URL: https://github.com/block/francis
- Owner: block
- License: apache-2.0
- Created: 2026-02-06T18:23:42.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-03-06T23:44:21.000Z (3 months ago)
- Last Synced: 2026-03-30T05:26:30.684Z (2 months ago)
- Language: Kotlin
- Size: 227 KB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Agents: AGENTS.md
Awesome Lists containing this project
README
# francis
> "Observation and experiment for gathering material, induction and deduction for elaborating it: these are our only good intellectual tools."
*- Francis Bacon (1561-1626), English philosopher and statesman, regarded as the father of empiricism*
A CLI for rigorous A/B performance testing on Android.
## Why Francis?
Francis wraps
[Android Macrobenchmark](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview),
providing a simple CLI that:
- installs APKs, configures instrumentation args, runs benchmarks, and
retrieves results
- monitors logcat and parses instrumentation output to provide actionable error
messages
Francis also provides commands to
- run [A/B tests](https://en.wikipedia.org/wiki/A/B_testing) of macrobenchmarks and
determine whether differences are statistically meaningful.
- collect simpleperf/perfetto traces
## Usage
```sh
# See list of supported commands:
francis --help
# See detailed docs for a specific command (e.g. bench):
francis bench --help
# Collect macrobenchmark results
francis bench --app app.apk --instrumentation benchmark.apk --test-symbol 'com.example.Example#benchmarkMethod'
# Collect results of an A/B test of a macrobenchmark
francis ab --instrumentation benchmark.apk --test-symbol 'com.example.Example#benchmarkMethod' \
--baseline-opts baseline-version-of-app.apk \
--treatment-opts treatment-version-of-app.apk
# Compare two sets of macrobenchmark results (e.g. result of `ab` command above)
francis compare baseline-results.json treatment-results.json
# Collect a manual perfetto trace (no instrumentation)
francis perfetto
# Collect a manual simpleperf trace (no instrumentation)
francis simpleperf
# Collect a perfetto trace of an instrumentation scenario
francis perfetto --app app.apk --instrumentation benchmark.apk --test-symbol 'com.example.Example#benchmarkMethod'
# Collect a simpleperf trace of an instrumentation scenario
francis simpleperf --app app.apk --instrumentation benchmark.apk --test-symbol 'com.example.Example#benchmarkMethod'
```
## Installation
```sh
brew install block/tap/francis
```
## Integrating with the Instrumentation SDK
Basic benchmark and A/B test usage works without the instrumentation SDK, but some features (modifying iteration count, perfetto/simpleperf trace collection of instrumentation scenarios) require modifying your instrumentation apk. Assuming you created your macrobenchmark in the style of [the official docs](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview#create-macrobenchmark), you'll make the following changes:
1. Add a dependency on `com.squareup.francis:instrumentation-sdk`. In your instrumentation apk's `build.gradle` (Groovy syntax):
```groovy
dependencies {
androidTestImplementation "com.squareup.francis:instrumentation-sdk"
}
```
or `build.gradle.kts` (Kotlin syntax):
```kotlin
dependencies {
androidTestImplementation("com.squareup.francis:instrumentation-sdk")
}
```
2. Replace `MacrobenchmarkRule` with `FrancisBenchmarkRule`:
```diff
- import androidx.benchmark.macro.junit4.MacrobenchmarkRule
+ import com.squareup.francis.FrancisBenchmarkRule
...
@get:Rule
- val benchmarkRule = MacrobenchmarkRule()
+ val benchmarkRule = FrancisBenchmarkRule()
```
This allows Francis to hook into [measureRepeated](https://developer.android.com/reference/kotlin/androidx/benchmark/macro/junit4/MacrobenchmarkRule#measureRepeated(kotlin.String,kotlin.collections.List,androidx.benchmark.macro.CompilationMode,androidx.benchmark.macro.StartupMode,kotlin.Int,kotlin.Function1,kotlin.Function1)) invocations and modify arguments to, e.g:
- change iteration count
- surround the measureBlock parameter with code to start/stop perfetto/simpleperf in order to capture traces
3. Optionally use `@Disable` (from the instrumentation SDK) instead of `@Ignore`:
```kotlin
import com.squareup.francis.Disable
@Disable("TRACKER-123")
@Test
fun expensiveBenchmark() {
// ...
}
```
`@Disable` skips the test by default, but Francis automatically sets `francis.overrideDisable`
from `--symbol`. This way you disable a test in CI, but still run it manually with Francis.
If `@Disable` is on a class, target that class (`com.example.BenchmarkClass`) to override it.
If `@Disable` is on a method, target that method
(`com.example.BenchmarkClass#expensiveBenchmark`) to override it.
## Development
You can use `scripts/francis` to build and run francis during development. If you don't have a specific app/instrumentation that you want to test it with, you can use `scripts/francis-demo` - it's the same as `scripts/francis` but it includes predefined app/instrumentation apks.