https://github.com/intersectmbo/uplc-cape
Comparative Artifact Performance Evaluation
https://github.com/intersectmbo/uplc-cape
Last synced: 5 months ago
JSON representation
Comparative Artifact Performance Evaluation
- Host: GitHub
- URL: https://github.com/intersectmbo/uplc-cape
- Owner: IntersectMBO
- License: apache-2.0
- Created: 2025-07-16T10:01:54.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-09-12T16:42:37.000Z (6 months ago)
- Last Synced: 2025-09-12T18:53:14.205Z (6 months ago)
- Language: Untyped Plutus Core
- Size: 1.97 MB
- Stars: 2
- Watchers: 0
- Forks: 3
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# UPLC-CAPE
_**C**omparative **A**rtifact **P**erformance **E**valuation for UPLC programs_
A framework for measuring and comparing UPLC programs generated by different Cardano smart contract compilers.
[](LICENSE) [](CONTRIBUTING.md)
---
## Table of Contents
- [Overview](#overview)
- [Quick Start](#quick-start)
- [Prerequisites](#prerequisites)
- [Setup](#setup)
- [Your first benchmark](#your-first-benchmark)
- [Live Performance Reports](#live-performance-reports)
- [Available benchmark scenarios](#available-benchmark-scenarios)
- [Usage (CLI)](#usage-cli)
- [Core commands](#core-commands)
- [Creating a Submission](#creating-a-submission)
- [Metrics Explained](#metrics-explained)
- [Project Structure](#project-structure)
- [Resources](#resources)
- [Version and Tooling Requirements](#version-and-tooling-requirements)
- [Development](#development)
- [Documentation (ADRs)](#documentation-adrs)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgments](#acknowledgments)
---
## Overview
UPLC-CAPE provides a structured, reproducible way for Cardano UPLC compilers authors and users to:
- Benchmark compiler UPLC output against standardized scenarios
- Compare results across compilers and versions
- Track optimization progress over time
- Share results with the community
Key properties:
- Consistent benchmarks and metrics (CPU units, memory units, script size, term size)
- Reproducible results with versioned scenarios and metadata
- Automation-ready structure for future tooling
---
## Quick Start
### Prerequisites
- Nix with flakes enabled
- Git
### Setup
```zsh
# Clone and enter repository
git clone https://github.com/IntersectMBO/UPLC-CAPE.git
cd UPLC-CAPE
# Enter development environment
nix develop
# Or, if using direnv (recommended)
direnv allow
# Verify CLI
scripts/cape.sh --help
# Or use the cape shim if available in PATH
cape --help
```
### Your first benchmark
```zsh
# List available benchmarks
cape benchmark list
# View a specific benchmark
cape benchmark fibonacci
cape benchmark two-party-escrow
# Generate JSON statistics for all benchmarks
cape benchmark stats
# Create a submission for your compiler
cape submission new fibonacci MyCompiler 1.0.0 myhandle
cape submission new two-party-escrow MyCompiler 1.0.0 myhandle
```
---
## Live Performance Reports
Latest benchmark reports: [UPLC-CAPE Reports](https://intersectmbo.github.io/UPLC-CAPE/)
---
## Available benchmark scenarios
| Benchmark | Type | Description | Status |
| --- | --- | --- | --- |
| [Fibonacci](scenarios/fibonacci.md) | Synthetic | Recursive algorithm performance | Ready |
| [Factorial](scenarios/factorial.md) | Synthetic | Recursive algorithm performance | Ready |
| [Two-Party Escrow](scenarios/two-party-escrow.md) | Real-world | Smart contract escrow validator | Ready |
| Streaming Payments | Real-world | Payment channel implementation | Planned |
| Simple DAO Voting | Real-world | Governance mechanism | Planned |
| Time-locked Staking | Real-world | Staking protocol | Planned |
---
## Usage (CLI)
For the full and up-to-date command reference, see [USAGE.md](USAGE.md).
### Core commands
```zsh
# Benchmarks
cape benchmark list # List all benchmarks
cape benchmark # Show benchmark details
cape benchmark stats # Generate JSON statistics for all benchmarks
cape benchmark new # Create a new benchmark from template
# Submissions
cape submission list # List all submissions
cape submission list # List submissions for a benchmark
cape submission new
cape submission verify # Verify correctness and validate schemas
cape submission measure # Measure UPLC performance
cape submission aggregate # Generate CSV performance report
cape submission report # Generate HTML report for a benchmark
cape submission report --all # Generate HTML reports for all benchmarks
```
### JSON Statistics
The `cape benchmark stats` command generates comprehensive JSON data for all benchmarks:
```zsh
# Output JSON statistics to console
cape benchmark stats
# Save to file
cape benchmark stats > stats.json
# Use with jq for filtering
cape benchmark stats | jq '.benchmarks[] | select(.submission_count > 0)'
```
The output includes formatted metrics, best value indicators, and submission metadata, making it ideal for generating custom reports or integrating with external tools.
---
## Creating a Submission
1. Choose a benchmark
```zsh
cape benchmark list
cape benchmark fibonacci
```
1. Create submission structure
```zsh
cape submission new fibonacci MyCompiler 1.0.0 myhandle
# → submissions/fibonacci/MyCompiler_1.0.0_myhandle/
```
1. Add your UPLC program
- Replace the placeholder UPLC with your fully-applied program (no parameters).
- Path:
- submissions/fibonacci/MyCompiler_1.0.0_myhandle/fibonacci.uplc
- The program should compute the scenario’s required result deterministically within budget.
1. Verify and measure
Use the unified verification command to ensure your submission is correct and schema-compliant, then measure performance.
- Verify correctness and JSON schemas (all submissions or a path):
```zsh
cape submission verify submissions/fibonacci/MyCompiler_1.0.0_myhandle
# or, verify everything
cape submission verify --all
```
- Measure and write metrics.json automatically:
- Measure all .uplc files under a path (e.g., your submission directory):
```zsh
cape submission measure submissions/fibonacci/MyCompiler_1.0.0_myhandle
# or, from inside the submission directory
cape submission measure .
```
- Measure every submission under submissions/:
```zsh
cape submission measure --all
```
- What verification does:
- Evaluates your UPLC program; if it reduces to BuiltinUnit, correctness passes
- Otherwise, runs the comprehensive test suite defined in `scenarios/{benchmark}/cape-tests.json`
- Validates your `metrics.json` and `metadata.json` against schemas
- What measure does automatically:
- Measures CPU units, memory units, script size, and term size for your .uplc file(s)
- Generates or updates a `metrics.json` with scenario, measurements, evaluator, and timestamp
- Keeps your existing `notes` and `version` if present; otherwise fills sensible defaults
- Works for a single file, a directory, or all submissions with `--all`
- Produces output that validates against `submissions/TEMPLATE/metrics.schema.json`
- **Aggregation Strategies**: The `measure` tool now runs multiple test cases per program and provides several aggregation methods for CPU and memory metrics:
- `maximum`: Peak resource usage across all test cases (useful for identifying worst-case performance)
- `sum`: Total computational work across all test cases (useful for overall efficiency comparison)
- `minimum`: Best-case resource usage (useful for identifying optimal performance)
- `median`: Typical resource usage (useful for understanding normal performance)
- `sum_positive`: Total resources for successful test cases only (valid execution cost)
- `sum_negative`: Total resources for failed test cases only (error handling cost)
Higher-level tooling can extract the most relevant aggregation for specific analysis needs.
- Resulting file example:
```json
{
"scenario": "fibonacci",
"version": "1.0.0",
"measurements": {
"cpu_units": {
"maximum": 185916,
"sum": 185916,
"minimum": 185916,
"median": 185916,
"sum_positive": 185916,
"sum_negative": 0
},
"memory_units": {
"maximum": 592,
"sum": 592,
"minimum": 592,
"median": 592,
"sum_positive": 592,
"sum_negative": 0
},
"script_size_bytes": 1234,
"term_size": 45
},
"evaluations": [
{
"name": "fibonacci_25_computation",
"description": "Pre-applied fibonacci(25) should return 75025",
"cpu_units": 185916,
"memory_units": 592,
"execution_result": "success"
}
],
"execution_environment": {
"evaluator": "plutus-core-executable-1.52.0.0"
},
"timestamp": "2025-01-15T00:00:00Z",
"notes": "Optional notes."
}
```
1. Provide metadata
Create `metadata.json` according to `submissions/TEMPLATE/metadata.schema.json` (see also `metadata-template.json`).
```json
{
"compiler": {
"name": "MyCompiler",
"version": "1.0.0"
},
"compilation_config": {
"target": "uplc",
"optimization_level": "O2",
"flags": []
},
"contributor": {
"name": "myhandle"
},
"submission": {
"date": "2025-01-15T00:00:00Z",
"source_available": false,
"implementation_notes": "Brief explanation of approach."
}
}
```
1. Document
- Add notes to README.md inside your submission folder (implementation choices, optimizations, caveats).
---
## Metrics Explained
| Metric | Description | Measurement |
| ------------ | ----------------------------------- | ------------------ |
| CPU Units | Computational cost for execution | CEK machine steps |
| Memory Units | Memory consumption during execution | CEK machine memory |
| Script Size | Size of serialized UPLC script | Bytes |
| Term Size | Size of the UPLC term | AST nodes |
---
## Project Structure
```text
UPLC-CAPE/
├── scenarios/ # Benchmark specifications
│ ├── TEMPLATE/ # Template for new scenarios
│ ├── fibonacci.md
│ ├── factorial.md
│ └── two-party-escrow.md
├── submissions/ # Compiler submissions (per scenario)
│ ├── TEMPLATE/ # Templates and schemas
│ │ ├── metadata.schema.json
│ │ ├── metadata-template.json
│ │ ├── metrics.schema.json
│ │ └── metrics-template.json
│ ├── fibonacci/
│ │ └── MyCompiler_1.0.0_handle/
│ └── two-party-escrow/
│ └── MyCompiler_1.0.0_handle/
├── scripts/ # Project CLI tooling
│ ├── cape.sh # Main CLI
│ └── cape-subcommands/ # Command implementations
├── lib/ # Haskell library code (validators, fixtures, utilities)
├── measure-app/ # UPLC program measurement tool
├── plinth-submissions-app/ # Plinth submission generator
├── test/ # Test suites
├── report/ # Generated HTML reports and assets
├── doc/ # Documentation
│ ├── domain-model.md
│ └── adr/
└── README.md
```
---
## Resources
- [Log4brains](https://github.com/thomvaill/log4brains)
- [Architecture Decision Records](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions)
---
## Version and Tooling Requirements
- Development environment: Nix shell (`nix develop`) with optional direnv (`direnv allow`).
- GHC: 9.6.6 (provided in Nix shell).
- Plutus Core target: 1.1.0.
- Use `plcVersion110` (for Haskell/PlutusTx code).
- Package baselines (CHaP):
- plutus-core >= 1.52.0.0
- plutus-tx >= 1.52.0.0
- plutus-ledger-api >= 1.52.0.0
- plutus-tx-plugin >= 1.52.0.0
---
## Development
Enter environment:
```zsh
nix develop
# or
direnv allow
```
Common tools:
- cape … (project CLI)
- cabal build (builds all Haskell components: library, executables, tests)
- treefmt (format all files)
- fourmolu (Haskell formatting)
- adr (Architecture Decision Records)
- mmdc -i file.mmd (diagram generation, if available)
---
## Documentation (ADRs)
ADRs document important design decisions (managed with Log4brains).
Helpful commands:
```zsh
adr new "Decision Title"
adr preview
adr build
adr help
```
---
## Contributing
We welcome contributions from compiler authors, benchmark designers, and researchers.
- Add a new benchmark:
```zsh
cape benchmark new my-new-benchmark
# edit scenarios/my-new-benchmark.md
```
- Add a submission:
```zsh
cape submission new existing-benchmark MyCompiler 1.0.0 myhandle
# fill uplc and json files, then open a PR
```
Please read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a PR.
---
## License
Licensed under the Apache License 2.0. See [LICENSE](LICENSE).
---
## Acknowledgments
- Plutus Core team for infrastructure and reference implementations
- Compiler authors and community contributors