https://github.com/russfellows/sai3-bench
A multi-protocol storage performance testing tool, inspired by vdbench, fio and warp. Part of the SAI3 project. Leverages the s3dlio Rust library
https://github.com/russfellows/sai3-bench
azure-blob benchmarking google-cloud-storage object-storage performance rust-lang s3 sai3 storage testing testing-tools
Last synced: 2 months ago
JSON representation
A multi-protocol storage performance testing tool, inspired by vdbench, fio and warp. Part of the SAI3 project. Leverages the s3dlio Rust library
- Host: GitHub
- URL: https://github.com/russfellows/sai3-bench
- Owner: russfellows
- License: gpl-3.0
- Created: 2025-07-25T20:07:51.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2026-04-23T21:54:49.000Z (2 months ago)
- Last Synced: 2026-04-23T23:33:52.591Z (2 months ago)
- Topics: azure-blob, benchmarking, google-cloud-storage, object-storage, performance, rust-lang, s3, sai3, storage, testing, testing-tools
- Language: Rust
- Homepage:
- Size: 4.01 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# sai3-bench: Multi-Protocol I/O Benchmarking Suite
[](https://github.com/russfellows/sai3-bench/releases)
[](https://github.com/russfellows/sai3-bench)
[](https://github.com/russfellows/sai3-bench)
[](LICENSE)
[](https://www.rust-lang.org/)
**🚀 NEW (v0.8.96)**: **Multi-endpoint S3 routing fixed + `S3_ENDPOINT_URIS` env var support** — All operation types (GET, PUT, LIST, STAT, DELETE) now correctly route through `MultiEndpointStore` when a `multi_endpoint:` block is present; previously only PUT was routed, causing GETs and LISTs to fall back to a single endpoint. A new `S3_ENDPOINT_URIS` environment variable (comma-separated URIs) lets you enable multi-endpoint load balancing at runtime without editing YAML — YAML config always takes precedence. Validation output clearly shows which endpoint source is active. Updated to s3dlio v0.9.96. +1 test (713 total). See [tests/configs/MULTI_ENDPOINT_README.md](tests/configs/MULTI_ENDPOINT_README.md) for usage.
**🚀 NEW (v0.8.94)**: **jemalloc global allocator + s3dlio v0.9.92** — Replaced the default glibc allocator with [tikv-jemallocator](https://crates.io/crates/tikv-jemallocator) v0.6 (`#[global_allocator]`), eliminating glibc arena contention and fragmentation. Profiling showed `malloc_consolidate` at ~3% and the allocator frame at ~52% of CPU cycles under load; jemalloc removes both bottlenecks. Measured improvement: **+3.6% throughput at t=32** (32,634 → 33,812 ops/s). Updated s3dlio dependency from v0.9.90 → **v0.9.92** (pinned tag).
**🚀 NEW (v0.8.92)**: **Credential forwarding + HTTP/2 + pre-flight improvements** — Controller now forwards cloud credentials (`AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`, `AZURE_STORAGE_*`) to agents over gRPC via `--env-file ` or from its own environment (disable with `--no-forward-env`). Agents apply credentials before pre-flight, eliminating the manual step of copying secrets to each host. See [docs/CREDENTIAL_FORWARDING.md](docs/CREDENTIAL_FORWARDING.md). s3dlio v0.9.90 adds **HTTP/2 (h2c) support** for S3-protocol `http://` endpoints: auto-probes h2c on first connection and falls back to HTTP/1.1 if refused; `https://` endpoints negotiate via TLS ALPN transparently. Enable via `S3DLIO_H2C=1` or `s3dlio_optimization.h2c: true` in YAML. Pre-flight fixes: per-agent endpoint filtering (fixes 64-error agent-2 regression), bucket-grouped output, actionable `[PERM]`/`[AUTH]`/`[CONF]`/`[NET]` error classification, agent version check table, and two new config validation warnings (redundant multi_endpoint, missing credentials hint). +57 tests (712 total).
**🚀 NEW (v0.8.90)**: **`populate_ledger.tsv` + dgen-data zero-copy fills** — Every prepare phase now writes a lightweight `populate_ledger.tsv` (always-on, independent of the KV cache) recording object counts, bytes, and throughput — usable even at trillion-object scale where listing is infeasible. Data generation rewritten using [`dgen-data`](https://crates.io/crates/dgen-data) v0.2.3: a rolling-pointer pool generates one 1 MB buffer and vends zero-copy `Bytes::slice()` windows per PUT, eliminating per-object allocation. PUT latency now split into **setup** vs. **I/O** histograms for better profiling.
**🚀 NEW (v0.8.89)**: **`enable_metadata_cache` config option** — disables the internal Fjall KV metadata cache for very large or simple workloads (> ~1 B objects/batch). Default `true` (fully backward-compatible). Set `false` to eliminate ~3.4 GB disk usage per 50 M objects and the ~15 s resume scan, at the cost of crash-resume capability. Both standalone and distributed dry-runs print a clear banner showing the current setting. Reference config: [`tests/configs/test_prepare_no_kvcache.yaml`](tests/configs/test_prepare_no_kvcache.yaml).
**🚀 NEW (v0.8.88)**: **KV cache compact encoding + coverage observability** — KV cache entries now use [postcard](https://crates.io/crates/postcard) binary encoding (56% smaller, 2× faster scans). At startup, a one-line cache summary reports object count and total logical storage (`📊 Cache summary: N objects | X.XX GiB`). Progressive WARN messages fire if a coverage scan exceeds 10 s. Preflight now queries the cache per-spec and logs coverage. Safe write-probe cycle validates writable endpoints before any benchmark I/O. Agent port changed to **7167** (was 7761) with automatic port-conflict detection on startup. +17 new tests.
**🚀 NEW (v0.8.86)**: **GCS RAPID storage fully working** (s3dlio v0.9.86) — `BidiWriteObject` PUTs and `BidiReadObject` GETs verified against Hyperdisk ML RAPID buckets. RAPID mode is auto-detected per bucket or forced via `gcs_rapid_mode: true`. Worker drain deadline bug fixed (execute stage now runs its full configured duration). Timer observability logs added.
**🚀 NEW (v0.8.70)**: **GCS RAPID gRPC support** (s3dlio v0.9.70) — per-trial channel count, range-download control, and write-chunk-size control sent over RPC to agents. **Autotune redesign** — all tuning parameters are YAML-only; new `channels_per_thread` parameter scales gRPC subchannels with thread count; `--dry-run` prints the full sweep plan (computed sizes, loop order, total cases, I/O estimate) before executing.
**🚀 NEW (v0.8.63)**: **Multi-endpoint checkpoint race condition fix** - Eliminates fatal workload aborts at 99% completion for shared storage. **s3dlio optimization support** - +76% GET throughput for large objects (≥64MB).
**🚀 NEW (v0.8.62)**: **Streaming prepare + dry-run memory sampling + stage-aligned perf-log timing** for safer large-scale runs.
**🚀 NEW (v0.8.61)**: **Explicit distributed stages + numeric barrier indices** for consistent orchestration across multi-agent runs. Use the new `convert` command to upgrade legacy YAML files.
**🚀 NEW (v0.8.60)**: **KV cache checkpoint restoration** - Complete resume capability! Checkpoints now automatically restored on startup, enabling agents to resume long-running prepare operations after crashes/restarts. Works for both standalone and distributed modes.
**🚀 (v0.8.53)**: **Critical multi-endpoint + directory tree fix** - GET/PUT/STAT/DELETE operations now correctly route to round-robin endpoints, fixing 0-ops workload failures. Enhanced dry-run shows ALL endpoints with full URIs.
**🚀 (v0.8.52)**: **Deferred retry for prepare failures** eliminates "missing object" errors during execution. Failed creates are automatically retried after the main loop with aggressive exponential backoff (10 attempts, up to 30s delay), ensuring completeness without impacting fast path performance. **Thousand separator display** in dry-run (64,032,768 files) and optional YAML input support ("64,032,768"). **Human-readable time units** in YAML: use "5m", "2h", "30s" instead of seconds (300, 7200, 30).
**🚀 NEW (v0.8.51)**: **Critical blocking I/O fixes** for large-scale deployments (>100K files). Configurable `agent_ready_timeout` (default 120s), non-blocking glob operations, and periodic yielding in prepare loops prevent executor starvation. [See docs/CHANGELOG.md](docs/CHANGELOG.md) for details.
**🚀 NEW (v0.8.50)**: **YAML-driven stage orchestration** with 6 stage types, **barrier synchronization** for coordinated multi-host testing, and **comprehensive timeout configuration** (global/stage/barrier hierarchy).
**🚀 NEW (v0.8.23)**: Pre-flight distributed configuration validation prevents common misconfigurations (base_uri with isolated mode) before execution.
**🚀 NEW (v0.8.22)**: Multi-endpoint load balancing with per-agent static endpoint mapping for shared storage systems with multiple endpoints (NFS, S3, or object storage).
A comprehensive storage performance testing tool supporting multiple backends through a unified interface. Built on the [s3dlio Rust library](https://github.com/russfellows/s3dlio) for multi-protocol support.
## 🚀 What Makes sai3-bench Unique?
1. **Universal Storage Testing**: Unified interface across 5 storage protocols (file://, direct://, s3://, az://, gs://)
2. **Directory Tree Workloads**: Configurable hierarchical structures for realistic shared filesystem testing
3. **Filesystem Operations**: Full support for nested paths and directory operations across all backends
4. **Pre-flight Validation**: Detect configuration errors before execution (filesystem access, distributed config mismatches)
5. **Workload Replay**: Capture production traffic and replay with microsecond fidelity (1→1, 1→N, N→1 remapping)
6. **Op-Log Management**: Sort, validate, and merge operation logs for analysis and replay
7. **Robust Distributed Execution**: Bidirectional streaming with sub-millisecond agent synchronization (v0.8.5+)
8. **Production-Grade Metrics**: HDR histograms with size-bucketed analysis and aggregate summaries
9. **Realistic Data Patterns**: Lognormal size distributions, configurable deduplication and compression
10. **Machine-Readable Output**: TSV export with per-bucket and aggregate rows for automated analysis
11. **Performance Logging**: Time-series perf-log with 31 columns including mean/p50/p90/p99 latencies, CPU metrics, and warmup filtering (v0.8.17+)
12. **Results Analysis Tool**: Excel spreadsheet generation consolidating multiple test results (sai3-analyze, v0.8.17+)
13. **Automatic Credential Distribution**: Controller forwards cloud credentials to agents over gRPC so each host needs no manual secret setup — with an allow-list, local-wins policy, and audit logging (v0.8.92+)
## 🎯 Supported Storage Backends
All operations work identically across protocols - just change the URI scheme:
- **File System** (`file://`) - Local filesystem with standard POSIX operations
- **Direct I/O** (`direct://`) - High-performance direct I/O bypassing page cache (optimized chunked reads)
- **Amazon S3** (`s3://`) - S3 and S3-compatible storage
- **Azure Blob** (`az://`) - Microsoft Azure Blob Storage
- **Google Cloud Storage** (`gs://` or `gcs://`) - Google Cloud Storage with native GCS API, including RAPID (Hyperdisk ML) storage (v0.8.86+)
See [Cloud Storage Setup](docs/CLOUD_STORAGE_SETUP.md) for authentication guides.
## 🚀 Quick Start
### One-Time Setup
**1. Install Rust** (if not already installed):
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env
```
**2. Clone and Build sai3-bench**:
```bash
git clone https://github.com/russfellows/sai3-bench.git
cd sai3-bench
cargo build --release
```
The build creates 4 executables in `target/release/`:
- `sai3-bench` - Single-node testing CLI
- `sai3bench-agent` - Distributed agent (runs on each test host)
- `sai3bench-ctl` - Distributed controller (coordinates agents)
- `sai3-analyze` - Results analysis tool (Excel export)
**3. Install Executables** (optional):
Choose one of the following installation methods:
**Option A: User-local install** (recommended, no sudo required):
```bash
cargo install --path .
```
Installs to `~/.cargo/bin/` (already in your PATH from Rust installation).
**Option B: System-wide install**:
```bash
sudo install -m 755 target/release/{sai3-bench,sai3bench-ctl,sai3bench-agent,sai3-analyze} /usr/local/bin/
```
Installs to `/usr/local/bin/` for all users.
**Option C: Run from build directory**:
```bash
# No installation needed - use full path:
./target/release/sai3-bench --version
./target/release/sai3bench-ctl --version
```
### Testing Modes
sai3-bench supports two testing modes: **Single-Node** and **Distributed**.
```
┌───────────────────────────────────────────────────────────────────┐
│ SINGLE-NODE MODE │
├───────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ │
│ │ │ I/O Operations │
│ │ sai3-bench │ ─────────────────────► Storage System │
│ │ │ (S3/NFS/Azure/etc) │
│ └──────────────┘ │
│ │
│ • Simple: One command to run workloads │
│ • Use for: Single host testing, development, quick validation │
│ • Command: ./sai3-bench run --config workload.yaml │
│ │
└───────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ DISTRIBUTED MODE │
├──────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ │
│ │ │ gRPC: Config, Start/Stop, Stats │
│ │ sai3bench-ctl │────────┬──────────┬──────────┐ │
│ │ (Controller) │ │ │ │ │
│ └──────────────────┘ ▼ ▼ ▼ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │sai3bench- │ │sai3bench- │ │sai3bench- │ │
│ │agent (Host 1)│ │agent (Host 2)│ │agent (Host N)│ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ I/O │ I/O │ I/O │
│ ▼ ▼ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Storage System (NFS/S3/Azure/etc) │ │
│ │ • Multiple endpoints for load balancing │ │
│ │ • Unified namespace across all endpoints │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ • Scalable: Generate load from multiple hosts │
│ • Use for: Large-scale testing, multi-endpoint storage │
│ • Command: ./sai3bench-ctl run --config distributed.yaml │
│ │
└──────────────────────────────────────────────────────────────────┘
```
### Running Your First Workload
**Single-Node Mode** - Test local filesystem:
```bash
# Create a simple config file
cat > my-test.yaml < distributed-test.yaml < **Note**: sai3-bench can also capture op-logs during benchmark runs with `--op-log`, but this is primarily for analyzing benchmark I/O patterns rather than capturing production workloads.
### Backpressure Handling (v0.8.9+)
When target storage can't sustain the recorded I/O rate:
```yaml
# replay_config.yaml - controls replay behavior
lag_threshold: 5s # Switch to best-effort when lag exceeds this
recovery_threshold: 1s # Switch back when lag drops below this
max_flaps_per_minute: 3 # Exit gracefully if oscillating too much
max_concurrent: 1000 # Maximum in-flight operations
drain_timeout: 10s # Timeout for draining on exit
```
```bash
sai3-bench replay --op-log trace.tsv.zst --config replay_config.yaml --target "s3://bucket/"
```
### URI Remapping
Transform source URIs during replay for migration testing:
```yaml
# remap.yaml - 1:1 bucket rename (simple migration)
rules:
- match:
bucket: "prod-bucket"
map_to:
bucket: "staging-bucket"
prefix: "migrated/"
```
```bash
# Apply remapping during replay
sai3-bench replay --op-log trace.tsv.zst --remap remap.yaml --target "s3://staging-bucket/"
```
**Advanced Remapping Strategies:**
- **1→1**: Simple bucket/prefix rename (migration validation)
- **1→N**: Fanout to replicas (`round_robin` or `sticky_key` distribution)
- **N→1**: Consolidate multiple sources to single target
- **N→M**: Regex-based transformations (e.g., `s3://` → `gs://` for cross-cloud)
See [remap_examples.yaml](tests/configs/remap_examples.yaml) for complete examples.
**Use Cases**: Pre-migration validation, performance regression testing, capacity planning, cross-cloud comparison.
## 💾 Storage Efficiency Testing
Test deduplication and compression with controlled data patterns.
**Important**: `dedup_factor` and `compress_factor` are **optional** - if omitted, both default to `1` (no dedup, no compression).
### Example 1: Default Behavior (No Dedup/Compression)
```yaml
prepare:
ensure_objects:
- base_uri: "s3://bucket/unique-media/"
count: 500
size_spec: 10485760 # 10 MB fixed size
fill: random
# dedup_factor: 1 (default - omitted, all blocks unique)
# compress_factor: 1 (default - omitted, incompressible)
```
### Example 2: Testing Storage Deduplication (3:1 Ratio)
```yaml
prepare:
ensure_objects:
- base_uri: "s3://bucket/vm-snapshots/"
count: 100
size_spec: 52428800 # 50 MB
fill: random
dedup_factor: 3 # 3:1 dedup (1/3 blocks unique, 2/3 duplicates)
compress_factor: 1 # No compression (incompressible data)
```
**Result**: 100 files × 50 MB = 5 GB logical, but only ~1.67 GB unique data (3:1 dedup).
### Example 3: Combined Dedup + Compression (5:1 and 2:1)
```yaml
prepare:
ensure_objects:
- base_uri: "s3://bucket/log-archives/"
count: 200
size_spec:
type: uniform
min: 5242880 # 5 MB
max: 52428800 # 50 MB
fill: random
dedup_factor: 5 # 5:1 dedup (1/5 blocks unique)
compress_factor: 2 # 2:1 compression (50% zeros)
```
**Result**: Avg 28.5 MB × 200 files = 5.7 GB logical → ~1.14 GB unique (5:1) → ~570 MB after compression (2:1).
### Dedup/Compression Ratios Explained
| Setting | Value | Meaning | Storage Impact |
|---------|-------|---------|----------------|
| `dedup_factor: 1` | 1:1 (default) | All blocks unique | No dedup savings |
| `dedup_factor: 3` | 3:1 | 1/3 unique, 2/3 duplicate | 67% space savings |
| `dedup_factor: 5` | 5:1 | 1/5 unique, 4/5 duplicate | 80% space savings |
| `compress_factor: 1` | 1:1 (default) | Incompressible | No compression savings |
| `compress_factor: 2` | 2:1 | 50% zeros | 50% compression savings |
| `compress_factor: 4` | 4:1 | 75% zeros | 75% compression savings |
**Fill Pattern Guidelines:**
| Pattern | Speed | Use Case |
|---------|-------|----------|
| `random` | Standard | Production benchmarks, realistic workloads (RECOMMENDED) |
| ***⚠️ `zero`*** | Fastest | ***DO NOT USE - triggers dedup/compression, unrealistic results*** |
**Use Cases**: Validate vendor dedup/compression claims, predict migration space requirements, model hot vs. cold data.
See [Data Generation Guide](docs/DATA_GENERATION.md) for detailed patterns.
## 📐 Realistic Size Distributions
Model real-world object storage patterns with statistical distributions:
```yaml
workload:
- op: put
path: "data/"
weight: 100
size_spec:
type: lognormal # Many small files, few large files
mean: 1048576 # Mean: 1 MB
std_dev: 524288 # Std dev: 512 KB
min: 1024 # Floor: 1 KB
max: 10485760 # Ceiling: 10 MB
fill: random # Cryptographic random (recommended)
```
**Why lognormal?** Research shows object storage naturally follows lognormal distributions (many small configs/thumbnails, few large videos/backups).
**Distribution Types:**
- `lognormal`: Realistic - many small, few large (requires `mean`, `std_dev`)
- `uniform`: Even spread between `min` and `max`
- Fixed size: Just use `size_spec: 1048576` (integer value)
See [Config Syntax](docs/CONFIG_SYNTAX.md) for complete options.
## 🌐 Distributed Testing
Generate large-scale coordinated load across multiple nodes with automated deployment:
### Automated SSH Deployment
```bash
# One-time setup: Configure passwordless SSH
sai3bench-ctl ssh-setup --hosts ubuntu@vm1,ubuntu@vm2,ubuntu@vm3
# Run distributed test: Agents deploy automatically
sai3bench-ctl run --config distributed-workload.yaml
```
### Configuration-Driven Agents
Define all agents in YAML with per-agent customization:
```yaml
distributed:
agents:
- address: "vm1.example.com"
id: "us-west-agent"
target_override: "s3://us-west-bucket/"
concurrency_override: 128
env: { AWS_PROFILE: "benchmark" }
- address: "vm2.example.com"
id: "us-east-agent"
target_override: "s3://us-east-bucket/"
ssh:
enabled: true
key_path: "~/.ssh/sai3bench_id_rsa"
deployment:
container_runtime: "docker" # or "podman"
image: "sai3bench:latest"
network_mode: "host"
```
### Flexible Scaling Strategies
**Scale-Out** (Multiple VMs): Maximum network bandwidth, fault tolerance
```yaml
# 8 VMs, 1 container each = 8× network interfaces
agents:
- { address: "vm1:7167", id: "agent-1" }
- { address: "vm2:7167", id: "agent-2" }
# ... vm3-vm8
```
**Scale-Up** (Single VM): Cost optimization, lower latency
```yaml
# 1 large VM, 8 containers on different ports
agents:
- { address: "big-vm:7167", id: "c1", listen_port: 7167 }
- { address: "big-vm:7168", id: "c2", listen_port: 7168 }
# ... c3-c8
```
### Cloud Automation
Pre-built scripts for rapid deployment:
- **GCP**: `scripts/gcp_distributed_test.sh` - Complete VM lifecycle automation
- **AWS/Azure**: `scripts/cloud_test_template.sh` - Customizable templates
- **Local**: `scripts/local_docker_test.sh` - Test distributed mode without cloud
### Key Features
- **Automated lifecycle**: SSH, container deployment, health checks, cleanup
- **Per-agent overrides**: Target storage, concurrency, environment variables, volumes
- **Graceful shutdown**: Ctrl+C handling with automatic container cleanup
- **Result aggregation**: Proper HDR histogram merging for accurate percentiles
- **Container flexibility**: Docker or Podman via YAML (no recompilation)
**Learn More**:
- [Distributed Testing Guide](docs/DISTRIBUTED_TESTING_GUIDE.md) - Complete workflows and patterns
- [SSH Setup Guide](docs/SSH_SETUP_GUIDE.md) - One-command SSH automation
- [Scale-Out vs Scale-Up](docs/SCALE_OUT_VS_SCALE_UP.md) - Performance and cost comparison
- [Cloud Scripts Guide](scripts/README.md) - GCP automation and templates
## ⚙️ Key Features
### I/O Rate Control (v0.7.1)
Throttle operation start rate with realistic arrival patterns:
```yaml
io_rate:
iops: 1000 # Target operations per second
distribution: exponential # Poisson arrivals (realistic)
# or "uniform" (fixed intervals)
# or "deterministic" (precise timing)
```
- **Inspired by rdf-bench**: Similar to `iorate=` parameter with enhanced distributions
- **Three distribution types**: Exponential (Poisson), Uniform (fixed), Deterministic (precise)
- **Drift compensation**: tokio::time::Interval for uniform distribution accuracy
- **Zero overhead when disabled**: Optional wrapper for maximum performance
- **Per-worker division**: Target IOPS automatically split across concurrent workers
See [I/O Rate Control Guide](docs/IO_RATE_CONTROL_GUIDE.md) for detailed usage and examples.
### TSV Export with Aggregate Rows
Machine-readable output with per-bucket and aggregate summary rows:
- **Per-bucket rows**: Statistics for each size bucket (zero, 1B-8KiB, 8KiB-64KiB, etc.)
- **Aggregate rows**: "ALL" rows combining all size buckets per operation type (GET/PUT/META)
- **Accurate latency merging**: HDR histogram merging for statistically correct percentiles
- **Distributed support**: Per-agent TSVs and consolidated TSV with overall aggregates
### Per-Operation Concurrency
Fine-grained worker pool control:
```yaml
concurrency: 32 # Global default
workload:
- op: get
path: "data/*"
weight: 70
concurrency: 64 # More GET workers
- op: put
path: "uploads/"
weight: 30
concurrency: 8 # Fewer PUT workers
```
### Config Validation
Verify YAML before execution:
```bash
sai3-bench run --config my-workload.yaml --dry-run
```
See [Config Syntax](docs/CONFIG_SYNTAX.md) for complete reference.
## 🛠️ Development
### Requirements
- Rust stable toolchain (2024 edition)
- `protoc` compiler for gRPC (distributed mode)
- Storage credentials for cloud backends
### Building & Testing
```bash
# Build
cargo build --release
# Run tests
cargo test
# Streaming replay tests (must run sequentially)
cargo test --test streaming_replay_tests -- --test-threads=1
```
See [Cloud Storage Setup](docs/CLOUD_STORAGE_SETUP.md) for authentication details.
## 🔗 Related Projects
- **[s3dlio](https://github.com/russfellows/s3dlio)** - The underlying multi-protocol storage library powering sai3-bench
- **[polarWarp](https://github.com/russfellows/polarWarp)** - Op-log analysis tool for parsing and visualizing s3dlio operation logs
## 📄 License
GPL-3.0 License - See [LICENSE](LICENSE) for details.