### Fork

```bash
# Fork a trace at step 3
agent-replay fork --from-step 3

# Fork with modified input
agent-replay fork --from-step 2 --modify-input '{"task":"revised prompt"}'

# Tag the fork
agent-replay fork --from-step 4 --tag experiment-1
```

### Evaluate

```bash
# Run all built-in deterministic checks
agent-replay eval

# Run a specific preset
agent-replay eval --preset hallucination-check
agent-replay eval --preset safety-check
agent-replay eval --preset completeness-check

# Run AI-powered evaluation (requires API key)
agent-replay eval --ai
agent-replay eval --preset ai-root-cause
agent-replay eval --preset ai-quality-review
agent-replay eval --preset ai-security-audit
agent-replay eval --preset ai-optimization

# Set a cost budget for AI evals
agent-replay eval --ai --max-cost 0.05

# Custom rubric file
agent-replay eval --rubric my-rubric.yaml

# JSON output
agent-replay eval --json
```

### Guardrails

```bash
# List all policies
agent-replay guard list

# Add a policy that blocks delete operations
agent-replay guard add --name no-deletes \
--pattern '{"step_type":"tool_call","name_contains":"delete"}' \
--action deny

# Test all policies against a trace
agent-replay guard test

# Remove a policy
agent-replay guard remove
```

### Export

```bash
# Export as JSON
agent-replay export --format json --output traces.json

# Export completed traces as JSONL
agent-replay export --format jsonl --status completed --output good.jsonl

# Build a golden dataset for regression testing
agent-replay export --format golden --tag production --output golden.json
```

### Dashboard

```bash
# Full-screen terminal dashboard with charts and stats
agent-replay dashboard

# Custom refresh interval
agent-replay dashboard --refresh 10
```

Keyboard: `q` quit, `r` refresh, arrow keys navigate.

### Configuration

```bash
# Show current config
agent-replay config list

# Set an API key for AI-powered evaluation
agent-replay config set ai.api_keys.anthropic sk-ant-...
agent-replay config set ai.api_keys.google AIza...
agent-replay config set ai.api_keys.openai sk-...

# Choose a specific provider instead of auto-detect
agent-replay config set ai.provider anthropic

# Test that your API key works
agent-replay config test-ai

# Read a config value
agent-replay config get ai.provider
```

You can also set API keys via environment variables: `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`. Environment variables take priority over config file values.

## Evaluation Presets

### Deterministic Presets

These run instantly with no API key required.

**hallucination-check** — Detects hallucination indicators:
- Flags excessive hedging language (30%)
- Checks if output is grounded in retrieval content (40%)
- Verifies no error steps present (30%)
- Threshold: 0.7

**safety-check** — Detects safety concerns:
- Flags dangerous tool calls like delete/drop/destroy (40%)
- Checks for PII in output (SSN, credit card, email patterns) (30%)
- Detects prompt injection patterns (30%)
- Threshold: 0.8

**completeness-check** — Validates execution completeness:
- Ensures at least one output step exists (40%)
- Verifies all tool calls have output (30%)
- Checks trace doesn't end with an error (30%)
- Threshold: 0.7

### AI-Powered Presets

These require an API key. They use the cheapest models by default (Haiku 4.5, Gemini 2.0 Flash, or GPT-4o-mini) and typically cost less than $0.01 per evaluation.

**ai-root-cause** — For failed traces. Identifies what went wrong, which step caused it, contributing factors, and suggests a fix. Returns a confidence score.

**ai-quality-review** — Scores any trace on four dimensions: relevance, completeness, coherence, and accuracy (each 1-10). Returns an overall quality score.

**ai-security-audit** — Checks for prompt injection, data exfiltration, unauthorized access patterns, and privilege escalation. Returns a risk level (none/low/medium/high/critical) and specific findings.

**ai-optimization** — Analyzes token efficiency and identifies redundant steps, unnecessary tool calls, and wasted context. Returns an efficiency score and specific optimization suggestions.

### Custom Rubrics

Create a YAML or JSON file with pattern-based criteria:

```yaml
name: my-custom-check
threshold: 0.8
criteria:
- name: has_greeting
pattern: "hello|hi|welcome"
expected: true
weight: 1
- name: no_profanity
pattern: "badword1|badword2"
expected: false
weight: 2
```

```bash
agent-replay eval --rubric my-rubric.yaml
```

## Trace Format

To ingest your agent's execution data, export it as JSON matching this structure:

```json
{
"agent_name": "my-agent",
"agent_version": "1.0.0",
"trigger": "user_message",
"status": "completed",
"input": { "task": "book a flight to Tokyo" },
"output": { "result": "Flight booked: AA 1234" },
"started_at": "2026-02-27T10:00:00.000Z",
"ended_at": "2026-02-27T10:00:03.200Z",
"total_duration_ms": 3200,
"total_tokens": 4500,
"total_cost_usd": 0.018,
"error": null,
"tags": ["production"],
"steps": [
{
"step_number": 1,
"step_type": "thought",
"name": "analyze_request",
"input": { "message": "book a flight to Tokyo" },
"output": { "intent": "flight_booking" },
"duration_ms": 120,
"tokens_used": 400
},
{
"step_number": 2,
"step_type": "tool_call",
"name": "search_flights",
"input": { "destination": "TYO" },
"output": { "flights": ["AA 1234", "UA 5678"] },
"duration_ms": 800,
"tokens_used": 200
}
]
}
```

Only `agent_name` is required. Everything else is optional.

### Step Types

| Type | Description |
|------|-------------|
| `thought` | Agent reasoning or planning |
| `tool_call` | External tool invocation |
| `llm_call` | LLM API call |
| `retrieval` | RAG / document retrieval |
| `output` | Response delivery |
| `decision` | Decision point |
| `error` | Error occurred |
| `guard_check` | Guardrail policy check |

## Guardrail Policies

Policies match against trace steps and trigger actions.

### Match Pattern

```json
{
"step_type": "tool_call",
"name_contains": "delete",
"name_regex": "drop|destroy",
"input_contains": "production",
"output_contains": "error"
}
```

All fields are optional. When multiple fields are specified, all must match (AND logic). `name_contains` does a case-insensitive substring match; `name_regex` uses a regular expression.

### Actions

| Action | Description |
|--------|-------------|
| `allow` | Explicitly allow matching steps |
| `deny` | Block matching steps |
| `warn` | Flag for review |
| `require_review` | Require human review before proceeding |

## AI Provider Setup

`agent-replay` auto-detects your API key in this priority order:

1. **Anthropic** (default model: `claude-haiku-4-5-20251001`)
2. **Google Gemini** (default model: `gemini-2.0-flash`)
3. **OpenAI** (default model: `gpt-4o-mini`)

Set a key via environment variable or config:

```bash
# Environment variable (recommended)
export ANTHROPIC_API_KEY=sk-ant-...

# Or store in config
agent-replay config set ai.api_keys.anthropic sk-ant-...

# Verify it works
agent-replay config test-ai
```

All AI presets use the cheapest available model. A typical evaluation costs less than $0.01.

## Programmatic API

You can also use `agent-replay` as a library:

```typescript
import { openDatabase, createTrace, getTraceById } from 'agent-replay';

const db = openDatabase('.agent-replay/traces.db');
const trace = createTrace(db, { agent_name: 'my-agent', status: 'completed' });
```

## Development