https://github.com/fullstop000/token-efficiency
Token efficiency metrics and dashboard for AI coding agents (Claude Code and friends).
https://github.com/fullstop000/token-efficiency
Last synced: about 1 month ago
JSON representation
Token efficiency metrics and dashboard for AI coding agents (Claude Code and friends).
- Host: GitHub
- URL: https://github.com/fullstop000/token-efficiency
- Owner: Fullstop000
- License: mit
- Created: 2026-05-16T15:59:33.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2026-05-16T18:29:23.000Z (2 months ago)
- Last Synced: 2026-05-16T20:36:36.128Z (2 months ago)
- Language: HTML
- Size: 33.2 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# token-efficiency
Token efficiency metrics, dashboard, and notes for AI coding agents.
This repo started as a tool to analyze [Claude Code](https://docs.claude.com/claude-code) session logs (`~/.claude/projects/*.jsonl`) and answer *where are my tokens going?* — but the broader question, **how do you measure token efficiency for an agentic system fairly**, has surprisingly little practical tooling. This is a place to put the tools and the thinking.
## What's here
- **CLI** — `token-efficiency parse | serve | report`
- **Dashboard** — single-file HTML page with overview, daily breakdown, per-session drill-down
- **Methodology** — [docs/METHODOLOGY.md](docs/METHODOLOGY.md): the metrics framework, what's directly computable vs. heuristic, what makes agent-vs-agent comparisons unfair
## Install
```bash
pipx install git+https://github.com/Fullstop000/token-efficiency
# or
pip install git+https://github.com/Fullstop000/token-efficiency
```
## Usage
```bash
# parse session logs into data.json
token-efficiency parse
# get prioritized recommendations with $-saving estimates
token-efficiency advise
# per-session waste observations (repeat reads, dup tool calls, etc.)
token-efficiency issues --top 20
token-efficiency issues --type repeat_file_read --severity high
# inspect rule definitions (YAML)
token-efficiency rules # list all
token-efficiency rules show repeat_file_read # full YAML for one rule
token-efficiency rules path # bundled vs. user-override path
# serve the dashboard (defaults to http://127.0.0.1:8765/)
token-efficiency serve --open
# parse + serve, AND enable the in-browser refresh button
token-efficiency serve --projects-dir ~/.claude/projects --open
# text summary, no browser
token-efficiency report
```
## Recommendations (`advise`)
The tool turns the metrics into a prioritized action list with **estimated monthly savings**, **confidence**, and **risk**. Each card shows the math behind the number — no hidden estimates. Categories shipped in v0.2:
- **Unused MCP servers** — disable a whole namespace if it has near-zero call coverage
- **Buggy / high-error tools** — surface tools with >20% error rate; retries are real money
- **Hot-file → CLAUDE.md candidates** — files Read 10+ times without edit
- **Session hygiene** — long-running sessions, cache-thrashing patterns
The dashboard's **Actions** tab shows the same list with per-card *Mark done* / *Dismiss* buttons (state persists in localStorage). The tool advises; you decide.
Mode: advise-only. The tool never edits your settings — it gives you the exact command to run.
## Issue rules (YAML, user-overridable)
Each issue type is defined by a YAML rule shipped with the package. Override any field by placing a YAML at `~/.token-efficiency/rules/.yaml` — the user file is shallow-merged onto the bundled default, so you only declare the fields you want to change. Bundled rules:
| Rule id | Detects |
|---|---|
| `repeat_file_read` | Same file Read 2+ times in one session |
| `dup_tool_call` | Same tool + identical input called consecutively |
| `large_unedited_read` | One Read of a >10k-token file never edited |
Example: drop your `repeat_file_read.yaml` threshold by writing only the override:
```yaml
# ~/.token-efficiency/rules/repeat_file_read.yaml
id: repeat_file_read
thresholds:
high_threshold: 3 # become 'high' at 3+ reads instead of 5
```
The dashboard's **Rules** tab is a read-only control panel: shows each rule's full YAML, source (bundled vs. user-override), enabled state, and live trigger counts against your dataset. No in-browser editing in v1 — edits go through files only.
Override the source directory or output path:
```bash
token-efficiency parse --projects-dir /path/to/projects --out my-data.json
token-efficiency serve --data my-data.json --port 9000
```
## What the dashboard shows
**Overview** — KPI cards (total cost, sessions, p50 / p90 cost per session, CHR p50, tool error rate, flagged count), cost-per-session histogram, CHR histogram, cost-by-model donut, daily cost trend, and a flagged-sessions list sorted by severity.
**Daily** — one row per day: sessions, total cost, p50 / p90 cost per session, total tokens, median CHR, median tool-error rate, flagged count. Click a row to drill into that day's sessions.
**Sessions** — full session list, filterable and sortable. Click a session to open a modal with: token decomposition, per-turn cost timeline, quality breakdown (tool errors, duplicate-input calls, read-waste, useful-tool ratio, output composition).
## Metrics
### Tier 1 — direct from log fields
| Metric | Formula |
|---|---|
| Cost ($) | Anthropic pay-per-token math: cache_read × 0.1, cache_write_5m × 1.25, cache_write_1h × 2, plus base input/output rates |
| CHR (cache hit ratio) | `cache_read / (cache_read + input_new + cache_write)` |
| Tool error rate | `tool_result.is_error == true` / total tool_use blocks |
| Turn count | Unique `requestId`s |
| Duration | `max(ts) − min(ts)` across records |
| Output composition | Char-share of `thinking` / `text` / `tool_use` blocks |
| Model mix | Cost share per model |
### Tier 2 — heuristic
| Metric | Heuristic |
|---|---|
| Read-waste | Count of files Read but never Edited/Written in this session |
| Duplicate-input calls | Tool calls with same name + serialized input as the previous call |
| Flags | Cache thrashing, tool retry storm, read-waste, high cost, very long session |
### Tier 3 — NOT computed
`pass@1`, task success, cost-per-resolved-issue (CPRI) — these require user-supplied labels for what counts as a successful session. Open issue if you want to design the labeling flow.
## The dollar figure
Cost is computed at **theoretical pay-per-token API rates** at the prices defined in [`token_efficiency/pricing.py`](token_efficiency/pricing.py). If you're on a Claude Code subscription plan, your actual bill is the subscription — the dollar figure here is useful for relative efficiency comparisons and spotting outliers, **not as an invoice**.
Edit `pricing.py` if your assumed rates differ.
## Status & roadmap
This is v0.1. Working: Claude Code JSONL parser + dashboard. Not yet:
- Parsers for other agent scaffolds (Codex CLI, Aider, Cursor)
- Tier 3 success labeling
- Cross-run distribution comparisons (this week vs. last week)
- Composite Agentic Value Score on a leaderboard
Contributions and issue reports welcome.
## License
MIT — see [LICENSE](LICENSE).