https://github.com/aptdnfapt/Ai-speedometer
Benchmark AI model speeds with TTFT, tokens/sec, and performance metrics .
https://github.com/aptdnfapt/Ai-speedometer
ai ai-sdk glm openai opencode qwen
Last synced: 20 days ago
JSON representation
Benchmark AI model speeds with TTFT, tokens/sec, and performance metrics .
- Host: GitHub
- URL: https://github.com/aptdnfapt/Ai-speedometer
- Owner: aptdnfapt
- Created: 2025-09-18T22:20:42.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2026-05-13T00:39:04.000Z (2 months ago)
- Last Synced: 2026-05-13T02:32:56.167Z (2 months ago)
- Topics: ai, ai-sdk, glm, openai, opencode, qwen
- Language: TypeScript
- Homepage:
- Size: 485 KB
- Stars: 19
- Watchers: 0
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# Ai-speedometer
A CLI tool for benchmarking AI models across multiple providers with parallel execution and performance metrics. fully opencode Compatible
[](https://discord.gg/6S7HwCxbMy)
Track OSS model speeds over time: [ai-speedometer.oliveowl.xyz](https://ai-speedometer.oliveowl.xyz/)
## Packages
The project is a monorepo split into three packages:
| Package | Description | Runtime |
|---|---|---|
| `ai-speedometer` | Full TUI + headless benchmark CLI | Bun 1.0+ |
| `ai-speedometer-headless` | Headless benchmark only, no TUI deps | Node.js 18+ or Bun |
| `@ai-speedometer/core` | Shared benchmark engine (private) | — |
## Install
### Standard (TUI + Headless)
Full interactive terminal UI and headless benchmark support. Requires [Bun](https://bun.sh).
```bash
bun install -g ai-speedometer
```
### Headless Only (Lightweight)
No TUI dependencies — runs on **Node.js or Bun**. Ideal for CI/CD, Docker, and scripts.
```bash
npm install -g ai-speedometer-headless
# or
bun install -g ai-speedometer-headless
```
Or run directly from source:
```bash
bun run packages/ai-speedometer/src/index.ts
```
## What It Measures
- **TTFT** (Time to First Token) - How fast the first response token arrives
- **Total Time** - Complete request duration
- **Tokens/Second** - Real-time throughput
- **F1000** (First to 1000) - Time to complete 1000 agentic requests (~300 tokens each), shown in hours. Captures real-world coding assistant performance better than TPS or TTFT alone. Lower is better.
- **Token Counts** - Input, output, and total tokens used
## Usage
### Interactive TUI (`ai-speedometer` only)
```bash
ai-speedometer
# or short alias and just use ! auto configured via opencode dont have to re add provider here .
aispeed
```
### Headless Benchmark (both packages)
```bash
# Verified provider
ai-speedometer --bench openai:gpt-4o
ai-speedometer-headless --bench openai:gpt-4o
# With explicit API key
ai-speedometer-headless --bench openai:gpt-4o --api-key "sk-..."
# Custom provider
ai-speedometer-headless --bench-custom myprovider:mymodel \
--base-url https://api.example.com \
--api-key "..."
# Pretty-print JSON output
ai-speedometer-headless --bench openai:gpt-4o --formatted
# Custom endpoint format
ai-speedometer-headless --bench-custom myprovider:mymodel \
--base-url https://... --endpoint-format chat/completions
```
## Features
- **opencode config Compatible aka models / provider from opencode works out of the box in this cli no need to re add them**
- **Monorepo Architecture** - Split into `core`, `ai-speedometer` (TUI), and `ai-speedometer-headless` (dedicated CLI)
- **Interactive TUI** - Full terminal UI with theming support, menus, search, and live benchmark progress
- **33 Themes** - Full theme system ported from opencode — dark themes (tokyonight, dracula, catppuccin, kanagawa, rosepine, nord, gruvbox, monokai, synthwave84, and more) and light themes (github, everforest, solarized, flexoki, vercel, mercury)
- **Live Theme Switcher** - Press `T` anywhere in the app to open a searchable theme picker — changes apply instantly across the entire UI and persist to config
- **Animated Progress Bar** - Smooth glow/dim sine wave animation during benchmarks, fully theme-aware
- **F1000 Metric** - First to 1000: measures real-world agentic coding performance (1000 requests × ~300 tokens)
- **FAQ/Learn Section** - In-app documentation explaining metrics, formulas, and resources
- **Headless Mode** - Run benchmarks without interactive UI via CLI flags — outputs JSON, perfect for CI/CD
- **Node.js Compatible Headless** - `ai-speedometer-headless` targets Node.js, no Bun required
- **REST API Benchmarking** - Works with all OpenAI-compatible providers
- **Parallel Execution** - Benchmark multiple models simultaneously
- **Provider Management** - Add verified, custom verified, and custom providers
## Development
```bash
# Run from source (monorepo)
bun run packages/ai-speedometer/src/index.ts
# Build all packages
bun run build
# Specific builds
bun run build:tui # -> packages/ai-speedometer/dist/
bun run build:headless # -> packages/ai-speedometer-headless/dist/
# Run tests
bun test
# Typecheck
bun run typecheck
```
## Configuration Files
API keys and configuration are stored in:
- **Verified + Custom Verified Providers**:
- Primary: `~/.local/share/opencode/auth.json`
- Backup: `~/.config/ai-speedometer/ai-benchmark-config.json` (verifiedProviders section)
- **Custom Providers**: `~/.config/ai-speedometer/ai-benchmark-config.json` (customProviders section)
- **Provider Definitions**: `./custom-verified-providers.json` (bundled at build time)
- **Theme**: set `"theme"` in `ai-benchmark-config.json`, or press `T` inside the TUI to switch live
- Dark: `tokyonight` (default), `dracula`, `catppuccin`, `catppuccin-frappe`, `catppuccin-macchiato`, `kanagawa`, `rosepine`, `nord`, `gruvbox`, `monokai`, `one-dark`, `nightowl`, `aura`, `ayu`, `carbonfox`, `cobalt2`, `cursor`, `material`, `matrix`, `opencode`, `orng`, `lucent-orng`, `osaka-jade`, `palenight`, `synthwave84`, `vesper`, `zenburn`
- Light: `github`, `everforest`, `solarized`, `flexoki`, `mercury`, `vercel`
## Requirements
- **`ai-speedometer`**: Bun 1.0+ (install from [bun.sh](https://bun.sh))
- **`ai-speedometer-headless`**: Node.js 18+ or Bun 1.0+
- API keys for AI providers
- Terminal with ANSI color support (TUI only)