Batching and Async Patterns

Foundational papers

• AWS Bedrock Batch Inference - Bedrock specific batch jobs across supported models.
• LiteLLM batch passthrough - Unified interface to OpenAI, Anthropic, and Vertex batch APIs.
• OpenAI Batch API - 50 percent off, 24 hour completion window, up to 50,000 requests per batch and 200 MB per file. Stacks with prompt caching for compounded discounts.
• Anthropic Message Batches API - 50 percent off, 24 hour window, up to 100,000 requests per batch with up to 300K output tokens via beta header.
• Google Gemini Batch Mode - 50 percent off, supports very large batches via JSONL upload on both Gemini API and Vertex AI.

Benchmarks and Leaderboards

Cost calculators and dashboards

• LLM-Stats - 296+ models with composite scores blending quality, price, and speed.
• Vellum LLM Leaderboard - Community trusted leaderboard across reasoning, coding, and math.
• TokenCost Leaderboard - Sortable by quality, value (quality per dollar), and speed.
• PricePerToken Leaderboards - Coding, reasoning, value, speed, and price filters.
• CostGoat LLM API Comparison - 324+ APIs ranked by quality, price, and value.
• OpenRouter Rankings - Usage based rankings across the OpenRouter platform.
• Artificial Analysis - The most cited LLM cost and quality benchmark with live data on quality, speed, and price across all major models.

Caching

Inference infrastructure KV cache

• vLLM - PagedAttention, continuous batching, and automatic prefix caching. The default open source LLM serving framework.
• SGLang - Fast LLM serving with RadixAttention prefix caching, particularly strong for prompts with shared structure.
• LMCache - Extends vLLM with GPU to CPU RAM to disk KV cache tiering for significant latency reduction on cache hits.

Provider prompt caching

• OpenAI Prompt Caching - 50 percent off cached input, automatic for prompts over 1024 tokens, no code changes required.
• Google Gemini Context Caching - Implicit and explicit caching, charged for cache storage. Best for very long contexts.
• AWS Bedrock Prompt Caching - Bedrock's wrapping of Anthropic style prompt caching, with cachePoint markers translated automatically by SDKs like LiteLLM.
• Anthropic Prompt Caching - 90 percent off cached input via explicit cache_control markers. The most aggressive provider discount available.

Semantic caching libraries

• GPTCache - The reference open source semantic cache with multiple embedding and storage backends, integrated with LangChain and llama_index.
• Redis LangCache - Closed source. Redis's official semantic cache product, production grade.
• Portkey semantic cache - Feature within the Portkey gateway with up to 40 percent reported cost reduction.

Calculators and Estimators

Cost calculators and dashboards

• OpenAI Tokenizer - Official browser tokenizer for cl100k_base and o200k_base.
• Claude Tokenizer - Community built tokenizer that uses the official Anthropic API.
• LiteLLM Pricing Calculator - Cost estimate endpoint built into the LiteLLM proxy that forecasts spend from expected token usage and request volume.
• PricePerToken - Comparison and leaderboards across 300+ models.
• TokenCost App - Leaderboard with quality versus cost scoring.
• CostGoat - Closed source. Desktop app for tracking OpenRouter and multi provider spend in real time.

Tokenizers

• tiktoken - The official OpenAI BPE tokenizer in Python.
• gpt-tokenizer - Pure JS tokenizer with the smallest bundle and fastest small text performance.
• tiktoken-go - Go port of tiktoken.
• tiktoken-rs - Rust port of tiktoken.
• SharpToken - C# port of tiktoken.
• jtokkit - Java port of tiktoken.
• Gemini countTokens - Official pre flight token counting via the Gemini SDK.
• Anthropic countTokens API - Official, free, network based. Currently the only ground truth source for Claude models from the official SDK.

Cost Aware Serving Research

Cost calculators and dashboards

• FrugalGPT - Chen, Zaharia, Zou. The seminal paper. Proposes prompt adaptation, LLM approximation, and LLM cascade with up to 98 percent cost reduction at GPT-4 quality on benchmark tasks.
• RouteLLM: Learning to Route LLMs with Preference Data - Ong et al. ICLR 2025. Introduces preference data trained routers between a strong and a weak model.
• Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing - Ding et al. ICLR 2024. Cost quality cascade routing.
• AutoMix: Automatically Mixing Language Models - Aggarwal et al. Smaller model self verifies before escalating to a larger model.
• xRouter: Training Cost-Aware LLMs Orchestration via Reinforcement Learning - RL trained cost aware orchestrator. Useful for thinking about routing as a learnable policy.
• Don't Break the Cache - Recent eval of prompt caching across OpenAI, Anthropic, and Google for agent workloads.
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone - SJTU. On device cost reduction through smart serving on mobile hardware.

Cost Tracking, Observability, and Budgets

Budgets and circuit breakers

• LiteLLM virtual keys and budgets - Per user, per team, and per key budgets enforced in the LiteLLM proxy. The most flexible self hostable budget layer.
• Finout - Closed source. Broader FinOps platform with LLM cost as one of many tracked services. Useful at organisation scale.

Commercial

• Datadog LLM Observability - Closed source. Cost breakdowns by provider, model, prompt, or user inside Datadog APM.
• LangSmith - Closed source. Deepest integration with LangChain and LangGraph, with input, output, and other costs separated per trace.
• Braintrust - Closed source. Eval first platform that attaches estimated cost and token count to every span in a trace.
• Galileo - Closed source. Eval first with quality alerting and drift detection.

Cost telemetry standards

• OpenTelemetry GenAI semantic conventions - The upstream standard for naming generative AI spans, attributes, and metrics. Adopted by Phoenix, OpenLLMetry, and Langtrace.

Mobile and on device

• react-native-llm-meter - Token usage, cost, and streaming TTFT for Claude, GPT, and Gemini calls from React Native and Expo apps. Disclosure: maintained by this list's author.

Open source

• agenttrace - Local-first TUI and CLI that turns AI coding agent trace logs into cost, token, latency, and health regression reports.
• Langfuse - Full LLM tracing, prompt management, evaluation, datasets, with token and cost tracking exposed via a metrics API. Self hostable on PostgreSQL and ClickHouse.
• Helicone - Proxy first observability with cost tracking, caching, and rate limits, no SDK install required. Self hostable.
• OpenLLMetry - OpenTelemetry instrumentation for LLMs from Traceloop. Vendor neutral, exports to any OTel backend.
• Phoenix - OpenTelemetry native observability via OpenInference, notebook friendly with strong RAG and drift detection.
• Opik - Tracing, evaluation, and prompt playground from Comet. Good fit if you also fine tune your own models.
• Laminar - Agent debugging first observability with transcript view, outcome tracking, and rollout debugging.
• Langtrace - OpenTelemetry based LLM tracing with a clean OTel implementation.
• OpenObserve - Unified LLM and infrastructure observability with significantly lower storage costs than Datadog.
• TruLens - Evaluation first observability, strong for RAG.
• PostHog LLM Observability - LLM observability inside the broader PostHog product analytics platform.
• SigNoz - OpenTelemetry native APM with LLM observability features.

Pricing Data

Foundational papers

• model_prices_and_context_window.json - The canonical community maintained pricing JSON, lives in LiteLLM. 100+ providers. Vendor or fetch from this if you're building anything cost aware.
• OpenRouter Models API - JSON API with current pricing for 300+ models, cached at the edge.
• llm-info - Community maintained model and pricing metadata for the JS ecosystem.

Prompt Compression

Inference infrastructure KV cache

• LLMLingua - Microsoft Research. Token level prompt compression with up to 20x ratio while preserving instruction quality. Includes LLMLingua, LongLLMLingua, and LLMLingua-2.

Provider Cost Optimization Guides

Speculative decoding

• OpenAI reasoning model best practices - Reasoning effort tradeoffs for o-series models. Higher effort costs more output tokens, often catastrophically more.
• AWS Bedrock cross region inference - Cross region inference profiles for cost arbitrage and provisioned throughput tradeoffs.
• Anthropic pricing reference - Per model rates plus the breakdown of cache write and read multipliers, useful when modelling cache economics.
• Anthropic extended thinking guide - Cost tradeoffs of the extended thinking feature, where reasoning tokens dominate output cost.
• OpenAI optimizing accuracy - Cost relevant guidance on instruction placement, response length controls, and structured output.

Quantization, Distillation, and Compression

Distillation

• Hugging Face TRL - Transformer Reinforcement Learning library with student teacher pipelines for distilling large model behaviour into smaller fine tuned models.
• DistillKit - Arcee AI's open source distillation toolkit for producing small high quality student models from larger teachers.

Foundational papers

• Speculative Decoding - Leviathan et al. The technique that powers 2 to 3x throughput improvements in vLLM and SGLang for self hosted serving.
• AWQ: Activation-aware Weight Quantization - Lin et al. The AWQ paper.
• GPTQ: Accurate Post-Training Quantization - Frantar et al. The GPTQ paper.
• SmoothQuant - Xiao et al. Activation outlier handling.
• BitNet b1.58 - Microsoft. 1.58 bit quantization research.

Quantization libraries

• bitsandbytes - Runtime 4 and 8 bit quantization from Tim Dettmers, with NF4, FP4, and LLM.int8. The only option that supports QLoRA training on quantized base models.
• ExLlamaV2 - EXL2 quantization, speed focused, optimized for low latency inference.
• llama.cpp and GGUF - The reference for CPU and Apple Silicon quantization. K quants retain about 92 to 95 percent of FP16 quality at about 25 percent of memory.
• MLC LLM - Universal compiler for deploying LLMs across hardware including mobile, web, and edge. Cross platform quantization.
• MLX - Apple Silicon native ML framework with quantization tools, from Apple ML Research.
• TensorRT-LLM - NVIDIA's optimized inference framework with INT8 and FP8 quantization, kernel fusion, and multi GPU.

Related Lists

Speculative decoding

• awesome-mobile-llm - On device and mobile LLMs, research flavoured.
• awesome-on-device-AI-systems - On device AI inference systems.
• Awesome-LLMOps - Broader LLMOps tooling.

Routing and Model Selection

Gateways and proxies

• LiteLLM - The reference open source LLM proxy. 100+ providers behind one OpenAI compatible API, with budgets, virtual keys, fallbacks, and spend logging.
• Portkey - AI gateway with semantic caching, guardrails, programmable routing, and observability. Self hostable or managed.
• Helicone AI Gateway - Rust based gateway with low overhead, built in caching, and load balancing.
• Vercel AI Gateway - Closed source. Default provider in the Vercel AI SDK.
• Kong AI Gateway - Closed source. Kong's AI plugins for teams already running Kong as their API layer.
• Bifrost - Rust based router with microsecond scale overhead at high RPS, plus built in budget controls and intelligent routing.
• TrueFoundry AI Gateway - Closed source. Enterprise gateway with RBAC, virtual models, and deployment flexibility.
• OpenRouter - Closed source. 300+ models behind one API key with cache aware routing and prompt caching.
• Cloudflare AI Gateway - Closed source. Edge native gateway with caching, rate limiting, and analytics. Free tier.

Smart and cost aware routers

• NotDiamond - Closed source. Per query model routing with quality aware predictions.

Self Hosting Starting Points

Foundational papers

• Ollama - The simplest way to run open source LLMs locally. Single binary, model library with one line pulls, OpenAI compatible API.
• LM Studio - Closed source. Desktop app for running and chatting with local models, with a built in OpenAI compatible server.