{"id":14779623,"url":"https://github.com/tensorzero/tensorzero","last_synced_at":"2026-01-16T17:38:29.766Z","repository":{"id":256260517,"uuid":"829640443","full_name":"tensorzero/tensorzero","owner":"tensorzero","description":"TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluation, and experimentation.","archived":false,"fork":false,"pushed_at":"2025-10-18T23:48:58.000Z","size":108107,"stargazers_count":10425,"open_issues_count":353,"forks_count":717,"subscribers_count":71,"default_branch":"main","last_synced_at":"2025-10-19T09:35:26.155Z","etag":null,"topics":["ai","ai-engineering","anthropic","artificial-intelligence","deep-learning","genai","generative-ai","gpt","large-language-models","llama","llm","llmops","llms","machine-learning","ml","ml-engineering","mlops","openai","python","rust"],"latest_commit_sha":null,"homepage":"https://tensorzero.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tensorzero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":"CLA.md"}},"created_at":"2024-07-16T21:00:53.000Z","updated_at":"2025-10-19T08:06:48.000Z","dependencies_parsed_at":"2024-11-20T22:39:20.195Z","dependency_job_id":"f5b58ac7-cf61-4ab0-b9b5-4da7dd34cf31","html_url":"https://github.com/tensorzero/tensorzero","commit_stats":{"total_commits":253,"total_committers":7,"mean_commits":"36.142857142857146","dds":0.4861660079051383,"last_synced_commit":"cf339e8a8b614b399a622c235ea512bf7f185ab4"},"previous_names":["tensorzero/tensorzero"],"tags_count":120,"template":false,"template_full_name":null,"purl":"pkg:github/tensorzero/tensorzero","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Ftensorzero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Ftensorzero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Ftensorzero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Ftensorzero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tensorzero","download_url":"https://codeload.github.com/tensorzero/tensorzero/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorzero%2Ftensorzero/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280173630,"owners_count":26284901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-20T02:00:06.978Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-engineering","anthropic","artificial-intelligence","deep-learning","genai","generative-ai","gpt","large-language-models","llama","llm","llmops","llms","machine-learning","ml","ml-engineering","mlops","openai","python","rust"],"created_at":"2024-09-17T01:00:54.767Z","updated_at":"2026-01-16T17:38:29.759Z","avatar_url":"https://github.com/tensorzero.png","language":"Rust","readme":"\u003cp\u003e\u003cpicture\u003e\u003cimg src=\"https://github.com/user-attachments/assets/47d67430-386d-4675-82ad-d4734d3262d9\" alt=\"TensorZero Logo\" width=\"128\" height=\"128\"\u003e\u003c/picture\u003e\u003c/p\u003e\n\n# TensorZero\n\n\u003cp\u003e\u003cpicture\u003e\u003cimg src=\"https://www.tensorzero.com/github-trending-badge.svg\" alt=\"#1 Repository Of The Day\"\u003e\u003c/picture\u003e\u003c/p\u003e\n\n**TensorZero is an open-source stack for _industrial-grade LLM applications_:**\n\n- **Gateway:** access every LLM provider through a unified API, built for performance (\u003c1ms p99 latency)\n- **Observability:** store inferences and feedback in your database, available programmatically or in the UI\n- **Optimization:** collect metrics and human feedback to optimize prompts, models, and inference strategies\n- **Evaluation:** benchmark individual inferences or end-to-end workflows using heuristics, LLM judges, etc.\n- **Experimentation:** ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.\n\nTake what you need, adopt incrementally, and complement with other tools.\n\n\u003cvideo src=\"https://github.com/user-attachments/assets/04a8466e-27d8-4189-b305-e7cecb6881ee\"\u003e\u003c/video\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/\" target=\"_blank\"\u003eWebsite\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/docs\" target=\"_blank\"\u003eDocs\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.x.com/tensorzero\" target=\"_blank\"\u003eTwitter\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/slack\" target=\"_blank\"\u003eSlack\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/discord\" target=\"_blank\"\u003eDiscord\u003c/a\u003e\u003c/b\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/docs/quickstart\" target=\"_blank\"\u003eQuick Start (5min)\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/docs/gateway/deployment\" target=\"_blank\"\u003eDeployment Guide\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/docs/gateway/api-reference\" target=\"_blank\"\u003eAPI Reference\u003c/a\u003e\u003c/b\u003e\n  ·\n  \u003cb\u003e\u003ca href=\"https://www.tensorzero.com/docs/gateway/deployment\" target=\"_blank\"\u003eConfiguration Reference\u003c/a\u003e\u003c/b\u003e\n\u003c/p\u003e\n\n---\n\n\u003e [!NOTE]\n\u003e\n\u003e ### **Coming Soon: TensorZero Autopilot**\n\u003e\n\u003e TensorZero Autopilot is an **automated AI engineer** (powered by the TensorZero Stack) that analyzes LLM observability data, optimizes prompts and models, sets up evals, and runs A/B tests.\n\u003e **[Learn more](https://www.tensorzero.com/)** **[Join the waitlist](https://tensorzerodotcom.notion.site/2d87520bbad380c9ad0dd19566b3bc91)**\n\n## Features\n\n### 🌐 LLM Gateway\n\n\u003e **Integrate with TensorZero once and access every major LLM provider.**\n\n- [x] **[Call any LLM](https://www.tensorzero.com/docs/gateway/call-any-llm)** (API or self-hosted) through a single unified API\n- [x] Infer with **[streaming](https://www.tensorzero.com/docs/gateway/guides/streaming-inference)**, **[tool use](https://www.tensorzero.com/docs/gateway/guides/tool-use)**, **[structured outputs (JSON)](https://www.tensorzero.com/docs/gateway/generate-structured-outputs)**, **[batch](https://www.tensorzero.com/docs/gateway/guides/batch-inference)**, **[embeddings](https://www.tensorzero.com/docs/gateway/generate-embeddings)**, **[multimodal (images, files)](https://www.tensorzero.com/docs/gateway/guides/multimodal-inference)**, **[caching](https://www.tensorzero.com/docs/gateway/guides/inference-caching)**, etc.\n- [x] **[Create prompt templates and schemas](https://www.tensorzero.com/docs/gateway/create-a-prompt-template)** to enforce a consistent, typed interface between your application and the LLMs\n- [x] Satisfy extreme throughput and latency needs, thanks to 🦀 Rust: **[\u003c1ms p99 latency overhead at 10k+ QPS](https://www.tensorzero.com/docs/gateway/benchmarks)**\n- [x] Use any programming language: **[integrate via our Python SDK, any OpenAI SDK, or our HTTP API](https://www.tensorzero.com/docs/gateway/clients)**\n- [x] **[Ensure high availability](https://www.tensorzero.com/docs/gateway/guides/retries-fallbacks)** with routing, retries, fallbacks, load balancing, granular timeouts, etc.\n- [x] **[Enforce custom rate limits](https://www.tensorzero.com/docs/operations/enforce-custom-rate-limits)** with granular scopes (e.g. user-defined tags) to keep usage under control\n- [x] **[Set up auth for TensorZero](https://www.tensorzero.com/docs/operations/set-up-auth-for-tensorzero)** to allow clients to access models without sharing provider API keys\n- [ ] Soon: spend tracking and budgeting\n\n\u003cbr\u003e\n\n**Supported Model Providers:**\n**[Anthropic](https://www.tensorzero.com/docs/gateway/guides/providers/anthropic)**,\n**[AWS Bedrock](https://www.tensorzero.com/docs/gateway/guides/providers/aws-bedrock)**,\n**[AWS SageMaker](https://www.tensorzero.com/docs/gateway/guides/providers/aws-sagemaker)**,\n**[Azure](https://www.tensorzero.com/docs/gateway/guides/providers/azure)**,\n**[DeepSeek](https://www.tensorzero.com/docs/gateway/guides/providers/deepseek)**,\n**[Fireworks](https://www.tensorzero.com/docs/gateway/guides/providers/fireworks)**,\n**[GCP Vertex AI Anthropic](https://www.tensorzero.com/docs/gateway/guides/providers/gcp-vertex-ai-anthropic)**,\n**[GCP Vertex AI Gemini](https://www.tensorzero.com/docs/gateway/guides/providers/gcp-vertex-ai-gemini)**,\n**[Google AI Studio (Gemini API)](https://www.tensorzero.com/docs/gateway/guides/providers/google-ai-studio-gemini)**,\n**[Groq](https://www.tensorzero.com/docs/gateway/guides/providers/groq)**,\n**[Hyperbolic](https://www.tensorzero.com/docs/gateway/guides/providers/hyperbolic)**,\n**[Mistral](https://www.tensorzero.com/docs/gateway/guides/providers/mistral)**,\n**[OpenAI](https://www.tensorzero.com/docs/gateway/guides/providers/openai)**,\n**[OpenRouter](https://www.tensorzero.com/docs/gateway/guides/providers/openrouter)**,\n**[SGLang](https://www.tensorzero.com/docs/gateway/guides/providers/sglang)**,\n**[TGI](https://www.tensorzero.com/docs/gateway/guides/providers/tgi)**,\n**[Together AI](https://www.tensorzero.com/docs/gateway/guides/providers/together)**,\n**[vLLM](https://www.tensorzero.com/docs/gateway/guides/providers/vllm)**, and\n**[xAI (Grok)](https://www.tensorzero.com/docs/gateway/guides/providers/xai)**.\nNeed something else? TensorZero also supports **[any OpenAI-compatible API (e.g. Ollama)](https://www.tensorzero.com/docs/gateway/guides/providers/openai-compatible)**.\n\n\u003cbr\u003e\n\n\u003cdetails open\u003e\n\u003csummary\u003e\u003cb\u003eUsage: Python \u0026mdash; TensorZero SDK\u003c/b\u003e\u003c/summary\u003e\n\nYou can access any provider using the TensorZero Python SDK.\n\n1. `pip install tensorzero`\n2. Optional: Set up the TensorZero configuration.\n3. Run inference:\n\n```python\nfrom tensorzero import TensorZeroGateway  # or AsyncTensorZeroGateway\n\n\nwith TensorZeroGateway.build_embedded(...) as t0:\n    response = t0.inference(\n        model_name=\"openai::gpt-4o-mini\",\n        # Try other providers easily: \"anthropic::claude-sonnet-4-5-20250929\"\n        input={\n            \"messages\": [\n                {\n                    \"role\": \"user\",\n                    \"content\": \"Write a haiku about TensorZero.\",\n                }\n            ]\n        },\n    )\n```\n\nSee **[Quick Start](https://www.tensorzero.com/docs/quickstart)** for more information.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eUsage: Python \u0026mdash; OpenAI SDK\u003c/b\u003e\u003c/summary\u003e\n\nYou can access any provider using the OpenAI Python SDK with TensorZero.\n\n1. `pip install tensorzero`\n2. Optional: Set up the TensorZero configuration.\n3. Run inference:\n\n```python\nfrom openai import OpenAI  # or AsyncOpenAI\nfrom tensorzero import patch_openai_client\n\nclient = OpenAI()\n\npatch_openai_client(client, ...)\n\nresponse = client.chat.completions.create(\n    model=\"tensorzero::model_name::openai::gpt-4o-mini\",\n    # Try other providers easily: \"tensorzero::model_name::anthropic::claude-sonnet-4-5-20250929\"\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Write a haiku about TensorZero.\",\n        }\n    ],\n)\n```\n\nSee **[Quick Start](https://www.tensorzero.com/docs/quickstart)** for more information.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eUsage: JavaScript / TypeScript (Node) \u0026mdash; OpenAI SDK\u003c/b\u003e\u003c/summary\u003e\n\nYou can access any provider using the OpenAI Node SDK with TensorZero.\n\n1. Deploy `tensorzero/gateway` using Docker.\n   **[Detailed instructions →](https://www.tensorzero.com/docs/gateway/deployment)**\n2. Set up the TensorZero configuration.\n3. Run inference:\n\n```ts\nimport OpenAI from \"openai\";\n\nconst client = new OpenAI({\n  baseURL: \"http://localhost:3000/openai/v1\",\n});\n\nconst response = await client.chat.completions.create({\n  model: \"tensorzero::model_name::openai::gpt-4o-mini\",\n  // Try other providers easily: \"tensorzero::model_name::anthropic::claude-sonnet-4-5-20250929\"\n  messages: [\n    {\n      role: \"user\",\n      content: \"Write a haiku about TensorZero.\",\n    },\n  ],\n});\n```\n\nSee **[Quick Start](https://www.tensorzero.com/docs/quickstart)** for more information.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eUsage: Other Languages \u0026 Platforms \u0026mdash; HTTP API\u003c/b\u003e\u003c/summary\u003e\n\nTensorZero supports virtually any programming language or platform via its HTTP API.\n\n1. Deploy `tensorzero/gateway` using Docker.\n   **[Detailed instructions →](https://www.tensorzero.com/docs/gateway/deployment)**\n2. Optional: Set up the TensorZero configuration.\n3. Run inference:\n\n```bash\ncurl -X POST \"http://localhost:3000/inference\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model_name\": \"openai::gpt-4o-mini\",\n    \"input\": {\n      \"messages\": [\n        {\n          \"role\": \"user\",\n          \"content\": \"Write a haiku about TensorZero.\"\n        }\n      ]\n    }\n  }'\n```\n\nSee **[Quick Start](https://www.tensorzero.com/docs/quickstart)** for more information.\n\n\u003c/details\u003e\n\n### 🔍 LLM Observability\n\n\u003e **Zoom in to debug individual API calls, or zoom out to monitor metrics across models and prompts over time \u0026mdash; all using the open-source TensorZero UI.**\n\n- [x] Store inferences and **[feedback (metrics, human edits, etc.)](https://www.tensorzero.com/docs/gateway/guides/metrics-feedback)** in your own database\n- [x] Dive into individual inferences or high-level aggregate patterns using the TensorZero UI or programmatically\n- [x] **[Build datasets](https://www.tensorzero.com/docs/gateway/api-reference/datasets-datapoints)** for optimization, evaluation, and other workflows\n- [x] Replay historical inferences with new prompts, models, inference strategies, etc.\n- [x] **[Export OpenTelemetry traces (OTLP)](https://www.tensorzero.com/docs/operations/export-opentelemetry-traces)** and **[export Prometheus metrics](https://www.tensorzero.com/docs/observability/export-prometheus-metrics)** to your favorite application observability tools\n- [ ] Soon: AI-assisted debugging and root cause analysis; AI-assisted data labeling\n\n\u003ctable\u003e\n\u003ctr\u003e\u003c/tr\u003e \u003c!-- flip highlight order --\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\" align=\"center\" valign=\"middle\"\u003e\u003cb\u003eObservability » UI\u003c/b\u003e\u003c/td\u003e\n\u003ctd width=\"50%\" align=\"center\" valign=\"middle\"\u003e\u003cb\u003eObservability » Programmatic\u003c/b\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\" align=\"center\" valign=\"middle\"\u003e\u003cvideo src=\"https://github.com/user-attachments/assets/a23e4c95-18fa-482c-8423-6078fb4cf285\"\u003e\u003c/video\u003e\u003c/td\u003e\n\u003ctd width=\"50%\" align=\"left\" valign=\"middle\"\u003e\n\n```python\nt0.experimental_list_inferences(\n  function_name=\"sales_agent\",\n  filters=BooleanMetricFilter(\n      metric_name=\"converted_sale\",\n      value=True,\n  ),\n  # + compound filters\n  # + search\n  # + pagination\n  # ... and more ...\n)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### 📈 LLM Optimization\n\n\u003e **Send production metrics and human feedback to easily optimize your prompts, models, and inference strategies \u0026mdash; using the UI or programmatically.**\n\n- [x] Optimize your models with supervised fine-tuning, RLHF, and other techniques\n- [x] Optimize your prompts with automated prompt engineering algorithms like **[GEPA](https://www.tensorzero.com/docs/optimization/gepa)** and MIPROv2\n- [x] Optimize your **[inference strategy](https://www.tensorzero.com/docs/gateway/guides/inference-time-optimizations)** with dynamic in-context learning, best/mixture-of-N sampling, etc.\n- [x] Enable a feedback loop for your LLMs: a data \u0026 learning flywheel turning production data into smarter, faster, and cheaper models\n- [ ] Soon: synthetic data generation\n\n### 📊 LLM Evaluation\n\n\u003e **Compare prompts, models, and inference strategies using evaluations powered by heuristics and LLM judges.**\n\n- [x] **[Evaluate individual inferences](https://www.tensorzero.com/docs/evaluations/inference-evaluations/tutorial)** with _inference evaluations_ powered by heuristics or LLM judges (\u0026approx; unit tests for LLMs)\n- [x] **[Evaluate end-to-end workflows](https://www.tensorzero.com/docs/evaluations/workflow-evaluations/tutorial)** with _workflow evaluations_ with complete flexibility (\u0026approx; integration tests for LLMs)\n- [x] Optimize LLM judges just like any other TensorZero function to align them to human preferences\n- [ ] Soon: more built-in evaluators; headless evaluations\n\n\u003ctable\u003e\n  \u003ctr\u003e\u003c/tr\u003e \u003c!-- flip highlight order --\u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" align=\"center\" valign=\"middle\"\u003e\u003cb\u003eEvaluation » UI\u003c/b\u003e\u003c/td\u003e\n    \u003ctd width=\"50%\" align=\"center\" valign=\"middle\"\u003e\u003cb\u003eEvaluation » CLI\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" align=\"center\" valign=\"middle\"\u003e\u003cimg src=\"https://github.com/user-attachments/assets/f4bf54e3-1b63-46c8-be12-2eaabf615699\"\u003e\u003c/td\u003e\n    \u003ctd width=\"50%\" align=\"left\" valign=\"middle\"\u003e\n\u003cpre\u003e\u003ccode class=\"language-bash\"\u003edocker compose run --rm evaluations \\\n  --evaluation-name extract_data \\\n  --dataset-name hard_test_cases \\\n  --variant-name gpt_4o \\\n  --concurrency 5\u003c/code\u003e\u003c/pre\u003e\n\u003cpre\u003e\u003ccode class=\"language-bash\"\u003eRun ID: 01961de9-c8a4-7c60-ab8d-15491a9708e4\nNumber of datapoints: 100\n██████████████████████████████████████ 100/100\nexact_match: 0.83 ± 0.03 (n=100)\nsemantic_match: 0.98 ± 0.01 (n=100)\nitem_count: 7.15 ± 0.39 (n=100)\u003c/code\u003e\u003c/pre\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n### 🧪 LLM Experimentation\n\n\u003e **Ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.**\n\n- [x] **[Run adaptive A/B tests](https://www.tensorzero.com/docs/experimentation/run-adaptive-ab-tests)** to ship with confidence and identify the best prompts and models for your use cases.\n- [x] Enforce principled experiments in complex workflows, including support for multi-turn LLM systems, sequential testing, and more.\n\n### \u0026 more!\n\n\u003e **Build with an open-source stack well-suited for prototypes but designed from the ground up to support the most complex LLM applications and deployments.**\n\n- [x] Build simple applications or massive deployments with GitOps-friendly orchestration\n- [x] **[Extend TensorZero](https://www.tensorzero.com/docs/operations/extend-tensorzero)** with built-in escape hatches, programmatic-first usage, direct database access, and more\n- [x] Integrate with third-party tools: specialized observability and evaluations, model providers, agent orchestration frameworks, etc.\n- [x] Iterate quickly by experimenting with prompts interactively using the Playground UI\n\n## Frequently Asked Questions\n\n**How is TensorZero different from other LLM frameworks?**\n\n1. TensorZero enables you to optimize complex LLM applications based on production metrics and human feedback.\n2. TensorZero supports the needs of industrial-grade LLM applications: low latency, high throughput, type safety, self-hosted, GitOps, customizability, etc.\n3. TensorZero unifies the entire LLMOps stack, creating compounding benefits. For example, LLM evaluations can be used for fine-tuning models alongside AI judges.\n\n**Can I use TensorZero with \\_\\_\\_?**\n\nYes.\nEvery major programming language is supported.\nIt plays nicely with the **[OpenAI SDK](https://www.tensorzero.com/docs/gateway/clients/)**, **[OpenTelemetry](https://www.tensorzero.com/docs/operations/export-opentelemetry-traces/)**, and **[every major LLM](https://www.tensorzero.com/docs/integrations/model-providers/)**.\n\n**Is TensorZero production-ready?**\n\nYes.\nTensorZero is used by companies ranging from frontier AI startups to the Fortune 50.\n\nHere's a case study: **[Automating Code Changelogs at a Large Bank with LLMs](https://www.tensorzero.com/blog/case-study-automating-code-changelogs-at-a-large-bank-with-llms)**\n\n**How much does TensorZero cost?**\n\nTensorZero Stack (LLMOps platform) is 100% self-hosted and open-source.\n\nTensorZero Autopilot (automated AI engineer) is a complementary paid product powered by the TensorZero Stack.\n\n**Who is building TensorZero?**\n\nOur technical team includes a former Rust compiler maintainer, machine learning researchers (Stanford, CMU, Oxford, Columbia) with thousands of citations, and the chief product officer of a decacorn startup. We're backed by the same investors as leading open-source projects (e.g. ClickHouse, CockroachDB) and AI labs (e.g. OpenAI, Anthropic). See our **[$7.3M seed round announcement](https://www.tensorzero.com/blog/tensorzero-raises-7-3m-seed-round-to-build-an-open-source-stack-for-industrial-grade-llm-applications/)** and **[coverage from VentureBeat](https://venturebeat.com/ai/tensorzero-nabs-7-3m-seed-to-solve-the-messy-world-of-enterprise-llm-development/)**. We're **[hiring in NYC](https://www.tensorzero.com/jobs)**.\n\n**How do I get started?**\n\nYou can adopt TensorZero incrementally. Our **[Quick Start](https://www.tensorzero.com/docs/quickstart)** goes from a vanilla OpenAI wrapper to a production-ready LLM application with observability and fine-tuning in just 5 minutes.\n\n## Demo\n\n\u003e **Watch LLMs get better at data extraction in real-time with TensorZero!**\n\u003e\n\u003e **[Dynamic in-context learning (DICL)](https://www.tensorzero.com/docs/gateway/guides/inference-time-optimizations#dynamic-in-context-learning-dicl)** is a powerful inference-time optimization available out of the box with TensorZero.\n\u003e It enhances LLM performance by automatically incorporating relevant historical examples into the prompt, without the need for model fine-tuning.\n\nhttps://github.com/user-attachments/assets/4df1022e-886e-48c2-8f79-6af3cdad79cb\n\n## Get Started\n\n**Start building today.**\nThe **[Quick Start](https://www.tensorzero.com/docs/quickstart)** shows it's easy to set up an LLM application with TensorZero.\n\n**Questions?**\nAsk us on **[Slack](https://www.tensorzero.com/slack)** or **[Discord](https://www.tensorzero.com/discord)**.\n\n**Using TensorZero at work?**\nEmail us at **[hello@tensorzero.com](mailto:hello@tensorzero.com)** to set up a Slack or Teams channel with your team (free).\n\n## Examples\n\nWe are working on a series of **complete runnable examples** illustrating TensorZero's data \u0026 learning flywheel.\n\n\u003e **[Optimizing Data Extraction (NER) with TensorZero](https://github.com/tensorzero/tensorzero/tree/main/examples/data-extraction-ner)**\n\u003e\n\u003e This example shows how to use TensorZero to optimize a data extraction pipeline.\n\u003e We demonstrate techniques like fine-tuning and dynamic in-context learning (DICL).\n\u003e In the end, an optimized GPT-4o Mini model outperforms GPT-4o on this task \u0026mdash; at a fraction of the cost and latency \u0026mdash; using a small amount of training data.\n\n\u003e **[Agentic RAG — Multi-Hop Question Answering with LLMs](https://github.com/tensorzero/tensorzero/tree/main/examples/rag-retrieval-augmented-generation/simple-agentic-rag/)**\n\u003e\n\u003e This example shows how to build a multi-hop retrieval agent using TensorZero.\n\u003e The agent iteratively searches Wikipedia to gather information, and decides when it has enough context to answer a complex question.\n\n\u003e **[Writing Haikus to Satisfy a Judge with Hidden Preferences](https://github.com/tensorzero/tensorzero/tree/main/examples/haiku-hidden-preferences)**\n\u003e\n\u003e This example fine-tunes GPT-4o Mini to generate haikus tailored to a specific taste.\n\u003e You'll see TensorZero's \"data flywheel in a box\" in action: better variants leads to better data, and better data leads to better variants.\n\u003e You'll see progress by fine-tuning the LLM multiple times.\n\n\u003e **[Image Data Extraction — Multimodal (Vision) Fine-tuning](https://github.com/tensorzero/tensorzero/tree/main/examples/multimodal-vision-finetuning)**\n\u003e\n\u003e This example shows how to fine-tune multimodal models (VLMs) like GPT-4o to improve their performance on vision-language tasks.\n\u003e Specifically, we'll build a system that categorizes document images (screenshots of computer science research papers).\n\n\u003e **[Improving LLM Chess Ability with Best-of-N Sampling](https://github.com/tensorzero/tensorzero/tree/main/examples/chess-puzzles/)**\n\u003e\n\u003e This example showcases how best-of-N sampling can significantly enhance an LLM's chess-playing abilities by selecting the most promising moves from multiple generated options.\n\n## Blog Posts\n\nWe write about LLM engineering on the **[TensorZero Blog](https://www.tensorzero.com/blog)**.\nHere are some of our favorite posts:\n\n- **[Bandits in your LLM Gateway: Improve LLM Applications Faster with Adaptive Experimentation (A/B Testing)](https://www.tensorzero.com/blog/bandits-in-your-llm-gateway/)**\n- **[Is OpenAI's Reinforcement Fine-Tuning (RFT) Worth It?](https://www.tensorzero.com/blog/is-openai-reinforcement-fine-tuning-rft-worth-it/)**\n- **[Distillation with Programmatic Data Curation: Smarter LLMs, 5-30x Cheaper Inference](https://www.tensorzero.com/blog/distillation-programmatic-data-curation-smarter-llms-5-30x-cheaper-inference/)**\n- **[From NER to Agents: Does Automated Prompt Engineering Scale to Complex Tasks?](https://www.tensorzero.com/blog/from-ner-to-agents-does-automated-prompt-engineering-scale-to-complex-tasks/)**\n","funding_links":[],"categories":["Rust","Recently Updated","Applications","数据 Data","Agentic Framework","Other LLM Frameworks","Langchain","A01_文本生成_文本对话","Repos","语言资源库","Agent Development","Librerías para usar NLP en español","\u003ca name=\"Rust\"\u003e\u003c/a\u003eRust"],"sub_categories":["[May 10, 2025](/content/2025/05/10/README.md)","MLOps","Videos Playlists","大语言对话模型及数据","rust","Herramientas de observabilidad"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorzero%2Ftensorzero","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftensorzero%2Ftensorzero","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorzero%2Ftensorzero/lists"}