{"id":49050913,"url":"https://github.com/briancaffey/hermes-otel","last_synced_at":"2026-06-28T00:01:37.108Z","repository":{"id":351835555,"uuid":"1211640020","full_name":"briancaffey/hermes-otel","owner":"briancaffey","description":"OTel Plugin for Hermes Agent","archived":false,"fork":false,"pushed_at":"2026-06-21T02:52:49.000Z","size":1409,"stargazers_count":27,"open_issues_count":1,"forks_count":12,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T04:21:04.554Z","etag":null,"topics":["arize-phoenix","grafana-tempo","hermes-agent","jaeger","langfuse","o11y","openinference","opentelemetry","otel","signoz"],"latest_commit_sha":null,"homepage":"https://briancaffey.github.io/hermes-otel/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/briancaffey.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-15T15:41:57.000Z","updated_at":"2026-06-21T02:52:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/briancaffey/hermes-otel","commit_stats":null,"previous_names":["briancaffey/hermes-otel"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/briancaffey/hermes-otel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/briancaffey%2Fhermes-otel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/briancaffey%2Fhermes-otel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/briancaffey%2Fhermes-otel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/briancaffey%2Fhermes-otel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/briancaffey","download_url":"https://codeload.github.com/briancaffey/hermes-otel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/briancaffey%2Fhermes-otel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34872279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arize-phoenix","grafana-tempo","hermes-agent","jaeger","langfuse","o11y","openinference","opentelemetry","otel","signoz"],"created_at":"2026-04-19T20:01:45.705Z","updated_at":"2026-06-28T00:01:37.100Z","avatar_url":"https://github.com/briancaffey.png","language":"Python","funding_links":[],"categories":["Skills, Plugins, and Extensions"],"sub_categories":["Skills and skill ecosystems"],"readme":"# hermes-otel\n\nOpenTelemetry plugin for [Hermes Agent](https://github.com/nousresearch/hermes-agent). Automatically exports LLM tool calls, model invocations, and API requests as OTel spans to any OTLP-compatible backend.\n\n## Backends\n\nTested with:\n- **[Phoenix](https://github.com/Arize-ai/phoenix)** (local or cloud) — traces + metrics\n- **[Langfuse](https://langfuse.com/docs)** (cloud or self-hosted) — traces only\n- **[LangSmith](https://smith.langchain.com/)** (LangChain's tracing platform) — traces only\n- **[SigNoz](https://signoz.io)** (cloud or self-hosted) — traces + metrics + logs\n- **[Jaeger](https://www.jaegertracing.io)** (local) — traces only\n- **[Grafana Tempo](https://grafana.com/oss/tempo/)** (local or Grafana Cloud) — traces only\n- **[Grafana LGTM](https://github.com/grafana/docker-otel-lgtm)** (local) — traces + metrics + logs\n- **[Uptrace](https://uptrace.dev)** (self-hosted) — traces + metrics + logs\n- **[OpenObserve](https://openobserve.ai)** (self-hosted) — traces + metrics + logs\n- **[Honeycomb](https://www.honeycomb.io/)** (cloud) — traces + metrics + logs — see [HONEYCOMB.md](HONEYCOMB.md)\n\nAny OTLP HTTP endpoint should work.\n\n- For Phoenix see [docker-compose/phoenix.yaml](docker-compose/phoenix.yaml)\n- For Langfuse see [https://langfuse.com/self-hosting/deployment/docker-compose](https://langfuse.com/self-hosting/deployment/docker-compose)\n- For Langsmith see [https://smith.langchain.com/](https://smith.langchain.com/)\n- For SigNoz see [docker-compose/signoz/](docker-compose/signoz/) (includes the upstream stack + port-remap notes)\n- For Grafana LGTM see [docker-compose/lgtm.yaml](docker-compose/lgtm.yaml) and [docker-compose/lgtm/README.md](docker-compose/lgtm/README.md)\n- For Uptrace see [docker-compose/uptrace.yaml](docker-compose/uptrace.yaml) and [docker-compose/uptrace/README.md](docker-compose/uptrace/README.md)\n- For OpenObserve see [docker-compose/openobserve.yaml](docker-compose/openobserve.yaml) and [docker-compose/openobserve/README.md](docker-compose/openobserve/README.md)\n\n## Installation\n\n```\nhermes plugins install briancaffey/hermes-otel\n```\n\nThe plugin lives in `~/.hermes/plugins/hermes_otel/` and Hermes auto-discovers it via `plugin.yaml`. However, the OTel dependencies must be installed into the **hermes-agent virtual environment** (where `hermes` itself runs):\n\n```bash\n# Install OTel runtime dependencies into the hermes-agent venv\n~/git/hermes-agent/venv/bin/pip install \\\n  opentelemetry-api \\\n  opentelemetry-sdk \\\n  opentelemetry-exporter-otlp-proto-http\n\n# Optional: for LangSmith time-ordered run IDs\n~/git/hermes-agent/venv/bin/pip install langsmith\n```\n\nYou can also install the plugin package itself in editable mode (this pulls in the same OTel deps automatically):\n\n```bash\n~/git/hermes-agent/venv/bin/pip install -e ~/.hermes/plugins/hermes_otel\n```\n\n### Running tests\n\nThe test suite uses its own isolated environment via `uv` and does **not** require the hermes-agent venv:\n\n```bash\ncd ~/.hermes/plugins/hermes_otel\n\n# Unit + integration tests (no Docker needed, \u003c1s)\nuv run --extra dev pytest\n\n# All E2E tests (requires Docker)\nuv run --extra dev --extra e2e pytest -m e2e\n\n# Phoenix E2E only (starts a single container)\nuv run --extra dev --extra e2e pytest -m phoenix\n\n# Langfuse E2E only (starts full stack via docker compose)\nuv run --extra dev --extra e2e pytest -m langfuse\n\n# Smoke tests — full pipeline: hermes API server -\u003e plugin -\u003e Langfuse\nuv run --extra dev --extra e2e pytest -m smoke\n```\n\nThe default `pytest` run excludes E2E and smoke tests and completes in under a second.\n\n#### Test tiers\n\nThe test suite is organized into four tiers, from fastest/simplest to slowest/most comprehensive:\n\n| Tier | Marker | Tests | What it tests | Requirements |\n|------|--------|-------|---------------|--------------|\n| Unit | (default) | 109 | Hook logic, tracer init, helpers, SpanTracker | None |\n| Integration | (default) | 19 | Full span export pipeline with InMemorySpanExporter, parent-child hierarchy, token roll-up, metrics | None |\n| E2E | `-m e2e` | 6 | OTLP export to real Phoenix/Langfuse, queried via GraphQL/REST API | Docker |\n| Smoke | `-m smoke` | 6 | Send real chats to hermes via OpenAI SDK, verify traces in Langfuse | hermes gateway + Langfuse |\n\n**Unit tests** (`tests/unit/`) cover:\n- `_safe_str`, `_to_int`, `_detect_session_kind` helper functions\n- `SpanTracker` class: span lifecycle, parent stack, end_all\n- `HermesOTelPlugin.init()` environment detection (Phoenix vs Langfuse vs LangSmith priority)\n- `NoopSpan` graceful degradation when OTel is unavailable\n- All 8 hook callbacks with mocked tracer (span names, attributes, metric recording, module-state management)\n\n**Integration tests** (`tests/integration/`) use a real OTel SDK with `InMemorySpanExporter` — no network needed:\n- Individual hook pairs produce correctly attributed spans\n- Parent-child nesting: Session \u003e LLM \u003e API \u003e Tool (verified via span context)\n- Full session lifecycle with token aggregation and session I/O roll-up\n- Metric counters and histograms via `InMemoryMetricReader`\n\n**E2E tests** (`tests/e2e/`) invoke hooks directly against real backends and query their APIs:\n- **Phoenix**: fires hooks, queries Phoenix GraphQL API at `/graphql` to verify spans\n- **Langfuse**: fires hooks, queries Langfuse REST API at `GET /api/public/observations` to verify observations\n\n**Smoke tests** (`tests/smoke/`) exercise the complete production pipeline:\n- **test_hermes_api**: verifies the hermes API server is functional (health, models, chat completion)\n- **test_hermes_langfuse**: sends real chats via OpenAI SDK to hermes, then queries Langfuse to confirm traces arrived with correct span names, tool spans, and token data\n\n#### E2E backends\n\n**Phoenix** — single container, starts in seconds:\n```bash\ndocker compose -f docker-compose/phoenix.yaml up -d\n# or let the test fixture start it automatically\n```\n\n**Langfuse** — full stack (Langfuse + Postgres + Redis + ClickHouse + MinIO), starts in ~60s:\n```bash\ndocker compose -f docker-compose/langfuse.yaml up -d\n# Pre-seeded API keys: lf_pk_test_hermes_otel / lf_sk_test_hermes_otel\n# UI at http://localhost:3000, OTEL endpoint at http://localhost:3000/api/public/otel\n```\n\nThe E2E fixtures will start/stop Docker services automatically if they aren't already running. If a service is already running on the expected port, it is reused.\n\n#### Smoke tests\n\nSmoke tests exercise the full pipeline end-to-end:\n\n```\nOpenAI SDK  --\u003e  hermes API server  --\u003e  LLM  --\u003e  OTEL plugin  --\u003e  Langfuse\n                 (port 8642)                       (hooks.py)        (port 3000)\n     \\                                                                   /\n      `--- pytest sends chat here                 pytest queries here ---`\n```\n\nThey require:\n\n1. **hermes-agent API server** running with the OTEL plugin loaded. Add to `~/.hermes/.env`:\n   ```\n   API_SERVER_ENABLED=true\n   ```\n   Then start the gateway:\n   ```bash\n   hermes gateway\n   ```\n2. **Langfuse** running with credentials configured in `~/.hermes/.env` (`OTEL_LANGFUSE_*` variables)\n\nTests skip automatically with a helpful message if either service is not reachable. The smoke tests poll the Langfuse observations API (up to 60-90s) to account for async trace ingestion.\n\n## Configuration\n\nYou can either pick **one** backend via environment variables (legacy mode,\nshown below), or fan **multiple** backends out in parallel via\n`config.yaml`. The two are mutually exclusive — when `backends:` is set in\nthe yaml file, env-var detection is skipped.\n\n### Multi-backend (`config.yaml`)\n\nA fully annotated template lives at [`config.yaml.example`](config.yaml.example)\nin the plugin root. Copy it to `config.yaml` and edit in place:\n\n```bash\ncp ~/.hermes/plugins/hermes_otel/config.yaml.example \\\n   ~/.hermes/plugins/hermes_otel/config.yaml\n```\n\n`config.yaml` is gitignored so local endpoints and (avoidable) secrets\nnever get committed. Only `config.yaml.example` is tracked. A minimal\nmulti-backend config looks like:\n\n```yaml\nbackends:\n  - type: phoenix\n    endpoint: http://localhost:6006/v1/traces\n  - type: jaeger\n    endpoint: http://localhost:4318/v1/traces\n  - type: tempo\n    endpoint: http://localhost:3200/v1/traces\n  - type: signoz\n    endpoint: http://localhost:4328/v1/traces\n    ingestion_key_env: OTEL_SIGNOZ_INGESTION_KEY   # secret from env\n  - type: langfuse\n    public_key_env: LANGFUSE_PUBLIC_KEY\n    secret_key_env: LANGFUSE_SECRET_KEY\n    base_url: https://cloud.langfuse.com\n  - type: otlp                                     # any other OTLP/HTTP collector\n    name: my-collector\n    endpoint: http://collector:4318/v1/traces\n    headers:\n      X-Auth: secret\n```\n\nEvery entry gets its own `BatchSpanProcessor` and (where supported) its own\n`PeriodicExportingMetricReader`. Each processor owns a background worker\nthread, so a slow or unreachable collector cannot block the agent's hot\npath or starve the others — span end is just a non-blocking enqueue. Both\ntrace and metrics export run in parallel across all configured backends.\n\nSupported `type` values: `phoenix`, `langfuse`, `signoz`, `jaeger`, `tempo`,\n`otlp`, `lgtm`, `uptrace`, `openobserve`. Use `otlp` for any collector\nthat doesn't have a dedicated type. Backends marked\ntraces-only (`langfuse`, `jaeger`, `tempo`) are auto-detected and skip\nthe metrics reader. Override with `metrics: true|false` per entry if\nneeded. See `config.yaml.example` for the full list of fields each type\naccepts — Uptrace takes a `dsn:` for the `uptrace-dsn` header, OpenObserve\ntakes `user:` / `password:` for HTTP Basic auth, and so on.\n\n### Full-conversation capture\n\nBy default the `llm.*` span's `input.value` is just the latest user turn.\nThe underlying `api.*` spans don't expose per-message detail. To see the\nentire message list the model actually saw, flip on\n`capture_conversation_history`:\n\n```yaml\ncapture_conversation_history: true\nconversation_history_max_chars: 40000   # safety cap; JSON is clipped with \"...\"\n```\n\nOr via env: `HERMES_OTEL_CAPTURE_CONVERSATION_HISTORY=true`. When enabled\nthe LLM span gets `input.value` = JSON-serialized history, `input.mime_type\n= application/json`, and `hermes.conversation.message_count`. Phoenix\npretty-prints the JSON in its Input panel; Langfuse / Jaeger / SigNoz show\nit as a large string. Respects the global `capture_previews` kill switch.\n\nSecrets should live in env vars (use the `*_env:` keys to reference them\nby name) rather than inline in yaml. LangSmith remains an env-var-only\nsingle-backend path; setting `LANGSMITH_TRACING=true` short-circuits the\nyaml backend list.\n\n### Single backend (env vars)\n\n**Pick one backend:**\n\n### Phoenix\n```bash\nexport OTEL_PHOENIX_ENDPOINT=\"http://localhost:6006/v1/traces\"\nexport OTEL_PROJECT_NAME=hermes-agent\n```\n\n### Langfuse\n```bash\n# Option A (plugin-specific vars):\nexport OTEL_LANGFUSE_PUBLIC_API_KEY=\"pk-lf-...\"\nexport OTEL_LANGFUSE_SECRET_API_KEY=\"sk-lf-...\"\n# Optional — defaults to EU cloud endpoint\nexport OTEL_LANGFUSE_ENDPOINT=\"https://cloud.langfuse.com/api/public/otel\"\n# For US region:\n# export OTEL_LANGFUSE_ENDPOINT=\"https://us.cloud.langfuse.com/api/public/otel\"\n\n# Option B (Langfuse-standard vars from docs):\n# export LANGFUSE_PUBLIC_KEY=\"pk-lf-...\"\n# export LANGFUSE_SECRET_KEY=\"sk-lf-...\"\n# export LANGFUSE_BASE_URL=\"https://cloud.langfuse.com\"  # or us.cloud/langfuse/self-hosted base URL\n```\n\n### LangSmith\n```bash\nexport LANGSMITH_TRACING=true\nexport LANGSMITH_API_KEY=\"lsv2_...\"\n# Optional — defaults to LangChain Cloud\nexport LANGSMITH_ENDPOINT=\"https://api.smith.langchain.com\"\n# Optional — project name for organizing traces\nexport LANGSMITH_PROJECT=\"hermes-langsmith-otel\"\n```\n\n\u003e **Note:** Install `langsmith` for better time-ordered run IDs: `pip install langsmith`. The plugin uses `langsmith.uuid7()` for run IDs when available, otherwise falls back to `uuid.uuid4()`.\n\n### SigNoz\n```bash\n# Self-hosted (see docker-compose/signoz/ — OTLP HTTP is remapped to 4328\n# to avoid colliding with Phoenix on 4318)\nexport OTEL_SIGNOZ_ENDPOINT=\"http://localhost:4328/v1/traces\"\nexport OTEL_PROJECT_NAME=hermes-agent\n\n# SigNoz Cloud — use the regional ingest URL + your ingestion key\n# export OTEL_SIGNOZ_ENDPOINT=\"https://ingest.us.signoz.cloud:443/v1/traces\"\n# export OTEL_SIGNOZ_INGESTION_KEY=\"sz-...\"\n```\n\nThe plugin sends both traces and metrics over OTLP/HTTP. When\n`OTEL_SIGNOZ_INGESTION_KEY` is set, the `signoz-ingestion-key` header is\nattached to both exporters.\n\n### Jaeger\n```bash\n# Jaeger ≥ 1.35 accepts OTLP/HTTP natively on port 4318\nexport OTEL_JAEGER_ENDPOINT=\"http://localhost:4318/v1/traces\"\nexport OTEL_PROJECT_NAME=hermes-otel-jaeger\n```\n\nJaeger is **traces-only** — the plugin skips metric export when this backend is selected. If you need token/tool/cost metrics alongside Jaeger traces, pair it with a Prometheus-compatible metrics sink or use a unified backend (Phoenix, SigNoz).\n\n### Grafana Tempo\n```bash\n# Tempo accepts OTLP/HTTP natively on port 4318\nexport OTEL_TEMPO_ENDPOINT=\"http://localhost:4318/v1/traces\"\nexport OTEL_PROJECT_NAME=hermes-otel-tempo\n```\n\nRun the upstream single-binary example (Tempo + MinIO + Grafana + Prometheus):\n\n```bash\ncd ~/git/grafana/tempo/example/docker-compose/single-binary\ndocker compose up -d\n# UI:   http://localhost:3000   (Grafana, anonymous admin)\n# OTLP: http://localhost:4318   (HTTP)  /  localhost:4317 (gRPC)\n```\n\nTempo is **traces-only** — the plugin skips metric export when this backend is selected. The upstream example already bundles Prometheus + Grafana, so token/tool/cost metrics can be routed there via a separate Prometheus remote-write or OTel collector if needed.\n\n### Honeycomb\n```bash\nexport HONEYCOMB_API_KEY=\"hcaik_...\"          # x-honeycomb-team header\n# Optional — defaults to the US ingest endpoint:\n# export OTEL_HONEYCOMB_ENDPOINT=\"https://api.eu1.honeycomb.io/v1/traces\"   # EU\nexport OTEL_PROJECT_NAME=hermes-otel-honeycomb\n```\n\nFor region selection (`us`/`eu`), a dataset, and the metrics `unknown_metrics`\ngotcha, use the multi-backend `config.yaml` form instead — see [HONEYCOMB.md](HONEYCOMB.md).\n\n### Optional\n```bash\nexport OTEL_PROJECT_NAME=\"hermes-agent\"   # Shown in Phoenix\nexport HERMES_OTEL_DEBUG=true             # Enable debug logging (see below)\n```\n\n### Debug logging\n\nThe plugin prints only essential startup messages (backend connected/failed, hook count) to stdout. For detailed per-span logging (span start/end, parent nesting, token counts, HTTP payloads), enable debug mode:\n\n```bash\nexport HERMES_OTEL_DEBUG=true\n```\n\nDebug output is written to `~/.hermes/plugins/hermes_otel/debug.log` and does not clutter hermes stdout.\n\n**Priority order:** LangSmith (if `LANGSMITH_TRACING=true`) \u003e Langfuse (if credentials set) \u003e SigNoz (`OTEL_SIGNOZ_ENDPOINT`) \u003e Uptrace (`OTEL_UPTRACE_ENDPOINT` + DSN) \u003e OpenObserve (`OTEL_OPENOBSERVE_ENDPOINT` + creds) \u003e Honeycomb (`HONEYCOMB_API_KEY`) \u003e Jaeger (`OTEL_JAEGER_ENDPOINT`) \u003e Tempo (`OTEL_TEMPO_ENDPOINT`) \u003e Phoenix (`OTEL_PHOENIX_ENDPOINT`).\n\n### Shaping knobs — `config.yaml` and `HERMES_OTEL_*` env vars\n\nBackend selection stays env-var-driven (above). For telemetry **shaping** — sampling, preview size, resource attributes, TTL, extra headers — you can also use a YAML file at `~/.hermes/plugins/hermes_otel/config.yaml`.\n\n**Precedence (per-field):** `HERMES_OTEL_*` env var \u003e `config.yaml` value \u003e default.\n\nExample `config.yaml`:\n\n```yaml\nenabled: true\nsample_rate: 0.25               # ParentBased(TraceIdRatioBased) — null/omit = sample everything\nroot_span_ttl_ms: 600000        # orphan-sweep threshold (10 min default)\nflush_interval_ms: 60000        # metrics export cadence\npreview_max_chars: 1200         # global clip_preview truncation fallback\n# Per-category overrides (each defaults to preview_max_chars when unset)\ntool_input_preview_max_chars: 1200\ntool_output_preview_max_chars: 2000\nllm_input_preview_max_chars: 1200\nllm_output_preview_max_chars: 1200\ncapture_previews: true          # false = suppress all input.value / output.value\ncapture_sender_id: false        # true = add platform-prefixed user.id to spans\nproject_name: hermes-prod       # supersedes OTEL_PROJECT_NAME\nglobal_tags:\n  team: platform\nresource_attributes:            # merged into Resource; overrides global_tags on key conflict\n  env: prod\n  region: us-east-1\nheaders:                        # merged onto outgoing OTLP requests\n  X-Scope-OrgID: tenant-a\n```\n\nEvery field can be overridden by env var with prefix `HERMES_OTEL_` (scalars only):\n\n| Field | Env var |\n|---|---|\n| `enabled` | `HERMES_OTEL_ENABLED` (`true`/`false`) |\n| `sample_rate` | `HERMES_OTEL_SAMPLE_RATE` (float 0..1, or `0` to disable) |\n| `root_span_ttl_ms` | `HERMES_OTEL_ROOT_SPAN_TTL_MS` |\n| `flush_interval_ms` | `HERMES_OTEL_FLUSH_INTERVAL_MS` |\n| `preview_max_chars` | `HERMES_OTEL_PREVIEW_MAX_CHARS` |\n| `tool_input_preview_max_chars` | `HERMES_OTEL_TOOL_INPUT_PREVIEW_MAX_CHARS` |\n| `tool_output_preview_max_chars` | `HERMES_OTEL_TOOL_OUTPUT_PREVIEW_MAX_CHARS` |\n| `llm_input_preview_max_chars` | `HERMES_OTEL_LLM_INPUT_PREVIEW_MAX_CHARS` |\n| `llm_output_preview_max_chars` | `HERMES_OTEL_LLM_OUTPUT_PREVIEW_MAX_CHARS` |\n| `capture_previews` | `HERMES_OTEL_CAPTURE_PREVIEWS` |\n| `capture_sender_id` | `HERMES_OTEL_CAPTURE_SENDER_ID` |\n| `project_name` | `HERMES_OTEL_PROJECT_NAME` |\n| `span_batch_max_queue_size` | `HERMES_OTEL_SPAN_BATCH_MAX_QUEUE_SIZE` |\n| `span_batch_schedule_delay_ms` | `HERMES_OTEL_SPAN_BATCH_SCHEDULE_DELAY_MS` |\n| `span_batch_max_export_batch_size` | `HERMES_OTEL_SPAN_BATCH_MAX_EXPORT_BATCH_SIZE` |\n| `span_batch_export_timeout_ms` | `HERMES_OTEL_SPAN_BATCH_EXPORT_TIMEOUT_MS` |\n| `force_flush_on_session_end` | `HERMES_OTEL_FORCE_FLUSH_ON_SESSION_END` |\n\n`pyyaml` is optional — if not installed, the YAML file is silently skipped and only env vars + defaults apply. Malformed YAML logs a single warning and falls back to defaults.\n\n#### Privacy mode\n\nSet `capture_previews: false` (or `HERMES_OTEL_CAPTURE_PREVIEWS=false`) to suppress every `input.value` / `output.value` attribute. Useful for shared deployments where message content can't leave the process. A one-line startup banner confirms the mode is active.\n\nSet `capture_sender_id: true` (or `HERMES_OTEL_CAPTURE_SENDER_ID=true`) to attach gateway sender identity to spans. The plugin emits the raw platform ID as `hermes.sender.id` and the backend-neutral user key as `user.id={platform}:{sender_id}`. For example, Slack user `U0B074344DP` becomes `user.id=slack:U0B074344DP`. The platform is already available on LLM spans as `llm.provider`. This is opt-in because IDs from Discord, Telegram, Slack, email, SMS, and similar platforms can identify users. CLI sessions usually omit it.\n\n### Per-turn summary attributes\n\nOn `on_session_end`, the root session/agent span is enriched with a summary of what happened in the turn — so dashboards don't need to JOIN across spans.\n\n| Attribute | Type | Meaning |\n|---|---|---|\n| `hermes.turn.tool_count` | int | distinct tool names invoked |\n| `hermes.turn.tools` | string | sorted CSV of distinct tool names (≤500 chars) |\n| `hermes.turn.tool_targets` | string | `\\|`-joined distinct file paths / URLs |\n| `hermes.turn.tool_commands` | string | `\\|`-joined distinct shell commands |\n| `hermes.turn.tool_outcomes` | string | sorted CSV of distinct outcome statuses |\n| `hermes.turn.skill_count` | int | distinct skill names inferred |\n| `hermes.turn.skills` | string | sorted CSV of distinct skill names |\n| `hermes.turn.api_call_count` | int | `pre_api_request` hook invocations |\n| `hermes.turn.final_status` | string | `completed` \\| `interrupted` \\| `incomplete` \\| `timed_out` |\n\nZero/empty aggregators are omitted rather than emitted as empty strings.\n\n### Tool identity, outcome, skill inference\n\nEach `tool.*` span now also carries:\n\n- `hermes.tool.target` — first non-empty value under args.`path` / `file_path` / `target` / `url` / `uri`.\n- `hermes.tool.command` — first non-empty value under args.`command` / `cmd`.\n- `hermes.tool.outcome` — one of `completed` · `error` · `timeout` · `blocked` · (explicit `status` field from the result, lowercased). Only `error` maps the span `StatusCode` to `ERROR`; timeouts/blocked stay `OK` so dashboards don't count them as failures.\n- `hermes.skill.name` — inferred from args paths matching `/skills/\u003cname\u003e/`. Does **not** match `/optional-skills/\u003cname\u003e/references/`. Also increments a `hermes.skill.inferred{skill_name, source}` counter so ops can audit hit rates.\n\n### Orphan-span sweep\n\nIf a session never fires `on_session_end` (e.g. host crash mid-turn), it would otherwise leak active-span state. A TTL-based sweeper (default 10 min, configurable via `root_span_ttl_ms`) runs at the top of every `pre_*` hook; sessions older than the TTL are finalized with `hermes.turn.final_status=timed_out` and span status `OK` (not `ERROR` — timeouts should not pollute error rates).\n\n### Non-blocking span export\n\nSpans are exported via OpenTelemetry's `BatchSpanProcessor`: `span.end()` enqueues the span to a bounded in-memory queue, and a background worker drains that queue in batches on a timer. This means a slow or unreachable OTLP backend no longer adds latency to every tool call / API request.\n\n**Export cadence:**\n- Background worker flushes every `span_batch_schedule_delay_ms` (default 1s).\n- At the end of each session (`on_session_end`), the plugin force-flushes so traces appear in the UI immediately rather than after the worker's next cycle. Disable with `force_flush_on_session_end: false` if you prefer to let the worker handle it.\n- On graceful process shutdown, an `atexit` handler flushes the queue once so nothing is lost.\n\n**Backpressure:** the queue is bounded by `span_batch_max_queue_size` (default 2048). If the agent outruns the exporter, the oldest enqueued spans are dropped — hermes keeps running rather than stalling.\n\n**Crash vs. graceful exit:** up to `schedule_delay_millis` worth of spans may be lost on a hard crash (SIGKILL, OOM). This is the standard OTel trade-off and mirrors every production tracing stack. Graceful shutdown (`hermes gateway stop`, SIGTERM) triggers the atexit flush.\n\n## How it works\n\nHermes fires lifecycle hooks. This plugin maps them to OTel spans:\n\n```\nTurn 1:\n  agent / cron (root, when session hooks are available)\n  ├── skill.{name} span (when a skill is loaded; spans load → turn end)\n  └── LLM span\n      └── API span (first call → stop or tool_calls)\n          └── Tool span(s) (if tools called)\n      ├── subagent.{role} span (when delegate_task is used)\n      │   └── agent (child run rejoins the trace) → its own LLM/API/tool spans\n      └── API span (second call → final response)\n```\n\n### Span hierarchy\n\n| Span | Kind | Contains |\n|------|------|----------|\n| `agent` / `cron` | AGENT | Session metadata, completion/interruption status, turn summary |\n| `skill.{name}` | SKILL | A skill loaded during the turn — `hermes.skill.name`, `hermes.skill.source` (`skill_view`/`path_match`), `hermes.skill.path`, `hermes.skill.result_status`. Spans from load to turn end; skills overlap freely |\n| `llm.{model}` | LLM | Model name, provider, user message (input), assistant response (output) |\n| `api.{model}` | LLM | Token counts (prompt + completion), duration, finish reason, cache tokens. On failure: `ERROR` status + recorded exception + retry metadata |\n| `tool.{name}` | TOOL | Tool name, arguments (input), result (output), error status |\n| `approval.{pattern}` | APPROVAL | Human-in-the-loop approval prompt — the **human-decision wait time**, `hermes.approval.choice` (`once`/`session`/`always`/`deny`/`timeout`), correlated to the gated tool via `gen_ai.tool.call.id` |\n| `subagent.{role}` | AGENT | Delegated child agent — role, goal, status, duration, summary; the child's own run nests beneath it so a multi-agent run is **one connected trace** |\n\n**Human-in-the-loop approvals:** when a tool trips a dangerous-command approval rule, the agent blocks waiting for a human (often the dominant chunk of a turn's wall-clock, previously invisible). The plugin opens an `approval.*` span (via the `pre_approval_request` / `post_approval_response` hooks) capturing the wait time and the decision (approve/deny/timeout) — works on CLI and gateway surfaces (Telegram, Discord, …), and feeds `hermes.approval.count` / `hermes.approval.duration` metrics for deny/timeout-rate tracking. Observer-only: telemetry never alters an approval. See [Span hierarchy → `approval.*`](website/docs/architecture/span-hierarchy.md#approval).\n\n**Sub-agent delegation:** when the agent calls `delegate_task`, the plugin opens a `subagent.{role}` span in the parent trace (via the `subagent_start` / `subagent_stop` hooks) and rejoins the delegated child's own root span underneath it. Without this, child agents export as dozens of disconnected traces. See [Span hierarchy → `subagent.*`](website/docs/architecture/span-hierarchy.md) and the `hermes.subagent.count` / `hermes.subagent.duration` metrics.\n\n**Skill execution windows:** when the agent loads a skill (the `skill_view` tool, or a read of a `/skills/\u003cname\u003e/` file), the plugin opens a `skill.{name}` span that runs until the turn ends — so a trace shows *which skills were active and for how long*, with overlaps. Controlled by `skill_spans` (default on). The plugin also ships a companion Hermes skill, `hermes_otel:observability` (load it with `skill_view`), which explains how to turn on and read this telemetry — and because skill loads are instrumented, opening it emits its own `skill.observability` span. See [Span hierarchy → `skill.*`](website/docs/architecture/span-hierarchy.md#skill).\n\n**API errors \u0026 retries:** failed provider requests (rate limits, timeouts, 5xx, network errors) close the `api.{model}` span as `ERROR` with an `exception` event and retry metadata (`error.type`, status code, `hermes.retry.count`, `hermes.retryable`), via the `api_request_error` hook — previously these ended `OK` and were invisible. Emits `hermes.api.error.count{error_type,status_class,retryable}` and `hermes.retry.count`.\n\n### Attribute conventions\n\nThe plugin emits **dual-convention** attributes so both backends work:\n\n| Metric | OpenTelemetry GenAI | Phoenix (OpenInference) |\n|--------|---------------------|------------------------|\n| Prompt tokens | `gen_ai.usage.input_tokens` | `llm.token_count.prompt` |\n| Completion tokens | `gen_ai.usage.output_tokens` | `llm.token_count.completion` |\n| Total tokens | `gen_ai.usage.total_tokens` | `llm.token_count.total` |\n| Cache read | `gen_ai.usage.cache_read.input_tokens` (`gen_ai.usage.cache_read_input_tokens` also kept) | `llm.token_count.prompt_details.cache_read` |\n| Cache write | `gen_ai.usage.cache_creation.input_tokens` (`gen_ai.usage.cache_creation_input_tokens` also kept) | `llm.token_count.prompt_details.cache_write` |\n| Reasoning | `gen_ai.usage.reasoning.output_tokens` | `llm.token_count.completion_details.reasoning` |\n\nReasoning (\"thinking\") tokens are emitted only by reasoning-capable models that report them. They are a **subset of the completion/output count** — already included in the completion and total figures — and are surfaced separately for visibility, so they should not be added on top of the total.\n\nLLM and API spans also expose standard GenAI request/response metadata where Hermes provides it, including `gen_ai.provider.name`, `gen_ai.request.model`, request parameters such as `gen_ai.request.temperature`, and response fields such as `gen_ai.response.model` and `gen_ai.response.finish_reasons`.\n\nPhoenix uses `input.value` and `output.value` for previews. When full prompt/response capture is explicitly enabled, the plugin also writes the corresponding GenAI content attributes (`gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.system_instructions`).\n\n## Trace propagation to MCP servers\n\nThe plugin forwards the active span's W3C `traceparent` on outbound MCP HTTP requests so an OTel-instrumented MCP server's spans join the **same trace** as the agent. It registers an `mcp_request_headers` hook (only when the host Hermes advertises it — a silent no-op otherwise) and exposes a public helper:\n\n```python\nfrom hermes_plugins.hermes_otel.hooks import get_current_traceparent\ntraceparent = get_current_traceparent(session_id)   # \"00-\u003ctrace\u003e-\u003cspan\u003e-01\" or None\n```\n\nSee [docs: MCP trace propagation](website/docs/configuration/mcp-trace-propagation.md).\n\n## File structure\n\n| File | Role |\n|------|------|\n| `plugin.yaml` | Plugin manifest — declares hooks to Hermes |\n| `__init__.py` | Entry point — initializes tracer, registers core hooks (+ session and sub-agent hooks when supported) |\n| `tracer.py` | OTel TracerProvider setup, span lifecycle management, parent/child tracking |\n| `hooks.py` | Hook implementations — maps Hermes events to OTel spans with attributes |\n| `debug_utils.py` | Optional debug logging and secret masking |\n| `docker-compose/` | Docker Compose files for Phoenix and Langfuse backends |\n| `tests/unit/` | Unit tests — helpers, SpanTracker, tracer init, hook callbacks |\n| `tests/integration/` | Integration tests — InMemorySpanExporter, span hierarchy, metrics |\n| `tests/e2e/` | E2E tests — real Phoenix/Langfuse via Docker |\n| `tests/smoke/` | Smoke tests — full pipeline through hermes API server to Langfuse |\n\n## Roadmap: additional backends\n\nThis plugin speaks plain OTLP/HTTP, so any OTLP-compatible backend should work today with no code changes — just point `OTEL_EXPORTER_OTLP_ENDPOINT` at it. The list below tracks backends I plan to formally test, add a `docker-compose/` file for, and (where applicable) cover with a smoke test.\n\n**Status legend:** ✅ supported \u0026 tested · 🟡 should work, not yet tested/documented · 🔲 planned\n\n| Backend | Signals | Deployment | Account / cost | Status |\n|---------|---------|------------|----------------|--------|\n| [Phoenix](https://github.com/Arize-ai/phoenix) | traces | Local (docker) · Arize AX cloud | OSS, no account · commercial cloud | ✅ |\n| [Langfuse](https://langfuse.com) | traces | Local (docker compose) · Cloud | OSS, no account · free tier + paid | ✅ |\n| [LangSmith](https://smith.langchain.com) | traces | Cloud only (self-host = enterprise) | Free personal tier · paid tiers | ✅ |\n| [Jaeger](https://www.jaegertracing.io) | traces | Local (single container) | OSS, no account needed | ✅ |\n| [SigNoz](https://signoz.io) | traces + metrics + logs | Local (docker compose) · Cloud | OSS, no account · free tier + paid cloud | ✅ |\n| [Grafana Tempo](https://grafana.com/oss/tempo/) | traces | Local (docker compose) · Grafana Cloud | OSS, no account · free tier + paid cloud | ✅ |\n| [Grafana LGTM](https://github.com/grafana/docker-otel-lgtm) | traces + metrics + logs | Local (single container) | OSS, no account | ✅ |\n| [OpenObserve](https://openobserve.ai) | traces + metrics + logs | Local (single binary / docker) · Cloud | OSS, no account · free tier + paid cloud | ✅ |\n| [Uptrace](https://uptrace.dev) | traces + metrics + logs | Local (docker compose) · Cloud | OSS, no account · free tier + paid cloud | ✅ |\n| [Honeycomb](https://www.honeycomb.io) | traces + metrics | Cloud only | Free tier + paid | 🔲 |\n| [New Relic](https://newrelic.com) | traces + metrics + logs | Cloud only | Free tier (100 GB/mo) + paid | 🔲 |\n| [Elastic APM](https://www.elastic.co/observability/application-performance-monitoring) | traces + metrics + logs | Local (docker) · Elastic Cloud | OSS self-host · trial + paid cloud | 🔲 |\n| [Datadog](https://www.datadoghq.com) | traces + metrics + logs | Cloud only | Trial only, paid thereafter | 🔲 |\n\n### Quick picks\n\n- **Fully offline / no account ever:** Phoenix, Langfuse (self-hosted), Jaeger, SigNoz, Grafana Tempo+Mimir, OpenObserve, Uptrace, Elastic APM self-host. All runnable via `docker compose up`.\n- **Free SaaS (personal / hobby tier, no credit card):** Langfuse Cloud, LangSmith, SigNoz Cloud, Grafana Cloud, Honeycomb, New Relic. Best if you don't want to run infrastructure.\n- **Paid only (credit card required after trial):** Datadog, Dynatrace, LangSmith self-hosted (enterprise plan).\n\n\u003e Free-tier limits change frequently — check each vendor's pricing page before committing. The table reflects what's advertised as of this writing.\n\n### Signals note\n\nJaeger and Tempo are both **traces only**. If you want both spans and the token/tool/cost metrics this plugin emits (via `PeriodicExportingMetricReader`), pair them with Prometheus, or pick one of the traces+metrics backends above.\n\n## Current limitations\n\n- **No full prompt capture** — Hermes hooks don't expose the fully-formed prompt (system message + conversation history + tool results) to plugins. API spans only receive metadata (token counts, model, duration). The raw user message and assistant response appear on the parent LLM span.\n- **Langfuse auth** — Requires both public and secret keys; Basic Auth is constructed automatically. If only one key is set, Langfuse mode won't activate.\n- **No gRPC** — Only OTLP over HTTP/JSON is used. gRPC exporters are not included.\n- **Single session per run** — Span tracking is in-memory; if Hermes restarts mid-session, active spans are lost. A TTL-based sweeper finalizes abandoned sessions (see \"Orphan-span sweep\" above), but the orphaned process's buffered spans still need a graceful `atexit` to flush.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbriancaffey%2Fhermes-otel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbriancaffey%2Fhermes-otel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbriancaffey%2Fhermes-otel/lists"}