{"id":47664213,"url":"https://github.com/hops-ops/observe-stack","last_synced_at":"2026-04-02T11:49:41.264Z","repository":{"id":339458310,"uuid":"1147573293","full_name":"hops-ops/observe-stack","owner":"hops-ops","description":"Crossplane configuration: complete observability stack composing kube-prometheus-stack, Loki, Tempo, k8s-monitoring, and Grafana Operator","archived":false,"fork":false,"pushed_at":"2026-03-17T18:11:02.000Z","size":56,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-18T07:48:09.823Z","etag":null,"topics":["crossplane","crossplane-configuration","grafana","kubernetes","loki","monitoring","observability","platform-engineering","prometheus","tempo"],"latest_commit_sha":null,"homepage":null,"language":"KCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hops-ops.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-02T00:09:16.000Z","updated_at":"2026-03-17T18:07:17.000Z","dependencies_parsed_at":"2026-02-19T22:00:42.933Z","dependency_job_id":null,"html_url":"https://github.com/hops-ops/observe-stack","commit_stats":null,"previous_names":["hops-ops/stack-observe","hops-ops/observe-stack"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/hops-ops/observe-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hops-ops%2Fobserve-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hops-ops%2Fobserve-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hops-ops%2Fobserve-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hops-ops%2Fobserve-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hops-ops","download_url":"https://codeload.github.com/hops-ops/observe-stack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hops-ops%2Fobserve-stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31305809,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T09:48:21.550Z","status":"ssl_error","status_checked_at":"2026-04-02T09:48:19.196Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crossplane","crossplane-configuration","grafana","kubernetes","loki","monitoring","observability","platform-engineering","prometheus","tempo"],"created_at":"2026-04-02T11:49:40.544Z","updated_at":"2026-04-02T11:49:41.259Z","avatar_url":"https://github.com/hops-ops.png","language":"KCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# observe-stack\n\nA single Crossplane resource that deploys a complete, production-wired observability stack: metrics, logs, traces, cost monitoring, and Grafana dashboards — all pre-integrated.\n\n## Why Observe?\n\n**Without Observe:**\n- 5+ Helm charts to install, configure, and maintain independently\n- Cross-component wiring is manual and error-prone (wrong URLs, wrong ports, missing datasources)\n- No deletion ordering — removing Prometheus before k8s-monitoring breaks metric collection silently\n- Grafana datasources configured by hand, often missing trace-to-log correlation\n- Upgrading one chart risks breaking integration with the others\n\n**With Observe:**\n- One resource, one API surface, all five components wired together automatically\n- Grafana datasources pre-configured with full trace-to-log and trace-to-metric correlation\n- Safe deletion ordering enforced via Usage resources (5 dependency edges)\n- Cross-component URLs derived from release names — rename a component and everything adjusts\n- Override any chart value while keeping cross-component defaults intact\n\n## What Gets Deployed\n\n```\n                  ┌─────────────────────────────────────┐\n                  │            Observe XR                │\n                  └──────────────┬──────────────────────┘\n                                 │\n         ┌──────────┬────────────┼────────────┬──────────────┐\n         ▼          ▼            ▼            ▼              ▼\n   ┌──────────┐ ┌──────┐  ┌──────────┐ ┌───────────┐ ┌────────────┐\n   │kube-prom │ │ Loki │  │  Tempo   │ │k8s-monitor│ │  Grafana   │\n   │  -stack  │ │      │  │          │ │   -ing    │ │  Operator  │\n   │(metrics) │ │(logs)│  │ (traces) │ │(collection│ │  (CRDs)    │\n   └──────────┘ └──────┘  └──────────┘ │+ OpenCost)│ └────────────┘\n                                       └───────────┘\n                                             │\n                  ┌──────────────────────────┘\n                  ▼\n   ┌──────────────────────────────────────────────┐\n   │  Grafana CR + 3 Datasource CRs              │\n   │  (Prometheus, Loki, Tempo — with correlation)│\n   └──────────────────────────────────────────────┘\n```\n\n**16 composed resources:** 5 Helm Releases + 6 Kubernetes Objects + 5 Usage protections\n\n| Component | Chart | Version | Purpose |\n|-----------|-------|---------|---------|\n| kube-prometheus-stack | prometheus-community | 82.2.0 | Prometheus, AlertManager, Grafana |\n| loki | grafana | 6.53.0 | Log aggregation (SingleBinary default) |\n| tempo | grafana | 1.24.4 | Distributed tracing (OTLP, Jaeger, Zipkin) |\n| k8s-monitoring | grafana | 3.8.0 | Collection via Alloy + OpenCost |\n| grafana-operator | grafana (OCI) | 5.21.4 | Grafana CRD management |\n\n## The Journey\n\n### Stage 1: Getting Started\n\nOne field required. Everything else has sensible defaults.\n\n```yaml\napiVersion: hops.ops.com.ai/v1alpha1\nkind: ObserveStack\nmetadata:\n  name: observe\n  namespace: default\nspec:\n  clusterName: my-cluster\n```\n\nThis deploys all 5 components into the `monitoring` namespace with:\n- Prometheus scraping all ServiceMonitors/PodMonitors cluster-wide\n- Loki in SingleBinary mode with filesystem storage\n- Tempo accepting OTLP, Jaeger, and Zipkin traces\n- k8s-monitoring collecting cluster metrics, pod logs, events, and cost data via OpenCost\n- Grafana with Loki/Tempo datasources pre-wired (including trace-to-log correlation)\n\n### Stage 2: Customizing for Your Team\n\nAdd labels, tune Grafana, adjust component settings.\n\n```yaml\napiVersion: hops.ops.com.ai/v1alpha1\nkind: ObserveStack\nmetadata:\n  name: observe\n  namespace: default\nspec:\n  clusterName: production-cluster\n  namespace: monitoring\n  labels:\n    team: platform\n  kubePrometheusStack:\n    values:\n      grafana:\n        adminPassword: changeme\n      prometheus:\n        prometheusSpec:\n          retention: 30d\n          storageSpec:\n            volumeClaimTemplate:\n              spec:\n                accessModes: [\"ReadWriteOnce\"]\n                resources:\n                  requests:\n                    storage: 50Gi\n  loki:\n    values:\n      loki:\n        storage:\n          type: s3\n          s3:\n            bucketnames: my-loki-bucket\n            region: us-east-1\n  k8sMonitoring:\n    values:\n      nodeExporter:\n        enabled: false\n```\n\n### Stage 3: Local Development\n\nFor Colima/kind/minikube — use `default` provider configs instead of cluster-named ones.\n\n```yaml\napiVersion: hops.ops.com.ai/v1alpha1\nkind: ObserveStack\nmetadata:\n  name: observe\n  namespace: default\nspec:\n  clusterName: local\n  helmProviderConfigRef:\n    name: default\n  kubernetesProviderConfigRef:\n    name: default\n  kubePrometheusStack:\n    values:\n      grafana:\n        adminPassword: local\n```\n\n### Stage 4: Full Override\n\nWhen you need complete control over a component's Helm values (bypassing all defaults):\n\n```yaml\nspec:\n  kubePrometheusStack:\n    overrideAllValues:\n      grafana:\n        enabled: false\n      prometheus:\n        prometheusSpec:\n          remoteWrite:\n            - url: https://mimir.example.com/api/v1/push\n```\n\n`overrideAllValues` replaces **all** defaults for that component — chart defaults, cross-component wiring, everything. Use `values` for additive changes instead.\n\n## Cross-Component Wiring\n\nThese integrations happen automatically:\n\n| From | To | What |\n|------|----|------|\n| Grafana | Loki | Datasource with `derivedFields` for trace ID extraction |\n| Grafana | Tempo | Datasource with `tracesToLogsV2`, `serviceMap`, `nodeGraph` |\n| Grafana | Prometheus | Default datasource |\n| Tempo | Prometheus | Metrics generator remote-write |\n| k8s-monitoring | Prometheus | Metrics push via `/api/v1/write` |\n| k8s-monitoring | Loki | Logs push via gateway `/loki/api/v1/push` |\n| k8s-monitoring | Tempo | Traces push via OTLP gRPC `:4317` |\n| OpenCost | Prometheus | Cost queries via `/api/v1/query` (OpenCost appends this path) |\n\n## Creation Order\n\nResources are created as their dependencies become ready:\n\n```mermaid\ngraph TD\n    XR[Observe XR] --\u003e kps[kube-prometheus-stack]\n    XR --\u003e loki[loki]\n    XR --\u003e tempo[tempo]\n    XR --\u003e k8smon[k8s-monitoring]\n    XR --\u003e grafop[grafana-operator]\n\n    grafop -.-\u003e|ready| instance[grafana-instance]\n    instance -.-\u003e|ready| ds-prom[datasource-prometheus]\n    instance -.-\u003e|ready| ds-loki[datasource-loki]\n    instance -.-\u003e|ready| ds-tempo[datasource-tempo]\n    instance -.-\u003e|ready| dash-overview[dashboard-opencost-overview]\n    instance -.-\u003e|ready| dash-ns[dashboard-opencost-namespace]\n```\n\nAll 5 Helm releases start immediately. Grafana CRs (instance, datasources, dashboards) wait for the operator to be ready.\n\n## Deletion Order\n\nUsage resources enforce safe teardown — dependents delete before the resources they depend on:\n\n```mermaid\ngraph LR\n    ds-prom[datasource-prometheus] --\u003e|blocks| instance[grafana-instance]\n    ds-loki[datasource-loki] --\u003e|blocks| instance\n    ds-tempo[datasource-tempo] --\u003e|blocks| instance\n    instance --\u003e|blocks| grafop[grafana-operator]\n    grafop --\u003e|blocks| kps[kube-prometheus-stack]\n\n    k8smon[k8s-monitoring] -.- free1[ ]\n    loki[loki] -.- free2[ ]\n    tempo[tempo] -.- free3[ ]\n\n    style free1 fill:none,stroke:none\n    style free2 fill:none,stroke:none\n    style free3 fill:none,stroke:none\n```\n\n| Phase | Deletes | Waits for |\n|-------|---------|-----------|\n| 1 | k8s-monitoring, loki, tempo, dashboards | nothing — immediate |\n| 2 | datasources | nothing — immediate |\n| 3 | grafana-instance | datasources gone |\n| 4 | grafana-operator | grafana-instance gone |\n| 5 | kube-prometheus-stack | grafana-operator gone |\n\nThe grafana chain ensures CRDs (managed by grafana-operator, installed by kps) stay alive until all CRs are cleaned up.\n\n## Sending Traces\n\nApplications send traces to the Alloy receiver:\n\n| Protocol | Endpoint |\n|----------|----------|\n| OTLP gRPC | `k8s-monitoring-alloy-receiver.monitoring:4317` |\n| OTLP HTTP | `k8s-monitoring-alloy-receiver.monitoring:4318` |\n\nOr directly to Tempo:\n\n| Protocol | Endpoint |\n|----------|----------|\n| OTLP gRPC | `tempo.monitoring:4317` |\n| OTLP HTTP | `tempo.monitoring:4318` |\n| Jaeger gRPC | `tempo.monitoring:14250` |\n| Zipkin | `tempo.monitoring:9411` |\n\n## Spec Reference\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `clusterName` | string | yes | — | Target cluster name; defaults provider config refs |\n| `namespace` | string | no | `monitoring` | Shared namespace for all components |\n| `labels` | map | no | `{}` | Custom labels merged with defaults |\n| `managementPolicies` | []string | no | `[\"*\"]` | Crossplane management policies |\n| `helmProviderConfigRef.name` | string | no | `clusterName` | Helm ProviderConfig name |\n| `helmProviderConfigRef.kind` | string | no | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` |\n| `kubernetesProviderConfigRef.name` | string | no | `clusterName` | Kubernetes ProviderConfig name |\n| `kubernetesProviderConfigRef.kind` | string | no | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` |\n| `\u003ccomponent\u003e.name` | string | no | chart name | Helm release name |\n| `\u003ccomponent\u003e.namespace` | string | no | `namespace` | Per-component namespace override |\n| `\u003ccomponent\u003e.values` | object | no | `{}` | Helm values merged with defaults |\n| `\u003ccomponent\u003e.overrideAllValues` | object | no | `{}` | Helm values replacing all defaults |\n\nComponents: `kubePrometheusStack`, `loki`, `tempo`, `k8sMonitoring`, `grafanaOperator`\n\n## Status\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `status.ready` | boolean | `true` when all composed resources report Ready |\n\n## Dependencies\n\n| Kind | Package | Version |\n|------|---------|---------|\n| Function | crossplane-contrib/function-auto-ready | \u003e=v0.6.0 |\n| Provider | crossplane-contrib/provider-kubernetes | \u003e=v1 |\n| Provider | crossplane-contrib/provider-helm | \u003e=v1 |\n\n## Development\n\n```bash\nmake render          # Render all examples\nmake render:minimal  # Render a single example\nmake validate        # Validate all rendered output\nmake test            # Run KCL unit tests (11 tests)\nmake e2e             # Run E2E tests against a live cluster\nmake build           # Build the Crossplane package\nmake publish tag=v1  # Build and push to registry\n```\n\n## License\n\nApache-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhops-ops%2Fobserve-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhops-ops%2Fobserve-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhops-ops%2Fobserve-stack/lists"}