{"id":51287604,"url":"https://github.com/tcarland/grafana-kube-stack","last_synced_at":"2026-06-30T07:30:25.930Z","repository":{"id":362088293,"uuid":"971973508","full_name":"tcarland/grafana-kube-stack","owner":"tcarland","description":"A Repository for installing and managing the *kustomization* of the various helm charts making up the Grafana (LGTM) Ecosystem.","archived":false,"fork":false,"pushed_at":"2026-06-25T14:18:56.000Z","size":984,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-25T15:27:58.894Z","etag":null,"topics":["grafana","lgtm-stack","prometheus-operator"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tcarland.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-24T10:45:00.000Z","updated_at":"2026-06-25T14:18:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tcarland/grafana-kube-stack","commit_stats":null,"previous_names":["tcarland/grafana-kube-stack"],"tags_count":38,"template":false,"template_full_name":null,"purl":"pkg:github/tcarland/grafana-kube-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcarland%2Fgrafana-kube-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcarland%2Fgrafana-kube-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcarland%2Fgrafana-kube-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcarland%2Fgrafana-kube-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tcarland","download_url":"https://codeload.github.com/tcarland/grafana-kube-stack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tcarland%2Fgrafana-kube-stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34957626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-30T02:00:05.919Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grafana","lgtm-stack","prometheus-operator"],"created_at":"2026-06-30T07:30:25.254Z","updated_at":"2026-06-30T07:30:25.918Z","avatar_url":"https://github.com/tcarland.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"Grafana Stack on Kubernetes\n===========================\nv26.06.26\n\nCopyright (c)2025-2026 Timothy C. Arland \u003ctcarland at gmail dot com\u003e\n\nSteps for customizing and deploying the [Grafana](https://grafana.com)\nEcosystem, consisting of Loki, Grafana, Tempo, and Mimir; the (LGTM) stack.\nThis project also includes deploying the [Prometheus](https://prometheus.io)\nCommunity chart.\n\n\n# Table of Contents\n\n- [Grafana Stack on Kubernetes](#grafana-stack-on-kubernetes)\n  * [Overview](#overview)\n    + [Data Flow](#data-flow)\n    + [Components Matrix](#components-matrix)\n    + [Architecture and Documentation](#architecture-and-documentations)\n    + [Requirements](#requirements)\n    + [Deployment Configuration](#deployment-configuration)\n    + [S3 Buckets](#s3-buckets)\n  * [Mimir](#mimir)\n    + [Mimir Ingress](#mimir-ingress)\n  * [Prometheus Operator](#prometheus-operator)\n    + [Prometheus Ingress](#prometheus-ingress)\n  * [Loki](#loki)\n    + [Loki Ingress](#loki-ingress)\n  * [Tempo](#tempo)\n    + [Tempo Ingress](#tempo-ingress)\n  * [Grafana](#grafana)\n    + [High Availability](#high-availability)\n    + [Grafana Ingress](#grafana-ingress)\n  * [Alloy](#alloy)\n    + [Ansible Deployment](#ansible-deployment)\n  * [Additional Document References](#additional-document-references)\n  * [Additional Notes](#additional-notes)\n\n\u003cbr\u003e\n\n---\n\n# Overview\n\nThis repository serves as a means for cleaner handling of secrets and\nenvironment configuration requirements to automate *helm* values\ngeneration.\n\nGiven the flexible pattern of handling various environment\nconfigurations with *kustomize*, the project uses the `--enable-helm`\nfeature of *kustomize* to manage environment *overlays* combining the\nuse of *kustomize* and *helm*; essentially acting as a wrapper to the\n*Grafana* and *Prometheus Community* helm charts.\n\n- [Grafana OSS Charts](https://github.com/grafana-community/helm-charts)\n- [Prometheus Community Charts](https://github.com/prometheus-community/helm-chart)\n\nIn addition to the OSS versions we can support the official Grafana, \nincluding Grafana Enterprise versions.\n\n- [Grafana Charts](https://github.com/grafana/helm-charts)\n\n\n\n## Data Flow\n```\n                    ┌───────────────────────────────┐\n                    │         Data Sources          │\n                    │ ───────────────────────────── │\n                    │ • Applications \u0026 Services     │\n                    │ • Kubernetes Logs             │\n                    │ • Prometheus Exporters        │\n                    │ • Tracing Instrumentation     │\n                    └──────────────┬────────────────┘\n                                   │\n             ┌─────────────────────┼─────────────────────┐\n             │                     │                     │\n             ▼                     ▼                     ▼\n   ┌────────────────┐    ┌────────────────┐     ┌────────────────┐\n   │     Loki       │    │     Mimir      │     │     Tempo      │\n   │ (Logs Backend) │    │ (Metrics Store)│     │ (Traces Store) │\n   └──────┬─────────┘    └──────┬─────────┘     └──────┬─────────┘\n          │                     │                      │\n          ▼                     ▼                      ▼\n   ┌──────────────────────────────────────────────────────────┐\n   │                    S3 (Object Storage)                   │\n   │  - Loki chunks, index, and ruler data                    │\n   │  - Mimir blocks (TSDB data)                              │\n   │  - Tempo trace blocks (compact traces)                   │\n   └──────────────────────────────────────────────────────────┘\n          ▲                     ▲                      ▲\n          │                     │                      │\n   ┌──────┴──────────┐   ┌──────┴─────────┐     ┌──────┴──────────┐\n   │     Caches      │   │     Caches     │     │     Caches      │\n   │ (Memcached,     │   │ (Memcached,    │     │ (Memcached,     │\n   │  Redis, etc.)   │   │  Redis, etc.)  │     │  Redis, etc.)   │\n   └──────┬──────────┘   └──────┬─────────┘     └──────┬──────────┘\n          │                     │                      │\n          └──────────────┬──────┴──────────────┬───────┘\n                         ▼                     ▼\n                    ┌────────────────────────────────┐\n                    │          Grafana UI            │\n                    │────────────────────────────────│\n                    │ • Unified visualization layer  │\n                    │ • Dashboards for Logs, Metrics │\n                    │   and Traces (correlated view) │\n                    │ • Alerting and data queries    │\n                    └────────────────────────────────┘\n```\n\n\u003cbr\u003e\n\n---\n\n\u003cbr\u003e\n\n## Components Matrix\n\n|       **Component**                                |  **Version**  | **Helm Chart** |\n| -------------------------------------------------- | ------------- | -------------- |\n| [Mimir](https://github.com/grafana/mimir)          | **v3.0.4**    |   *6.0.6*      |\n| [Kube-Prometheus-Stack](https://github.com/prometheus-community/helm-charts) |  **  |  *83.7.0* |\n|    --\u003e  Prometheus Operator                        | **v0.90.1**   |   \" \" |\n|    --\u003e  Prometheus                                 | **v3.11.2**   |   \" \" |\n| [Grafana](https://github.com/grafana/grafana)      | **v12.4.3**   |   *11.6.1*    |\n|    --\u003e  PostgresDb  (*optional*)\n| [Loki](https://github.com/grafana/loki)            | **v3.7.2**    |   *15.0.1*    |\n| [Tempo](https://github.com/grafana/tempo)          | **v2.10.5**   |   *2.23.1*    |\n| [Alloy](https://github.com/grafana/alloy)          | **v1.16.3**   |   *1.9.0*     |\n\n** Note that Chart tags can be located in the source repository for most components\n   save for Grafana, Loki and Tempo having been relocated to the \n   [grafana-community](https://github.com/grafana-community/helm-charts) repository.\n\n\u003cbr\u003e\n\n## Architecture and Documentation\n\nEach component in the stack uses a distributed set of microservices running as pods \nin Kubernetes. Refer to the official Grafana documentation for each component for \ndetails of the internal architecture.\n\n- [Loki](https://grafana.com/docs/loki/v3.7.x/)\n- [Grafana](https://grafana.com/docs/grafana/v12.4/)\n- [Tempo](https://grafana.com/docs/tempo/v2.10.x/)\n- [Mimir](https://grafana.com/docs/mimir/v3.0.x/)\n- [Alloy](https://grafana.com/docs/alloy/v1.16/)\n\n\u003cbr\u003e\n\n## Requirements\n\n- [kustomize](https://github.com/kubernetes-sigs/kustomize) : v5.8.1\n- [helm](https://github.com/helm/helm) : v4.2.0\n- [yq](https://github.com/mikefarah/yq) : v4.53.3\n- [mc](https://github.com/minio/mc) : latest stable (if using MinIO)\n- [istio](https://github.com/istio/istio) : v1.30.2 (for the Ingress Controller)\n- [aws](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) \n- **httpd-tools** : system package for htpasswd (deprecated; for use with nginx only)\n\n\u003cbr\u003e\n\n---\n\n\u003cbr\u003e \n\n## Deployment Configuration\n\nThe project makes use of an Environment configuration for defining \nvarious parameters and secrets used by the various components.\n\nEach environment defines it's own configuration under a directory \nin `env`.  The project will ignore all configuration from being \nchecked in, so the overlay of those secrets should be managed \noutside of this project.\n\n- Create an Environment Configuration from the template.\n  ```sh\n  mkdir ./env/myenvname/\n  cp ./env/env.template !$/myenvname/myenvname.env\n  # set configuration and secrets in myenvname.env\n  ```\n\n- Namespace\n  The default namespace for the stack is `monitoring`. If a different\n  namespace is desired, update the *base/kustomization.yaml* files\n  or create overlays as needed.\n\n- Helm Charts\n  The project uses *kustomize* to wrap the management and use of Helm charts.\n  As a result *kustomize* requires the `--enable-helm` command switch when \n  running *build*. This can become tedious, so a *bash* function and alias \n  are provided to automatically detect the use of the helm wrapper and will \n  automatically add the switch when running kustomize.\n  ```sh\n  source env/kustom.sh\n  ```\n\n\n### Node labels\nThe *Prometheus Community Chart* includes *kube-state-metrics* and other \nk8s ecosystem components, and some *DaemonSets* or others are configured \nto run on worker nodes and contain a `node-selector` stanza.  Ensure the \nworker nodes are labeled accordingly.\n```sh\n$ k get nodes\nNAME                   STATUS   ROLES           AGE    VERSION\ndev-control-plane      Ready    control-plane   3d4h   v1.32.5\ndev-control-plane2     Ready    control-plane   3d4h   v1.32.5\ndev-control-plane3     Ready    control-plane   3d4h   v1.32.5\ndev-worker             Ready    worker          3d4h   v1.32.5\ndev-worker2            Ready    worker          3d4h   v1.32.5\ndev-worker3            Ready    worker          3d4h   v1.32.5\ndev-worker4            Ready    worker          3d4h   v1.32.5\ndev-worker5            Ready    worker          3d4h   v1.32.5\ndev-worker6            Ready    worker          3d4h   v1.32.5\n```\n\nLabel cluster 'worker' nodes:\n```sh\nnodes=$(kubectl get nodes --no-headers | \\\n        awk '{ if($2 == \"Ready\" \u0026\u0026 $3 !~ /control/) { print $1 } }')\nfor n in $nodes; do\n    kubectl label node $n node-role.kubernetes.io/worker=;\ndone\n```\n\n\u003cbr\u003e\n\n## S3 Buckets\n\nThe necessary buckets are scraped from the generated helm *values* files and\ncreated via `mc mb` or alternatively `aws s3`. If neither tool is available,\nthe buckets needed are displayed and must be manually created prior to applying\nmanifests.\n```sh\n./bin/grafana-stack-setup.sh dev\n-\u003e Found Minio Client first, using 'mc mb dev/'...\n -\u003e Ingress controller type set to 'istio'\n -\u003e Creating Loki values from template\n   -\u003e Using Loki distributed chart\n   -\u003e Copying Loki ingress certificates\n -\u003e Creating s3 secrets.env for Mimir\n -\u003e Creating Prom/Grafana values from templates\n   -\u003e Copying Grafana ingress certs\n   -\u003e Copying Prometheus ingress certs\n -\u003e Creating Tempo values from template\n   -\u003e Copying Tempo ingress certs\n -\u003e Alloy config from template\n -\u003e Needed S3 Buckets:\nmimir-dev-alertmanager\nmimir-dev-blocks\nmimir-dev-ruler\nloki-dev-chunk\nloki-dev-ruler\nloki-dev-admin\ntempo-dev-traces\n```\n\nWhen testing or re-running the setup script frequently, subsequent runs\nof the create s3 buckets can be skipped by setting the var *S3_SKIP_CREATE*.\n```bash\nexport S3_SKIP_CREATE=true\n```\n\n\u003cbr\u003e\n\n---\n\n# Mimir\n\nMimir acts as distributed metric storage and is core to the stack, so is the \nfirst component installed from the LGTM stack.\n\nFirst fetch the chart prior to the install.\n```sh\nkustomize build --enable-helm mimir/ | less\n```\n\nThis is similar to running the *helm* template command. Note that the\n*charts* path is seeded after running `kustomize build` against `./mimir/`.\n\nThe equivalent helm command would be:\n```sh\nhelm template mimir ./base/charts/mimir-distributed-5.8.0/mimir-distributed \\\n  -f base/mimir-values.yaml \\\n  -f mimir-structuredConfig.yaml \\\n  -n monitoring\n```\n\nInstall by shipping the output to *kubectl*\n```sh\nkustomize build --enable-helm mimir/ | kubectl apply -f -\n```\n\nThe `mimir-gateway` is used to expose Mimir and provide an authentication \nlayer for external clients. Note that Mimir can be used directly as a \nreplacement of *Prometheus* using the endpoint `/api/v1/push` which \nimplements the same *remote_write* api as the Prometheus endpoint: \n`api/v1/write`.\n\nPrometheus is near complete deprecation given that the *exporter* pattern \nused previously with Prometheus has be incorporated into *Grafana Alloy* \nand can ship metrics to either Prometheus or Mimir.\n\n\n## Mimir Ingress\n\nIngress manifests are provided in `mimir/ingress`\n```sh\nkustomize build mimir/ingress/istio | k apply -f -\n```\n\n\u003cbr\u003e\n\n---\n    \n# Prometheus Operator \n\n    (*Optional*)\n\nThe *Prometheus Operator* is installed via the community \nhelm [chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)\n\nNote that the kustomize manifests make use of a *node-selector* for\ntargeting *worker* nodes. Typically, *control-plane* nodes are already\nprovisioned with the role *node-role.kubernetes.io/control-plane*, but\nworker nodes often start with no role labels.\n\nOne can set the worker role on targeted nodes or all workers.\n```bash\nnodes=$(kubectl get nodes --no-headers | \\\n    awk '{ if($2 == \"Ready\" \u0026\u0026 $3 !~ /master/) { print $1 } }')\nfor n in $nodes; do\n    kubectl label node $n node-role.kubernetes.io/worker=;\ndone\n```\n\nEnsure the charts are pulled to the local *charts* cache by running\n*kustomize build* first.\n```sh\nkustomize build --enable-helm prometheus/\n```\n\nPrometheus *CRDs* are rather large (for helm), which has forced\nmoving them to a separate chart. Kubectl (and Kustomize) do not\ndirectly account for the large size on the client-side of *kubectl*\nand will throw an error when trying to directly apply the chart.\nInstead, the *CRDs* are installed independently first.\n```sh\ncd prometheus/base/charts/kube-prometheus-stack-${prometheus_version}\nkubectl apply -f kube-prometheus-stack/charts/crds/crds/ \\\n  --force-conflicts=true \\\n  --server-side=true\n```\n\nWith CRDs applied, now install prometheus\n```sh\nkustomize build --enable-helm prometheus/ | kubectl apply -f -\n```\n \n## Prometheus Ingress\n\nIngress resources are provided for *Istio* or *Nginx* (deprecated) and \nare configured when the environment configuration includes settings for \n*GRAFANA_DOMAINNAME*, *PROMETHEUS_DOMAINNAME* and *INGRESS_NAMESPACE*. \nThis will also look for certificates in the `env/$envname/certs/` path \nand copy them to *ingress/$ingress/base* path where `$ingress` will be \nset to *nginx* or *istio* depending on the \nvalue of *INGRESS_NAMESPACE*. \nNote the setup scripts specifically look for filenames of *prometheus.crt*\nand *prometheus.key*.\n```sh\ningress=\"istio\" # or nginx\nkustomize build prometheus/ingress/$ingress/ | kubectl apply -f -\n```\n\n\u003cbr\u003e\n\n---\n\n# Loki\n\nLoki supports a few different deployment modes, *Simple-Scalable*\nand *Distributed*.  The *distributed* chart deploys all services\nas pods whereas *simple-scalable* focuses on scaling the main\ncomponents. This is controlled by setting the LOKI_DISTRIBUTED\nvariable.\n\nNote that *distributed* is the recommneded path by Grafana. \n\nRecent versions of the chart configure a *podAntiAffinity* to \nforce that components run on a separate host. In some smaller \nor virtualized clusters, this can cause an issue thus this is \ndisabled by default in our values by setting `affinity: null`\nfor the components that define it. This can be removed or \ncommented for larger production deployments where the requiremnts \ncan be met.\n\nFetch the chart first for validation.\n```sh\nkustomize build --enable-helm loki | less\n```\n\nInstall the chart via *kustomize*\n```sh\nkustomize build --enable-helm loki/ | kubectl apply -f -\n```\n\n## Loki Ingress\n\nTo ship logs to Loki from outside the K8s cluster we must expose the \nLoki Distributors via the *Loki Gateway* which acts as a load balancer\nand endpoint for clients.\n\nIngress manifests for *Nginx* or *Istio* are provided as `loki/ingress/nginx`\nand `loki/ingress/istio` respectively. The setup script looks for the \nconfiguration variable *LOKI_DOMAINNAME* and configures the correct ingress \nbased on *INGRESS_NAMESPACE* as well as copying certificates from the envdir or \n`env/$envname/certs/loki.*`.  TLS Certificates should be in PEM format \nand named as `loki.crt` and `loki.key`.\n```sh\nkustomize build loki/ingress/$ingress/ | kubectl apply -f -\n```\n\n\u003cbr\u003e\n\n---\n\n# Tempo\n\n**Note that recent Tempo releases require Kubernetes 1.29+**\n\nThe current *Tempo* chart does not take S3 credentials from a secret\nlike Mimir, so a *values.template* is used to generate the input\nfor the Tempo chart. The setup script creates the *values* from the \nenvironment configuration.\n\nFetch the chart first for validation.\n```sh\nkustomize build --enable-helm tempo | less\n```\n\nInstall the chart via *kustomize*\n```sh\nkustomize build --enable-helm tempo/ | kubectl apply -f -\n```\n\n## Tempo Ingress\n\nTempo primarily needs two ports ingressed, both http/2 based, though\nboth are intended to have TLS, first for standard *https* and the \nother port, 4317, for *grpc-otlp*. The ingress controller can forward \nthese either directly to the *distributor* service, or use the \n*tempo-gateway*. The project defaults to the *tempo-gateway* for its \nability to use authentication, where the agent credentials are used.\n\n\u003cbr\u003e\n\n---\n\n# Grafana \n\nGrafana follows our typical pattern for installation using *kustomize*.\nRunning `kustomize build grafana/` provides the standard, non-enterprise \ninstall.  If the license file is located in `env/$envname/files` then it \nis assumed that an *overlay* is created for applying the Grafana Enterprise\nversions. If the overlay path does not yet exist, the setup script will \nseed an overlay using the envname and copy the example overlay as \n`grafana/overlays/$envname/kustomization.yaml` which sets the license \nsecret and used the *Enterprise* image for the container.\n```sh\nkustomize build --enable-helm grafana/overlays/$envname/ | kubectl apply -f -\n```\n\n## High-Availability\n\nGrafana supports HA mode but requires an alternate database configuration \nto the default built-in *sqllite*. This deployment uses HA by default, though \nthat can be easily reverted by changing the replica count and commenting the \n`[database]` configuration of the *grafana.ini*.\n\n### PostgresDb\n\nPostgres is the default db type supported here and should be installed first.\nThe setup script uses the *Grafana* Admin principal credentials for the pg db.\nThis deployment utilizes a customized Postgres Container Image to provision \nthe database for access by the admin account. See this [Readme](grafana/postgresdb/resources/README-postgres.md)\nfor building the custom image. Note the example postgres overlay for using\nthis image.\n\nAlso note that a *StorageClass* should be set using `Retain` for the database \nvia the *grafana/postgresdb/base/pgsql-pvc.yaml* manifest.\n\nInstall Postgres.\n```sh\nkustomize build grafana/postgresdb/overlays/mydeployment | k apply -f -\n```\n\n### Validate Grafana \n\nValidate the Grafana Values file and ensure chart is seeded by running\nkustomize without applying.\n```sh\nkustomize build --enable-helm grafana/  | less # or grafana/overlays/myenv \n```\n\nRepeat to install\n```sh\nkustomize build --enable-helm grafana/  | kubectl apply -f -\n```\n\n## Grafana Ingress\n\nAs for other components, there are ingress manifests for *Nginx* (deprecated) \nand *Istio* which will be configured by the setup script when *GRAFANA_DOMAINNAME* \nis defined along with *INGRESS_NAMESPACE*.\n```sh\nkustomize build grafana/ingress/$ingress/ | kubectl apply -f -\n```\n\n\u003cbr\u003e\n\n---\n\n# Alloy\n\nThe *Alloy* binary can be installed via RHEL or Debian package repositories or as a \nstandalone binary.\n\nHosts that have *Alloy* provisioned locally will need the `gnupg` package.\n\n*Grafana* has an [Ansible Collection](https://grafana.com/docs/alloy/latest/set-up/install/ansible/) \nthat can be used to manage Alloy deployments, however it deploys the binary\nas the `root` user. \n\nThe playbook provided as `alloy/ansible` installs the *Alloy* binary \nas a service account user and group instead. Refer to the Alloy Ansible [Readme](alloy/ansible/README-ansible.md)\n\n**Note** that the configured endpoints for Alloy (mostly all use a protocol designation \n(eg. https://) except for *Tempo*, whose endpoints are only \u003cSERVICE:PORT\u003e.\n\nAs stated in the *Mimir* section, when collecting metrics, they can be routed to \neither *Prometheus* (legacy) or directly to *Mimir* (preferred). Currently, the \nproject has configured both Mimir and Prometheus* to be exposed external to the cluster \nwith authentication for agents using the `prometheus.remote_write` endpoint. Internal \nto the cluster we do not enforce auth and we can route direct the the distributor, but \nnote that the Prometheus and Mimir *remote_write* endpoints have a different API path\nrespectively.\n\n- Prometheus  :  http://prometheus:9090/api/v1/write\n- Mimir       :  http://mimir-distributor:8080/api/v1/push\n\n\n## Ansible Deployment\n\nThe *Ansible Playbook* provided performs the following steps:\n\n- A service account or a local *alloy* user is created to allow the \n  system service to run as non-root. The deployed user should be added \n  to any groups necessary to gather metrics and read logs.\n  ```sh\n  sudo useradd --no-create-home --groups \"adm,systemd-journal\" --shell /bin/false alloy\n  ```\n\n- A default environment file is created to define the *Alloy* command \n  options as `/etc/default/alloy`.\n\n- A configuration file is provided to capture logs and record system metrics.\n  The default install dir is `/var/lib/alloy`.\n\n- A *service* file is then added to `/etc/systemd/system` to allow \n  *systemd* to manage  the *Alloy* service.\n\n- Alloy has a configuration reference [here](https://grafana.com/docs/alloy/latest/reference/)\n\n\u003cbr\u003e\n\n---\n\n# Additional Document References\n\n|                            |                              |\n| -------------------------- | ---------------------------- |\n| Loki API Reference         | https://grafana.com/docs/loki/latest/reference/ |\n| Installing with Istio      | https://grafana.com/docs/loki/latest/setup/install/istio/ |\n| Log Retention              | https://grafana.com/docs/loki/latest/operations/storage/retention/ |\n| Enterprise Logs enablement | https://grafana.com/docs/enterprise-logs/latest/setup/helm/#configure-your-gel-license |\n| Alloy Config Scenarios     | https://github.com/grafana/alloy-scenarios |\n| Prometheus Feature Flags   | https://prometheus.io/docs/prometheus/latest/feature_flags/ |\n| Tempo CLI                  | https://grafana.com/docs/tempo/latest/operations/tempo_cli/ |\n| Tempo Validation           | https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/test/set-up-test-app/ |\n\nNote that much of the LGTM documentation for OSS overlaps with the\nGrafana Enterprise Logs, Metrics, Traces documentation for installation, \nbut typically has its own document overlay of enterprise enablement details.\n\n\u003cbr\u003e\n\n---\n\n# Additional Notes\n\n## Grafana (and Postgres) PVCs\n\nEnsure a policy of *Retain* is set on the PV. Usually inherited from \nthe *StorageClass*. If the PVC was created with a default SC or is \nset to *Delete*, it can be changed accordingly.\n```sh\npv=$(kubectl get pv | grep grafana | awk '{ print $1 }')\nkubectl patch pv $pv -p '{\"spec\":{\"persistentVolumeReclaimPolicy\": \"Retain\"}}'\n```\n\nOnce the Grafana deployment has been dropped, remove any existing `claimRef` \nfrom the PV and ensure PV is *Released*.\n```sh\nkubectl patch pv $pv -p '{\"spec\":{\"claimRef\": null}}'\n```\n\nAlternatively, set volumeName on the *pvc* manifest before deploying. This\nwould require a patch to the manifests (or overlay):\n```yaml\nspec:\n  volumeName: grafana\n  accessModes:\n    - ReadWriteOnce\n```\n\n## Prometheus - Node Exporters\n\nIf choosing not to use Alloy...\n\nNote that job names should be unique within Prometheus.\n```yaml\n    additionalScrapeConfigs:\n      - job_name: 'node_exporter_host'\n        static_configs:\n          - targets: ['\u003cNODE_EXPORTER_IP_OR_HOSTNAME\u003e:9100']\n            labels:\n              instance: '\u003cNODE_EXPORTER_NAME\u003e'\n```\n\n## Loki Notes\n\nSome additional notes regarding [Loki](resources/loki-notes.md)\n\n\u003cbr\u003e\n\n---\n```\nCreated 2025.05.05\nby Timothy C. Arland \u003ctcarland at gmail dot com\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftcarland%2Fgrafana-kube-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftcarland%2Fgrafana-kube-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftcarland%2Fgrafana-kube-stack/lists"}