{"id":18767766,"url":"https://github.com/lightstep/opentelemetry-prometheus-sidecar","last_synced_at":"2025-04-13T06:32:30.073Z","repository":{"id":46022408,"uuid":"300441486","full_name":"lightstep/opentelemetry-prometheus-sidecar","owner":"lightstep","description":"The OpenTelemetry Prometheus sidecar runs alongside the Prometheus Server and exports your metrics data to an OpenTelemetry Protocol endpoint.","archived":false,"fork":false,"pushed_at":"2022-07-28T15:46:00.000Z","size":1254,"stargazers_count":15,"open_issues_count":0,"forks_count":1,"subscribers_count":52,"default_branch":"main","last_synced_at":"2025-03-26T23:21:56.770Z","etag":null,"topics":["metrics","opentelemetry","opentelemetry-collector","prometheus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lightstep.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-01T22:32:07.000Z","updated_at":"2024-10-04T13:26:02.000Z","dependencies_parsed_at":"2022-09-15T19:00:45.431Z","dependency_job_id":null,"html_url":"https://github.com/lightstep/opentelemetry-prometheus-sidecar","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightstep%2Fopentelemetry-prometheus-sidecar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightstep%2Fopentelemetry-prometheus-sidecar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightstep%2Fopentelemetry-prometheus-sidecar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightstep%2Fopentelemetry-prometheus-sidecar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lightstep","download_url":"https://codeload.github.com/lightstep/opentelemetry-prometheus-sidecar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248674678,"owners_count":21143760,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["metrics","opentelemetry","opentelemetry-collector","prometheus"],"created_at":"2024-11-07T19:08:30.134Z","updated_at":"2025-04-13T06:32:26.493Z","avatar_url":"https://github.com/lightstep.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenTelemetry Prometheus sidecar\n\n\u003e ❗ **This sidecar is no longer recommend**. Please the OpenTelemetry Collector Prometheus receiver instead, [documentation on setting up and configuring the collector for Prometheus](https://docs.lightstep.com/docs/replace-prometheus-with-an-otel-collector-on-kubernetes) is available on the [Lightstep Observability Learning Portal](https://docs.lightstep.com).\n\nThis repository contains a sidecar for the\n[Prometheus](https://prometheus.io/) Server that sends metrics data to\nan [OpenTelemetry](https://opentelemetry.io) Metrics Protocol endpoint.  This\nsoftware is derived from the [Stackdriver Prometheus\nSidecar](https://github.com/Stackdriver/stackdriver-prometheus-sidecar).\n\n![OpenTelemetry Prometheus Sidecar Diagram](docs/img/opentelemetry-prometheus-sidecar.png)\n\n## Repository Status (2/21/2021)\n\nThis repository is being maintained by Lightstep and will be donated\nto OpenTelemetry.  [We are moving this\nrepository](https://github.com/open-telemetry/community/issues/575)\ninto the [OpenTelemetry](https://opentelemetry.io/)\n[organization](http://github.com/open-telemetry) and [will continue\ndevelopment on a public\nfork](https://github.com/open-telemetry/prometheus-sidecar) of the\n[upstream Stackdriver Prometheus\nsidecar](https://github.com/Stackdriver/stackdriver-prometheus-sidecar)\nrepository.\n\nThis code base is 100% OpenTelemetry and Prometheus, not a\nLightstep-specific sidecar, functioning to read data collected and\nwritten by Prometheus, convert into the OpenTelemetry data model, and\nwrite to an OpenTelemetry endpoint.\n\nThis sidecar sends [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md) [version 0.7](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v0.7.0) (or later versions) over gRPC.\n\n## Sidecar design\n\nThe sidecar includes:\n\n1. Prometheus write-ahead log (WAL) reader\n2. Metadata cache that tracks active instruments\n3. Configured settings:\n* Additional resource attributes to apply to all metric timeseries\n* Renaming and prefixing to change the name of metric timeseries\n* Filters to avoid reporting specific metric timeseries\n* Specify whether to use use int64 (optional) vs. double (default) protocol encoding\n\nSidecar operates by continually:\n1. Reading the prometheus WAL log (package retrieval and tail);\n1. Refreshing its view of the instrument metadata (package metadata);\n1. Transforming WAL samples into OpenTelemetry Protocol(OTLP) metrics (package retrieval);\n1. Sending OTLP metrics to the destination endpoint (package otlp).\n\n\nResources to understand how the WAL log works can be found [here](https://prometheus.io/docs/prometheus/latest/storage/) and [here](https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/).\n\n## Installation\n\nLightstep publishes Docker images of this binary named\n`lightstep/opentelemetry-prometheus-sidecar:${VERSION}`, with the\nlatest release always tagged `latest`.\n\nTo build from source, please clone this repository.  You will build a\nDocker image, push it to a private container registry, and then run\nthe container as described below.  To test and build a Docker image\nfor the current operating system, simply:\n\n```\nexport DOCKER_IMAGE_NAME=my.image.reposito.ry/opentelemetry/prometheus-sidecar\nexport DOCKER_IMAGE_TAG=$(cat ./VERSION)\nmake docker\ndocker push ${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}\n```\n\n## Deployment\n\nThe sidecar is deployed next to an already running Prometheus server.\n\nAn example command-line:\n\n```\nopentelemetry-prometheus-sidecar \\\n  --destination.endpoint=${DESTINATION} \\\n  --destination.header=\"lightstep-access-token=${VALUE}\" \\\n  --destination.attribute=\"service.name=${SERVICE}\" \\\n  --diagnostics.endpoint=${DIAGNOSTICS} \\\n  --diagnostics.header=\"lightstep-access-token=${VALUE}\" \\\n  --diagnostics.attribute=\"service.name=${SERVICE}\" \\\n  --prometheus.wal=${WAL} \\\n  --prometheus.endpoint=${PROMETHEUS} \\\n```\n\nwhere:\n\n* `DESTINATION`: Destination address https://host:port for sending prometheus metrics\n* `DIAGNOSTICS`: Diagnostics address https://host:port for sending sidecar telemetry\n* `VALUE`: Value for the `Custom-Header` request header\n* `SERVICE`: Value for the `service.name` resource attribute\n* `WAL`: Prometheus' WAL directory, defaults to `data/wal`\n* `PROMETHEUS`: URL of the Prometheus UI.\n\n[See the list of command-line flags below](#configuration).\n\nSettings can also be passed through a configuration file.  For example:\n\n```\ndestination:\n  endpoint: https://otlp.io:443\n  headers:\n    Custom-Header: custom-value\n  attributes:\n    service.name: my-service-name\ndiagnostics:\n  endpoint: https://otlp.io:443\n  headers:\n    Custom-Header: custom-value\n  attributes:\n    service.name: my-service-name\nprometheus:\n  wal: /prometheus/wal\n  endpoint: http://192.168.10.10:9191\n```\n\n[See an example configuration yaml file here](./config/sidecar.example.yaml)\n\nThe sidecar requires write access to the directory to store its progress between restarts.\n\n### Kubernetes and Helm setup\n\nTo configure the sidecar for a Prometheus server installed using the\n[Prometheus Community Helm Charts](https://github.com/prometheus-community/helm-charts).\n\n#### Sidecar with Prometheus Helm chart\nTo configure the core components of the Prometheus sidecar, add the following definition to your custom `values.yaml`:\n\n```\nserver:\n  sidecarContainers:\n  - name: otel-sidecar\n    image: lightstep/opentelemetry-prometheus-sidecar\n    imagePullPolicy: Always\n    args:\n    - --prometheus.wal=/data/wal\n    - --destination.endpoint=$(DESTINATION)\n    - --destination.header=lightstep-access-token=AAAAAAAAAAAAAAAA\n    - --diagnostics.endpoint=$(DIAGNOSTIC)\n    - --diagnostics.header=lightstep-access-token=AAAAAAAAAAAAAAAA\n    volumeMounts:\n    - name: storage-volume\n      mountPath: /data\n    ports:\n    - name: admin-port\n      containerPort: 9091\n    livenessProbe:\n      httpGet:\n        path: /-/health\n        port: admin-port\n      periodSeconds: 30\n      failureThreshold: 2\n```\n#### Sidecar with kube-prometheus-stack Helm chart\nTo configure the sidecar using the Prometheus Operator via the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) helm chart:\n\n*NOTE: the volume configured in the sidecar must be the same volume as prometheus uses*\n\n```\nprometheus:\n  prometheusSpec:\n    containers:\n      - name: otel-sidecar\n        image: lightstep/opentelemetry-prometheus-sidecar:latest\n        imagePullPolicy: Always\n\n        args:\n        - --prometheus.wal=/prometheus/prometheus-db/wal\n        - --destination.endpoint=$(DESTINATION)\n        - --destination.header=lightstep-access-token=AAAAAAAAAAAAAAAA\n        - --diagnostics.endpoint=$(DIAGNOSTIC)\n        - --diagnostics.header=lightstep-access-token=AAAAAAAAAAAAAAAA\n\n        #####\n        ports:\n        - name: admin-port\n          containerPort: 9091\n\n        #####\n        livenessProbe:\n          httpGet:\n            path: /-/health\n            port: admin-port\n          periodSeconds: 30\n          failureThreshold: 2\n\n        #####\n        resources:\n          requests:\n            ephemeral-storage: \"50M\"\n\n        volumeMounts:\n        - name: prometheus-db\n          mountPath: /prometheus\n\n```\n\nThe [upstream Stackdriver Prometheus sidecar Kubernetes\nREADME](https://github.com/Stackdriver/stackdriver-prometheus-sidecar/blob/master/kube/README.md)\ncontains more examples of how to patch an existing Prometheus\ndeployment or deploy the sidecar without using Helm.\n\n### Configuration\n\nMost sidecar configuration settings can be set through command-line\nflags, while a few more rarely-used options are only settable through\na yaml configuration file.  To see all available command-line flags,\nrun `opentelemetry-prometheus-sidecar --help`.  The printed usage is\nshown below:\n\n```\nusage: opentelemetry-prometheus-sidecar [\u003cflags\u003e]\n\nThe OpenTelemetry Prometheus sidecar runs alongside the Prometheus\n(https://prometheus.io/) Server and sends metrics data to an OpenTelemetry\n(https://opentelemetry.io) Protocol endpoint.\n\nFlags:\n  -h, --help                     Show context-sensitive help (also try\n                                 --help-long and --help-man).\n      --version                  Show application version.\n      --config-file=CONFIG-FILE  A configuration file.\n      --destination.endpoint=DESTINATION.ENDPOINT\n                                 Destination address of a OpenTelemetry Metrics\n                                 protocol gRPC endpoint (e.g.,\n                                 https://host:port). Use \"http\" (not \"https\")\n                                 for an insecure connection.\n      --destination.attribute=DESTINATION.ATTRIBUTE ...\n                                 Destination resource attributes attached to\n                                 OTLP data (e.g., MyResource=Value1). May be\n                                 repeated.\n      --destination.header=DESTINATION.HEADER ...\n                                 Destination headers used for OTLP requests\n                                 (e.g., MyHeader=Value1). May be repeated.\n      --destination.timeout=DESTINATION.TIMEOUT\n                                 Destination timeout used for OTLP Export()\n                                 requests\n      --destination.compression=DESTINATION.COMPRESSION\n                                 Destination compression used for OTLP requests\n                                 (e.g., snappy, gzip, none).\n      --diagnostics.endpoint=DIAGNOSTICS.ENDPOINT\n                                 Diagnostics address of a OpenTelemetry Metrics\n                                 protocol gRPC endpoint (e.g.,\n                                 https://host:port). Use \"http\" (not \"https\")\n                                 for an insecure connection.\n      --diagnostics.attribute=DIAGNOSTICS.ATTRIBUTE ...\n                                 Diagnostics resource attributes attached to\n                                 OTLP data (e.g., MyResource=Value1). May be\n                                 repeated.\n      --diagnostics.header=DIAGNOSTICS.HEADER ...\n                                 Diagnostics headers used for OTLP requests\n                                 (e.g., MyHeader=Value1). May be repeated.\n      --diagnostics.timeout=DIAGNOSTICS.TIMEOUT\n                                 Diagnostics timeout used for OTLP Export()\n                                 requests\n      --diagnostics.compression=DIAGNOSTICS.COMPRESSION\n                                 Diagnostics compression used for OTLP requests\n                                 (e.g., snappy, gzip, none).\n      --prometheus.wal=PROMETHEUS.WAL\n                                 Directory from where to read the Prometheus\n                                 TSDB WAL. Default: data/wal\n      --prometheus.endpoint=PROMETHEUS.ENDPOINT\n                                 Endpoint where Prometheus hosts its UI, API,\n                                 and serves its own metrics. Default:\n                                 http://127.0.0.1:9090/\n      --prometheus.max-point-age=PROMETHEUS.MAX-POINT-AGE\n                                 Skip points older than this, to assist\n                                 recovery. Default: 25h0m0s\n      --prometheus.health-check-request-timeout=PROMETHEUS.HEALTH-CHECK-REQUEST-TIMEOUT  \n                                 Timeout used for health-check requests to the prometheus endpoint. Default: 5s\n      --prometheus.scrape-interval=PROMETHEUS.SCRAPE-INTERVAL ...\n                                 Ignored. This is inferred from the Prometheus\n                                 via api/v1/status/config\n      --admin.port=ADMIN.PORT    Administrative port this process listens on.\n                                 Default: 9091\n      --admin.listen-ip=ADMIN.LISTEN-IP\n                                 Administrative IP address this process listens\n                                 on. Default: 0.0.0.0\n      --security.root-certificate=SECURITY.ROOT-CERTIFICATE ...\n                                 Root CA certificate to use for TLS connections,\n                                 in PEM format (e.g., root.crt). May be\n                                 repeated.\n      --opentelemetry.max-bytes-per-request=OPENTELEMETRY.MAX-BYTES-PER-REQUEST\n                                 Send at most this many bytes per request.\n                                 Default: 65536\n      --opentelemetry.min-shards=OPENTELEMETRY.MIN-SHARDS\n                                 Min number of shards, i.e. amount of\n                                 concurrency. Default: 1\n      --opentelemetry.max-shards=OPENTELEMETRY.MAX-SHARDS\n                                 Max number of shards, i.e. amount of\n                                 concurrency. Default: 200\n      --opentelemetry.metrics-prefix=OPENTELEMETRY.METRICS-PREFIX\n                                 Customized prefix for exporter metrics. If not\n                                 set, none will be used\n      --filter=FILTER ...        PromQL metric and attribute matcher which must\n                                 pass for a series to be forwarded to\n                                 OpenTelemetry. If repeated, the series must\n                                 pass any of the filter sets to be forwarded.\n      --startup.timeout=STARTUP.TIMEOUT\n                                 Timeout at startup to allow the endpoint to\n                                 become available. Default: 10m0s\n      --healthcheck.period=HEALTHCHECK.PERIOD\n                                 Period for internal health checking; set at a\n                                 minimum to the shortest Promethues scrape\n                                 period\n      --healthcheck.threshold-ratio=HEALTHCHECK.THRESHOLD-RATIO\n                                 Threshold ratio for internal health checking.\n                                 Default: 0.5\n      --log.level=LOG.LEVEL      Only log messages with the given severity or\n                                 above. One of: [debug, info, warn, error]\n      --log.format=LOG.FORMAT    Output format of log messages. One of: [logfmt,\n                                 json]\n      --log.verbose=LOG.VERBOSE  Verbose logging level: 0 = off, 1 = some, 2 =\n                                 more; 1 is automatically added when log.level\n                                 is 'debug'; impacts logging from the gRPC\n                                 library in particular\n      --leader-election.enabled  Enable leader election to choose a single writer.\n      --leader-election.k8s-namespace=LEADER-ELECTION.K8S-NAMESPACE  \n                                 Namespace used for the leadership election lease.\n      --disable-supervisor       Disable the supervisor.\n      --disable-diagnostics      Disable diagnostics by default; if unset,\n                                 diagnostics will be auto-configured to the\n                                 primary destination\n\n```\n\nTwo kinds of sidecar customization are available only through the\nconfiguration file.  An [example sidecar yaml configuration documents\nthe available options](./config/sidecar.example.yaml).\n\nCommand-line and configuration files can be used at the same time,\nwhere command-line parameter values override configuration-file\nparameter values, with one exception.  Configurations that support\na map from string to string, including both request headers and\nresource attributes, are combined from both sources.\n\n#### Startup safety\n\nThe sidecar waits until Prometheus finishes its first scrape(s) to\nbegin processing the WAL, to ensure that target information is\navailable before the sidecar tries loading its metadata cache.\n\nThis information is obtained through the Prometheus\n`api/v1/status/config` endpoint.  The sidecar will log which intervals\nit is waiting for during startup.  When using very long scrape\nintervals, raise the `--startup.timeout` setting so the sidecar will\nwait long enough to begin running.\n\n#### Throughput and memory configuration\n\nThe following settings give the user control over the amount of memory\nused by the sidecar for concurrent RPCs.\n\nThe sidecar uses a queue to manage distributing data points from the\nPrometheus WAL to a variable number of workers, referred to as\n\"shards\".  Each shard assembles a limited size request, determined by\n`--opentelemetry.max-bytes-per-request` (default: 64kB).\n\nThe number of shards varies in response to load, based on the observed\nlatency.  Upper and lower bounds on the number of shards are\nconfigured by `--opentelemetry.min-shards` and\n`--opentelemetry.max-shards`.\n\nCPU usage is determined by the number of shards and the workload.  Use\n`--opentelemetry.max-shards` to limit the maximum CPU used by the\nsidecar.\n\n#### Validation errors\n\nThe sidecar reports validation errors using conventions established by\nLightstep for conveying information about _partial success_ when\nwriting to the OTLP destination.  These errors are returned using gRPC\n\"trailers\" (a.k.a. http2 response headers) and are output as metrics\nand logs.  See the `sidecar.metrics.failing` metric to diagnose\nvalidation errors.\n\n#### Metadata errors\n\nThe sidecar may encounter errors between itself and Prometheus,\nincluding failures to locate metadata about a targets that Prometheus\nno longer knows about.  Missing metadata, Prometheus API errors, and\nother forms of inconsistency are reported using\n`sidecar.metrics.failing` with `key_reason` and `metric_name` attributes.\n\n#### OpenTelemetry Resource Attributes\n\nUse the `--destination.attribute=KEY=VALUE` flag to add additional OpenTelemetry resource attributes to all exported timeseries.\n\n#### Prometheus External Labels\n\nPrometheus external labels are used for diagnostic purposes but are not attached to exported timeseries.\n\n#### Filters\n\nThe `--filter` flag allows to provide filters which all series have to pass before being sent to the destination. Filters use the same syntax as [Prometheus instant vector selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#instant-vector-selectors), e.g.:\n\n```\nopentelemetry-prometheus-sidecar --filter='{__name__!~\"cadvisor_.+\",job=\"k8s\"}' ...\n```\n\nThis drops all series which do not have a `job` label `k8s` and all metrics that have a name starting with `cadvisor_`.\n\nFor equality filter on metric name you can use the simpler notation, e.g. `--filter='metric_name{label=\"foo\"}'`.\n\nThe flag may be repeated to provide several sets of filters, in which case the metric will be forwarded if it matches at least one of them.\n\n#### Metric renames\n\nTo change the name of a metric as it is exported, use the\n`metric_renames` section in the configuration file:\n\n```yaml\nmetric_renames:\n  - from: original_metric_name\n    to: new_metric_name\n# - ...\n```\n\n#### Static metadata\n\nTo change the output type, value type, or description of a metric\ninstrument as it is exported, use the `static_metadata` section in the\nconfiguration file:\n\n```yaml\nstatic_metadata:\n  - metric: some_metric_name\n    type: counter # or gauge, or histogram\n    value_type: double # or int64\n    help: an arbitrary help string\n# - ...\n```\n\nNote:\n\n* All `static_metadata` entries must have `type` specified.\n* If `value_type` is specified, it will override the default value type for counters and gauges. All Prometheus metrics have a default type of double.\n\n#### High Availability Prometheus Setup\n\nIn a HA prometheus setup, a prometheus sidecar can be attached to each replica. All sidecars will write all metric points, it is responsibility of the backend of choice to deduplicate these points.\n\nThe leader election can be enabled in order to restrict which replica will send metrics to the Collector, reducing the amount of metrics transferred on the wire.\n\nOne of the sidecars will be elected as the Leader, this leader sidecar will tail the Prometheus WAL log, transform and send OTLP metrics to the Collector. All other non-leader sidecars will be in a stand-by mode, it will tail the Prometheus WAL log, but will not send any data to the Collector.\n\nIf the leader sidecar fails, a new Leader will be elected and will resume sending data to the collector.\n\nThe leader election uses the kubernetes coordination API to elect a leader. Ensure that there is a service account for Prometheus in Kubernetes and then bind it to the role with the following permissions:\n```yaml\nrules:\n  - apiGroups:\n      - coordination.k8s.io\n    resources:\n      - leases\n    verbs:\n      - '*'\n```\n\nAfter to the service account permissions are set up, set the argument flag  `--leader-election.enabled` on the prometheus sidecar.\n\nTo change the namespace used for the leadership election lease, set `--leader-election.k8s-namespace=LEADER-ELECTION.K8S-NAMESPACE`.\n\n## Monitoring\n\nWhen run in the default configuration, the sidecar will self-monitor for liveness and kill the process when it becomes unhealthy.  Sidecar liveness can be monitored directly using the `/-/health` endpoint.  We recommend a period of 30 seconds and `failureThreshold: 2`, for example in your Kubernetes deployment:\n\n```\n    ports:\n    - name: admin-port\n      containerPort: 9091\n    livenessProbe:\n      httpGet:\n        path: /-/health\n        port: admin-port\n      periodSeconds: 30\n      failureThreshold: 2\n```\n\n## Diagnostics\n\nThe sidecar is instrumented with the OpenTelemetry-Go SDK and runs with standard instrumentation packages, including runtime and host metrics.\n\nBy default, diagnostics are autoconfigured to the primary destination.  Configuration options are available to disable or configure alternative destinations and options.\n\n### Configuration Options\n\nSeparate diagnostics settings can be configure to output OTLP similar to configuring the primary destination, for example:\n\n```\ndiagnostics:\n  endpoint: https://otel-collector:443\n  headers:\n    Custom-Header: custom-value\n  timeout: timeout-value\n  attributes:\n    extra.resource: extra-value\n```\n\nLikewise, these fields can be accessed using `--diagnostics.endpoint`,\n`--diagnostics.header`, `--diagnostics.timeout`, and `--diagnostics.attribute`.\n\n#### Log levels\n\nThe Prometheus sidecar provides options for logging in the case of diagnosing an issue.\n* We recommend starting with setting the `--log.level` to be `debug`, `info`, `warn`, `error`.\n* Additional options are available to set the output format of the logs (`--log.format` to be `logfmt` or `json`), and the number of logs to recorded (`--log.verbose` to be `0` for off, `1` for some, `2` for more)\n\n#### Disabling Diagnostics\n\nTo disable diagnostics, set the argument flag `--disable-diagnostics`.\n\n### Diagnostic Outputs\n\nThe Prometheus Sidecar has two main processes, the supervisor and a subordinate process.  Diagnostic traces are sent from the supervisor process to monitor lifecycle events and diagnostic metrics are sent from the subordinate process.\n\n#### Sidecar Supervisor Tracing\n\nTraces from the supervisor process are most helpful for diagnosing unknown problems.  It signals lifecycle events and captures the stderr, attaching it to the trace.\n\nThe sidecar will output spans from its supervisor process with service.name=`opentelemetry-prometheus-sidecar-supervisor`.  There are two kinds of spans: operation=`health-client` and operation=`shutdown-report`.  These spans are also tagged with the attribute `sidecar-health` with values `ok`, `not yet ready` or `first time ready`.\n\n#### Sidecar Metrics\n\nMetrics from the subordinate process can help identify issues once the first metrics are successfully written.  There are three standard host and runtime metrics to monitor:\n\n**Key Success Metrics**\n\nThese metrics are key to understanding the health of the sidecar.\nThey are periodically printed to the console log to assist in\ntroubleshooting.\n\n| Metric Name | Type | Description | Additional Attributes |\n| --- | --- | --- | ---|\n| sidecar.points.produced | counter | number of points read from the prometheus WAL | |\n| sidecar.points.dropped | counter | number of points dropped due to errors | `key_reason`: metadata, validation |\n| sidecar.points.skipped | counter | number of points skipped by filters | |\n| sidecar.queue.outcome | counter | outcome of the sample in the queue | `outcome`: success, failed, retry, aborted |\n| sidecar.series.dropped | counter | number of series or metrics dropped | `key_reason`: metadata, validation |\n| sidecar.series.current | gauge | number of series in the cache | `status`: live, filtered, invalid |\n| sidecar.metrics.failing | gauge | failing metric names and explanations | `key_reason`, `metric_name` |\n\n**Progress Metrics**\n\nTwo metrics indicate the sidecar's progress in processing the\nPrometheus write-ahead-log (WAL).  These are measured in bytes and are\nbased on the assumption that each WAL segment is a complete 128MB.\nBoth of these figures are exported as `current_segment_number *\n128MB + segment_offset`.  The difference (i.e., `sidecar.wal.size -\nsidecar.wal.offset`) indicates how far the sidecar has to catch up,\nassuming complete WAL segments.\n\n| Metric Name | Type | Description |\n| --- | --- | --- |\n| sidecar.wal.size | gauge | current writer position of the Prometheus write-ahead log |\n| sidecar.wal.offset | gauge | current reader position in the Prometheus write-ahead log |\n\n**Host and Runtime Metrics**\n\n| Metric Name | Type | Description | Additional Attributes |\n| --- | --- | --- | ---|\n| process.cpu.time | counter | cpu seconds used | `state`: user, sys |\n| system.network.io | counter | bytes sent and received | `direction`: read, write |\n| runtime.go.mem.heap_alloc | gauge | memory in use | |\n\n**Internal Metrics**\n\nThese metrics are diagnostic in nature, meant for characterizing\nperformance of the code and individual Prometheus installations.\nOperators can safely ignore these metrics except to better understand\nsidecar performance.\n\n| Metric Name | Type | Description | Additional Attributes |\n| --- | --- | --- | ---|\n| sidecar.connect.duration | histogram | how many attempts to connect (and how long) | `error`: true, false |\n| sidecar.export.duration | histogram | how many attempts to export (and how long) | `error`: true, false |\n| sidecar.monitor.duration | histogram | how many attempts to scrape Prometheus /metrics (and how long) | `error`: true, false |\n| sidecar.metadata.fetch.duration | histogram | how many attempts to fetch metadata from Prometheus (and how long) | `mode`: single, batch; `error`: true, false |\n| sidecar.queue.capacity | gauge | number of available slots for samples (i.e., points) in the queue, counts buffer size times current number of shards | |\n| sidecar.queue.running | gauge | number of running shards, those which have not exited | |\n| sidecar.queue.shards | gauge | number of current shards, as set by the queue manager | |\n| sidecar.queue.size | gauge | number of samples (i.e., points) standing in a queue waiting to export | |\n| sidecar.series.defined | counter | number of series defined in the WAL | |\n| sidecar.metadata.lookups | counter | number of calls to lookup metadata | `error`: true, false |\n| sidecar.refs.collected | counter | number of WAL series refs removed from memory by garbage collection | `error`: true, false |\n| sidecar.refs.notfound | counter | number of WAL series refs that were not found during lookup | |\n| sidecar.segment.opens | counter | number of WAL segment open() calls | |\n| sidecar.segment.reads | counter | number of WAL segment read() calls | |\n| sidecar.segment.bytes | counter | number of WAL segment bytes read | |\n| sidecar.segment.skipped | counter | number of skipped WAL segments | |\n| sidecar.leadership | gauge | indicate if this sidecar is the leader or not | |\n\n## Upstream\n\nThis repository was copied into a private reposotitory from [this upstream fork](https://github.com/Stackdriver/stackdriver-prometheus-sidecar/tree/1361301230bcfc978864a8f4c718aba98bc07a3d) of `stackdriver-prometheus-sidecar`, dated July 31, 2020.\n\n## Compatibility\n\nThe matrix below lists the versions of Prometheus Server and other dependencies that have been qualified to work with releases of `opentelemetry-prometheus-sidecar`. If the matrix does not list whether they are compatible, please assume they are not verified yet but can be compatible. Feel free to contribute to the matrix if you have run the end-to-end test between a version of `opentelemetry-prometheus-sidecar` and Prometheus server.\n\n| Sidecar Version | Compatible Prometheus Server Version(s)   | Incompatible Prometheus Server Version(s) |\n| -- | -- | -- |\n| **0.1.x** | 2.15, 2.16, 2.18, 2.19, 2.21, 2.22, 2.23, 2.24 | 2.5 |\n\n## Troubleshooting\n\nThe following describes known scenarios that can be problematic for the sidecar.\n\n### Sidecar is falling behind\n\nIt's possible for the sidecar to not have enough resources by default to keep up with the WAL in high throughput scenarios. When this case occurs, the following message will be displayed in the sidecar's logs:\n\n```\npast WAL segment not found, sidecar may have dragged behind. Consider increasing min-shards, max-shards and max-bytes-per-request value\n```\n\nThis message means that the sidecar is looking for a WAL segment file that has been removed, usually due to Prometheus triggering a checkpoint. It's possible to look at the delta between `sidecar.wal.size` (total wal entries) and `sidecar.wal.offset` (where the sidecar currently is) to determine if the sidecar has enough resources to keep up. If the offset is increasingly further behind the size, it's recommended to increase the timeseries emitted per request using the following configuration options:\n\n- `--opentelemetry.max-bytes-per-request` configures the maximum number of timeseries sent with each request from the sidecar to the OTLP backend.\n- `--opentelemetry.max-shards` configures the number of parallel go routines and grpc connections used to transmit the data.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightstep%2Fopentelemetry-prometheus-sidecar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flightstep%2Fopentelemetry-prometheus-sidecar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightstep%2Fopentelemetry-prometheus-sidecar/lists"}