https://github.com/nicolastakashi/metricsdedupprocessor

OpenTelemetry Processor for Metrics Deduplication
https://github.com/nicolastakashi/metricsdedupprocessor

Last synced: about 7 hours ago
JSON representation

OpenTelemetry Processor for Metrics Deduplication

• Host: GitHub
• URL: https://github.com/nicolastakashi/metricsdedupprocessor
• Owner: nicolastakashi
• License: apache-2.0
• Created: 2024-07-13T23:09:04.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-07-16T20:32:41.000Z (4 months ago)
• Last Synced: 2024-09-20T06:40:02.584Z (about 2 months ago)
• Language: Go
• Size: 30.3 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Metrics Dedup Processor

| Status        |           |

| ------------- |-----------|

| Stability     | [development]: metrics   |

| Distributions | [] |

| Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Aprocessor%2Fgroupby%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aprocessor%2Fgroupby) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Aprocessor%2Fgroupby%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aprocessor%2Fgroupby) |

| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@nicolastakashi](https://www.github.com/nicolastakashi) |

[development]: https://github.com/open-telemetry/opentelemetry-collector#development

## Description

This processor deduplicates metrics by ensuring that only metrics from the in-use replica are retained. It identifies the in-use replica by examining the data points attributes of incoming metrics to find the replica label. For more details, check the [Defining in use replica](#defining-in-use-replica) section.

This explanation highlights that the processor inspects the metric resource attributes to determine the replica label.

```yaml

processors:

  metricsdedup:

    replica_label: replica

    swap_timeout: 5m

```

| Name | Description | Default |

|------|-------------|---------|

| `replica_label` | The label key that identifies the replica. | `replica` |

| `swap_timeout` | The duration that defines the time window to swap the in-use replica. | `1m` |

## Defining in use replica

The logic used to determine the in-use replica is inspired by [Cortex](https://grafana.com/blog/2019/10/03/deduping-ha-prometheus-samples-in-cortex) component but adopts a simpler approach to avoid the need for distributed consensus on day zero.

Below is the sequence diagram that describes the logic used to determine the in-use replica:

```mermaid

sequenceDiagram

    participant Processor

    participant Receiver

    participant ReplicaInfo

    Receiver->>Processor: Send metric with replica

    Processor->>ReplicaInfo: Load current replica info

    alt No current replica

        Processor->>ReplicaInfo: Set new replica and update timestamp

    else Current replica exists

        alt Replica matches current replica

            Processor->>ReplicaInfo: Update timestamp

        else Replica different from current replica

            alt Timestamp older than swap timeout

                Processor->>ReplicaInfo: Set new replica and update timestamp

            else Timestamp within swap timeout

                Processor->>ReplicaInfo: Retain current replica

            end

        end

    end

    Processor->>Receiver: Return in-use replica

```

This approach ensures that only metrics from the in-use replica are processed, reducing the number of metrics processed and stored.