https://github.com/krzko/gcp-anomaly-detector

🔢 Anomaly detection service for Google Cloud Monitoring metrics, utilising statistical analysis to identify significant deviations in time-series data
https://github.com/krzko/gcp-anomaly-detector

anomaly anomaly-detection gcm gcp gcp-monitoring z-score

Last synced: 7 days ago
JSON representation

🔢 Anomaly detection service for Google Cloud Monitoring metrics, utilising statistical analysis to identify significant deviations in time-series data

• Host: GitHub
• URL: https://github.com/krzko/gcp-anomaly-detector
• Owner: krzko
• Created: 2023-10-16T23:27:51.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-01-16T03:05:07.000Z (10 months ago)
• Last Synced: 2024-06-21T08:26:18.818Z (5 months ago)
• Topics: anomaly, anomaly-detection, gcm, gcp, gcp-monitoring, z-score
• Language: Go
• Homepage:
• Size: 3.36 MB
• Stars: 3
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# gcp-anomaly-detector

This tool is designed to detect anomalies in Google Cloud Monitoring metrics. It fetches historical and recent metrics data, computes baseline statistics, and identifies anomalies based on Z-score analysis. The Z-score represents how many standard deviations an element is from the mean, and a high absolute Z-score indicates a potential anomaly.

## Configuration

The tool is configured using a YAML file. Here's an example configuration file with explanations for each field:

```yaml

metrics:

  - 'custom.googleapis.com/otel/foo_connection_count'

  - 'custom.googleapis.com/otel/foo_current_connections'  # List of metric types to monitor

filters:

  custom.googleapis.com/otel/foo_connection_count: 'resource.type="generic_task" AND metric.labels."environment"="dev"'

  custom.googleapis.com/otel/foo_current_connections": 'resource.type="generic_task" AND metric.labels."environment"="dev"'  # Filters to apply when fetching metrics

baseline_duration: 7  # Baseline duration in days

polling_time: 60  # Polling time in seconds

project_id: foo-bar-dev-1a2b3c  # GCP Project ID

recent_duration: 60  # Recent metrics duration in minutes

z_score_threshold: 3.00  # Z-score threshold for anomaly detection

```

## Usage

1. Create a configuration file following the example above.

2. Build the tool:

```sh

go build

```

3. Authenticate to Google Cloud

```sh

gcloud auth login --update-ad

export GOOGLE_APPLICATION_CREDENTIALS="/Users/$USER/.config/gcloud/application_default_credentials.json"

```

4. Run the tool:

```sh

./gcp-anomaly-detector

```

The tool will load the configuration, initialise a baseline using historical metrics data, and start polling recent metrics data at the specified interval. It will log the Z-scores for each metric and report any anomalies detected based on the configured Z-score threshold.

## Understanding Z-Score

The Z-score is a statistical measurement that describes a value's relationship to the mean of a group of values. It is measured in terms of standard deviations from the mean. In this tool, a high absolute Z-score (e.g., 3.0 or -3.0) indicates a potential anomaly.

* A positive Z-score indicates the data point is higher than the mean.

* A negative Z-score indicates the data point is lower than the mean.

The `z_score_threshold` in the configuration file determines the Z-score value at which a data point is considered an anomaly. For example, with a `z_score_threshold` of 3.00, any data point with a Z-score of 3.0 or -3.0 and above would be flagged as an anomaly.