https://github.com/cosh/prometheustoadx
Ingest data originating from Prometheus to Kusto
https://github.com/cosh/prometheustoadx
kusto prometheus
Last synced: 10 months ago
JSON representation
Ingest data originating from Prometheus to Kusto
- Host: GitHub
- URL: https://github.com/cosh/prometheustoadx
- Owner: cosh
- License: mit
- Created: 2019-07-18T18:21:46.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2024-08-27T12:42:28.000Z (almost 2 years ago)
- Last Synced: 2025-04-08T18:54:27.031Z (about 1 year ago)
- Topics: kusto, prometheus
- Language: C#
- Homepage:
- Size: 712 KB
- Stars: 19
- Watchers: 0
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Azure Data Explorer Storage adapter for Prometheus.io
This is a a set of two [functions](https://azure.microsoft.com/en-us/services/functions/) which are needed for the "remote_write" and "remote_read" configuration of [Prometheus](https://prometheus.io/). It enables the user to use [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) to read and write metrics. Some more details can be found on this [article](https://www.linkedin.com/pulse/prometheusio-azure-data-explorer-henning-rauch/).
## Architecture
## HowTo
### 1. *Deploy* the infrastructure
(take a look at the architecure and have fun in the Azure portal or contribute some proper ARM templates :D)
### 2. *Create* tables in ADX
```
.create table RawData ingestion json mapping "RawDataMapping" '[{ "column" : "d", "datatype" : "dynamic", "path" : "$"}]'
.create function Update_SearchExplosion()
{
RawData
| extend id = new_guid()
| mv-expand d.Samples
| extend value=todouble(d_Samples.Value), ts=datetime(1970-01-01) + tolong(d_Samples.Timestamp)*1ms, labels=d.Labels
| project-away d_Samples
| mv-expand labels
| extend label=tostring(labels.Name), labelValue=tostring(labels.Value)
| project-away labels
| extend p = pack(label, labelValue)
| summarize label=make_bag(p), timeseries=any(d), value=any(value) by ['id'], ts
| project-away ['id']
}
.create table SearchExplosion (ts: datetime, label: dynamic, timeseries: dynamic, value: real)
.alter table SearchExplosion policy update
@'[{"IsEnabled": true, "Source": "RawData", "Query": "Update_SearchExplosion()", "IsTransactional": false, "PropagateIngestionProperties": false}]'
```
### 3. *Configure* prometheus
Sample config extension for prometheus.yml:
```
remote_write:
- url: "https://.azurewebsites.net/api/Write"
remote_timeout: 30s
queue_config:
capacity: 100000
max_shards: 1000
max_samples_per_send: 1000
batch_send_deadline: 5s
max_retries: 10
min_backoff: 30ms
max_backoff: 100ms
remote_read:
- url: "https://.azurewebsites.net/api/Read"
read_recent: true
```