{"id":19525562,"url":"https://github.com/alantang888/prometheus-downsampler","last_synced_at":"2025-04-26T10:30:46.728Z","repository":{"id":191368541,"uuid":"145084460","full_name":"alantang888/prometheus-downsampler","owner":"alantang888","description":"Collect data from Prometheus and downsampling","archived":false,"fork":false,"pushed_at":"2018-08-28T07:35:53.000Z","size":48,"stargazers_count":6,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T11:24:53.458Z","etag":null,"topics":["downsampling","long-term-storage","prometheus"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alantang888.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-08-17T07:02:11.000Z","updated_at":"2024-11-19T02:56:44.000Z","dependencies_parsed_at":"2023-08-29T15:27:26.558Z","dependency_job_id":null,"html_url":"https://github.com/alantang888/prometheus-downsampler","commit_stats":null,"previous_names":["alantang888/prometheus-downsampler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alantang888%2Fprometheus-downsampler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alantang888%2Fprometheus-downsampler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alantang888%2Fprometheus-downsampler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alantang888%2Fprometheus-downsampler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alantang888","download_url":"https://codeload.github.com/alantang888/prometheus-downsampler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250972521,"owners_count":21516372,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["downsampling","long-term-storage","prometheus"],"created_at":"2024-11-11T01:05:33.638Z","updated_at":"2025-04-26T10:30:46.485Z","avatar_url":"https://github.com/alantang888.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prometheus Downsampler\nThis program use for collect Prometheus data for last n minutes (default is 5 minutes). \nThen take average on each metrics. Output to a text file.\n\n### Solution\n\u003cdel\u003eUse with another Prometheus for store the downsampled data. For our case, we set the long term Prometheus retention to 2 years (Still testing. Hope it will work as expected).\u003c/del\u003e\n\nTested for a while. Memory usage on prometheus keep growing. It may cause by metrics didn't update recently but not reach retention will keep that index in memory. Now will try [thanos].\n\nThis program only output a text file on K8S empty dir. Then use a nginx in same pod to expose the output to long-term Prometheus.\nAnd need to set `honor_labels: true` inside long term Prometheus scrape job. Otherwise some conflicted labels will be renamed.\n\n![Downsampler with 2 Prometheus](https://github.com/alantang888/prometheus-downsampler/blob/master/other_resource/Prometheus_Downsampler_Solution.png)\n\n\n### Config\nThere 4 parameters can config. You can either use `args` or `environment variable`.\n- Source Prometheus endpoint\n    - Default: http://127.0.0.1:9090\n    - Args: -s\n    - Environame variable: PDS_SOURCE\n- Output file path\n    - Default: /tmp/prometheus_downsample_output.txt\n    - Args: -o\n    - Environame variable: PDS_OUTPUT\n- Interval in minute for collect data from source Prometheus\n    - Default: 5m\n    - Args: -i\n    - Environame variable: PDS_INTERVAL\n- Max concurrent connection to source Prometheus\n    - Default: 50\n    - Args: -c\n    - Environame variable: PDS_CONCURRENT\n    \nExample: Your prometheus endpoint is `http://192.168.1.20:9090` and want to downsample data for every `10 minutes`:\n```bash\ngo run prometheus-downsampler.go -s http://192.168.1.20:9090 -i 10m\n```\nor\n```bash\n./prometheus-downsampler -s http://192.168.1.20:9090 -i 10m\n```\n\n### How it work\n1. Call [Querying label values] API to get all metric names\n1. Call [Range Queries] API to get every metrics with 1 minute step\n1. Take average on each metrics\n1. Write all metrics with [exposition format] to a temp file\n1. Rename the temp file to output file name\n\n### Issue\nThis program can handle collect a longer time range data. Then group them to every n minute and take average.\n**But due to below reasons. Now only process for single time group.**\n- [exposition format] mention `Each line must have a unique combination of a metric name and labels. Otherwise,\n the ingestion behavior is undefined.`. But didn't mention is it safe if have different timestamp\n- Also tested for a while with export 1 hour data with 12 data points. The long term Prometheus lost some of data point.\n\n### Why not use remote_write with InfluxDB\nBecause we scrape over 650K metrics every 10 second (With 2 Prometheus servers for HA). \nWe tried use `remote_write` to InfluxDB (Single server, not enterprise edition). \nBut it cause InfluxDB very high CPU usage, no response and OOM dead very soon. Also make the operation Prometheus dead.\nSo we try on different way (This project).\nAlso just need a little bit modify on Grafana dashboard. No need to re-build all dashboard (Don't know why use Prometheus `remote_read` from InfluxDB for Grafana always got proxy timeout from Grafana).\n\n[thanos]: https://github.com/improbable-eng/thanos\n[Querying label values]: https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values\n[Range Queries]: https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries\n[exposition format]: https://prometheus.io/docs/instrumenting/exposition_formats/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falantang888%2Fprometheus-downsampler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falantang888%2Fprometheus-downsampler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falantang888%2Fprometheus-downsampler/lists"}