{"id":27442404,"url":"https://github.com/drpsychick/ansible_influx_downsampling","last_synced_at":"2025-06-14T18:37:20.781Z","repository":{"id":55594398,"uuid":"146104060","full_name":"DrPsychick/ansible_influx_downsampling","owner":"DrPsychick","description":"Ansible role to setup downsampling with continuous queries on influxDB, optionally with backfilling and compaction of existing raw data.","archived":false,"fork":false,"pushed_at":"2022-08-27T11:59:39.000Z","size":1692,"stargazers_count":17,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-02T03:44:02.139Z","etag":null,"topics":["aggregation","ansible-role","backfill","continuous-queries","downsampling","influxdb"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DrPsychick.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-25T14:52:44.000Z","updated_at":"2024-04-23T15:23:15.000Z","dependencies_parsed_at":"2022-08-15T04:00:25.856Z","dependency_job_id":null,"html_url":"https://github.com/DrPsychick/ansible_influx_downsampling","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPsychick%2Fansible_influx_downsampling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPsychick%2Fansible_influx_downsampling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPsychick%2Fansible_influx_downsampling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DrPsychick%2Fansible_influx_downsampling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DrPsychick","download_url":"https://codeload.github.com/DrPsychick/ansible_influx_downsampling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248983010,"owners_count":21193500,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregation","ansible-role","backfill","continuous-queries","downsampling","influxdb"],"created_at":"2025-04-15T00:27:31.248Z","updated_at":"2025-04-15T00:27:31.863Z","avatar_url":"https://github.com/DrPsychick.png","language":"Shell","funding_links":["https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=FTXDN7LCDWUEA\u0026source=url","https://github.com/sponsors/DrPsychick"],"categories":[],"sub_categories":[],"readme":"## DrPsychick.ansible_influx_downsampling\n\n[![Build Status](https://travis-ci.com/DrPsychick/ansible_influx_downsampling.svg?branch=master)](https://travis-ci.com/DrPsychick/ansible_influx_downsampling)\n[![license](https://img.shields.io/github/license/drpsychick/ansible_influx_downsampling.svg)](https://github.com/drpsychick/ansible_influx_downsampling/blob/master/LICENSE)\n[![Paypal](https://img.shields.io/badge/donate-paypal-00457c.svg?logo=paypal)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=FTXDN7LCDWUEA\u0026source=url)\n[![GitHub Sponsor](https://img.shields.io/badge/github-sponsor-blue?logo=github)](https://github.com/sponsors/DrPsychick)\n\n\nConfigure influxDB for downsampling\n===================================\n\nMotivation:\n-----------\nInfluxDB uses a default retention policy that keeps data **forever** in 7 day shards - in RAW format (data points every 10 or 30 seconds, depending on your input configuration).\nOf course this is a good default, but once you have old data and want to introduce downsampling without loosing data, its a **lot** of manual work to setup all the queries etc.\n\nSo ... I have done this for you!\n\nTwo usage scenarios:\n* You already have an influxdb running and it's getting BIG, so you want to introduce downsampling on-the-fly to make things faster and cheaper\n* You intend to use influxdb and want to set it up with downsampling in mind (so it does not grow big over time in the first place)\n\nHonestly the two use cases are not much different. The biggest difference is the time it takes to run through the playbook when you enable backfilling. Of course, if you work on existing data, don't forget to **have a proper backup!**\n\nDemo\n----\n\nSee the result in action:\n\n![Watch the demo](examples/grafana-demo.gif \"Watch the demo\")\n\nThis requires to setup multiple datasources in grafana (one for each aggregation level) and a \"Datasource\" variable, like so:\n\n![Setup database variable](examples/grafana-database-variable.png \"Database variable\")\n\n\nPreparation\n-----------\nAs preparation you don't need much, except knowing how exactly you want to downsample your data as you need to setup your configuration first.\nOnce configured, you can run each level separately and/or repeatedly without any issue as the role only creates/adds what is missing (ok, CQs are always recreated as there is no way to alter them). This way you can iterate your way towards a final setup and add compaction as last step, once you're happy with the result.\n\n![Levels of downsampling](examples/influx-downsampling.png \"Levels of downsampling\")\n\nSetup\n-----\n\nEasiest setup is create a role in your own repository and adding this:\n* Decide on the name of the setup, let's call the role \"influx-setup\" and the setup \"frank\"\n* *hint* you can have any number of setups configured in this role. You just always have to load first **your** role (defining the setup) and then **DrPsychick.ansible_influx_downsampling** for each setup.\n\n`roles/influx-setup/tasks/main.yml`\n```\n- name: \"Include definition from influxdb_{{vars_name}}.yml\"\n  include_vars: influxdb_{{vars_name}}.yml\n    when: vars_name is defined\n\n# you can add your own tasks here that will be executed prior to `ansible_influx_downsampling` role\n```\n\n`roles/influx-setup/vars/influxdb_frank.yml`\n--\u003e take one from the examples directory as a base for your own: [examples/](examples/)\n\nNow in your playbook, include both roles:\n\n`influx-setup.yml`\n```\n- name: InfluxDB\n  hosts: localhost\n  roles:\n    - { role: influx-setup, vars_name: \"frank\" }\n    - { role: DrPsychick.ansible_influxdb_downsampling}\n```\n\nConfiguration\n-------------\n\nNow the most tricky part. You need to know what aggregation levels you want and define them. Moreover, especially if you don't use standard telegraf plugins for input, you need to **define the aggregation queries for your measurements**.\n\nInfluxDB has this awesome feature to do dynamic aggregation (`SELECT MEAN(*) FROM :MEASUREMENT GROUP BY *`), **but** it automatically prepends `mean_` to every field in the measurement and this would mean, you **cannot reuse the dashboard you use on your RAW data**.\n\nThe solution is simple, but requires work. You have to name your columns individually as the original column, idependent of the aggregation method you use.\n\nExample:\n```\nSELECT MEAN(usage_user) FROM cpu; # on RAW\nSELECT MEAN(usage_user) AS usage_user INTO telegraf_14d.rp_14d.cpu FROM cpu; # in aggregate queries\n```\n\nThe good thing about defining the aggregation method individually is that you can mix it and choose what is best for the individual column. And once it's defined, it can be reused for all aggregation levels.\nAdditionally you can choose to add more aggregations (min, mean and max) and have a dashboard which includes them.\n\n```\nSELECT LAST(string_value) AS string_value # for strings\nSELECT MAX(users) AS users # for metrics where you're interested in the MAX\n...\n```\n\nThe configuration goes into your `vars` file or you can choose to setup global vars for it.\nIf you use generic input plugins from telegraf or other typical sources, **please add them to `defaults/main.yml` and send me a pull request**, so others can profit from them too.\n\nQuery definition or override:\n```\nmy_ansible_influx_queries:\n  # override default settings, use german performance counter names\n  win_cpu: \u003e\n    SELECT mean(\"Benutzerzeit_(Percent)\") AS \"Benutzerzeit_(Percent)\"\n    , mean(\"DPC-Zeit_(Percent)\") AS \"DPC-Zeit_(Percent)\"\n    , mean(\"Interruptzeit_(Percent)\") AS \"Interruptzeit_(Percent)\"\n    , mean(\"Leerlaufzeit_(Percent)\") AS \"Leerlaufzeit_(Percent)\"\n    , mean(\"Privilegierte_Zeit_(Percent)\") AS \"Privilegierte_Zeit_(Percent)\"\n    , mean(\"Prozessorzeit_(Percent)\") AS \"Prozessorzeit_(Percent)\"\n```\n\nA complete setup can be found in [examples/full-5level-backfill-compact/](examples/full-5level-backfill-compact/)\n\nRun time options\n----------------\n* set ansible variable `recreate_cqs=yes` and all CQs will be recreated, so the role is running through all measurements and levels of compaction (not touching existing data)\n\nAttention\n=========\nIf you enable **backfill**:\n* Check the size of your data first. Depending on the amount of series in a measurement, you need to configure the time range for backfilling. A good default is \"1d\" - but you may not want to run 365 queries for a year...\n* Timeouts: Your InfluxDB as well as the calls in this playbook may time out! Or you may hit other limits in the influxdb.conf.\n* Stats: They will **likely** crash because they query big time ranges with lots of data. Only turn them on for small backfill jobs.\n\nIf you enable **backfill and compact**:\n* After a complete run with compaction you need to check and recreate all continuous queries based on the new default RP of the source.\n* Better: specify \"..\" as source to always use the default RP of the source.\n\nResults of my full setup can be found in [examples/full-5level-backfill-compact/](examples/full-5level-backfill-compact/)\n\nUse Cases\n=========\n\n* Just setup default RETENTION POLICY: [examples/basic.yml](examples/basic.yml)\n* Clone RAW data into new RETENTION POLICY and drop old data: [examples/compact.yml](examples/compact.yml)\n* Full 5 level downsampling including compaction: [examples/full-5level-backfill-compact/](examples/full-5level-backfill-compact/)\n\nHistory\n=======\n\nFuture Version:\n\n* [ ] refactor/cleanup variables\n   * [x] introduce \"mode\" = setup, migrate, compact\n   * [ ] separate task files\n* [ ] add changed_when conditions (e.g. drop+create CQ should be \"changed\")\n* [ ] add RP shard duration option\n* [ ] shift CQs by \"spread\" seconds: 60+/-5sec EVERY 5m+-1s,2s,3s,... + step in seconds\n* [ ] TEST: levels use \"default\" RP on source, then switch (compact) : CQs should continue to run fine\n\nVersion 0.3: Complete incl. automatic compaction, tests and good examples.\n\n* [x] multiple/full examples -\u003e see examples/\n* [x] more tests:\n   * [x] run backfill without CQ during operation and switch RP\n   * [x] setup with 2 levels and CQ\n   * [x] recreate CQs\n   * [x] CQs on empty (default) RP of source (sourcedb..measurement)\n   * [x] Merge recreate with migrate TEST, fix recreate CQs!\n* [x] howto switch retention policy (cleanup after all is setup)\n   * [x] Case: copy from \"autogen\", no CQ, drop source after backfill + set default RP -\u003e see test\n* [x] implement \"offset\" for CQs, it shifts time!\n* [x] backfill gap twice (to keep gap as small as possible)\n\nVersion 0.2: Fully working and tested. No deleting of data. Stats + CQ update.\n\n* [x] Update description + basic readme\n* [x] Check variables upfront (define clear dependencies) and print useful error messages before acting\n* [x] fix: continuous_query is required even if empty (bad usability)\n* [x] more tests:\n   * [x] test parallel tests\n   * [x] prepare seeding (generator or file?)\n   * [x] run downsampling + backfill on existing DB (needs seed)\n   * [x] run backfill with step X (on RP with 7d)\n* [x] set RP default yes/no\n* [x] improve/extend dict structure (BC break!)\n* [x] update continuous queries (drop+create)\n* [x] stats (total data points written per DB / average downsampling ratio)\n* [x] support selective group by in backfill and continuous query\n\nVersion 0.1:\n\n* basic functionality\n* create databases + retention policy\n* backfill measurements\n* create continuous queries\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdrpsychick%2Fansible_influx_downsampling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdrpsychick%2Fansible_influx_downsampling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdrpsychick%2Fansible_influx_downsampling/lists"}