{"id":13643296,"url":"https://github.com/ccin2p3/samplerr","last_synced_at":"2025-04-22T16:43:42.664Z","repository":{"id":3856621,"uuid":"51133818","full_name":"ccin2p3/samplerr","owner":"ccin2p3","description":"Round robin timeseries middleware based on riemann and elasticsearch","archived":false,"fork":false,"pushed_at":"2023-11-08T12:59:31.000Z","size":487,"stargazers_count":15,"open_issues_count":10,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-29T16:51:10.170Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ccin2p3.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-05T08:01:07.000Z","updated_at":"2023-01-11T13:13:29.000Z","dependencies_parsed_at":"2023-11-08T14:00:13.335Z","dependency_job_id":null,"html_url":"https://github.com/ccin2p3/samplerr","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccin2p3%2Fsamplerr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccin2p3%2Fsamplerr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccin2p3%2Fsamplerr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccin2p3%2Fsamplerr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ccin2p3","download_url":"https://codeload.github.com/ccin2p3/samplerr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250279725,"owners_count":21404439,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:45.412Z","updated_at":"2025-04-22T16:43:42.592Z","avatar_url":"https://github.com/ccin2p3.png","language":"Clojure","funding_links":[],"categories":["Clojure"],"sub_categories":[],"readme":"# samplerr\n\n## Introduction\n\nThe main goal of this project is to provide a means for long term relevant storage for your metrics.\nIt borrows some of [rrdtool](http://rrdtool.org/)'s concepts and leverages the power of a modern storage backend: [elasticsearch](http://elastic.co/products/elasticsearch).\n\nThe idea is to downsample metrics to multiple sampling rates by consolidating those using meaningful aggregation functions: multiple archive stores with different resolutions.\nDifferent resolution archives are mainly useful for two reasons:\n\n1. Keep storage space in bounds\n2. Keep data amount in bounds at query time\n\nDifferent consolidation functions (*e.g.* min, max, avg, *etc.*) are mainly useful for keeping track of what matters in the metrics you keep.\n\nsamplerr keeps storage low and client queries fast by purging high-resolution data periodically and creates\n[elasticsearch aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html) to point the clients to the highest available resolution.\n\n![elasticsearch aliases](doc/samplerr.gif)\n\n## How it works\n\n![sampler diagram](doc/samplerr.png)\n\nIn this example, samplerr ingests a metric which has a 5s interval. It then downsamples it to 3 different archives with different consolidation functions.\nIt keeps different retention policies for each elasticsearch index. For instance, the highest resolution data (30s) is kept for two days, while the lowest resolution (3h) is kept for 1 year.\nThe disk footprint is the same for all three data stores.\n\n## Features\n\n* multiple resolution archives\n* consolidation functions\n* constant round robin storage footprint per metric with respect to time\n* transparent query across all archives\n\nIts architecure is modular, so you can use any of its following main functions:\n\n* *Downsample* metrics using consolidation functions\n* *Persist* metrics to the storage backend\n* *Rotate* archive references\n* *Purge* expired archives\n\n## Implementation\n\nThe current implementation:\n\n* is a [riemann](http://riemann.io/) plugin\n* writes your metrics to [elasticsearch](http://elastic.co/products/elasticsearch)\n* aggregates data using arbitrary clojure functions\n* aggregates data in realtime into different round robin time-based elasticsearch indices (archives)\n* manages your time based elasticsearch aliases to point to highest possible resolution data\n* ensures your metric stays within storage boundaries\n\n## Installation\n\nAfter cloning the repo, you can build the plugin using [leiningen](/technomancy/leiningen)\n\n```\nlein uberjar\n```\n\nThis will create a plugin jar named `samplerr-x.y.z-SNAPSHOT-standalone.jar` which you can include into your *java classpath*, *e.g.*:\n\n```\njava -cp /usr/lib/riemann/riemann.jar:/usr/lib/riemann/samplerr-0.1.1-SNAPSHOT-standalone-up.jar riemann.bin start /etc/riemann/riemann.config\n```\n\nOn debian or redhat you could also add the classpath using the `EXTRA_CLASSPATH` variable available respectively in `/etc/default/riemann` or `/etc/sysconfig/riemann`.\n\n## Synopsis\n\n```clojure\n(load-plugins)\n(require '[clj-time.core :as t])\n\n(let [elastic      (samplerr/connect {:hosts [\"http://localhost:9200\"]})\n      index-prefix \".samplerr\"\n      alias-prefix \"samplerr\"\n      cfunc        [{:func samplerr/average :name \"avg\"}\n                    {:func samplerr/minimum :name \"min\"}\n                    {:func samplerr/maximum :name \"max\"}]\n      archives     [{:tf \"YYYY.MM.dd\" :step (t/seconds 20) :ttl   (t/days 2) :cfunc cfunc}\n                    {:tf \"YYYY.MM\"    :step (t/minutes 10) :ttl (t/months 2) :cfunc cfunc}\n                    {:tf \"YYYY\"       :step    (t/hours 1) :ttl (t/years 10) :cfunc cfunc}]\n      rotate       (samplerr/periodically-rotate {:interval (t/days 1) :conn elastic :index-prefix index-prefix :alias-prefix alias-prefix :archives archives})\n      persist      (batch 1000 10 (samplerr/persist {:index-prefix index-prefix :index-type \"samplerr\" :conn elastic}))]\n\n  (streams\n    (where (tagged \"collectd\")\n      (by [:host :service]\n       (samplerr/down archives persist))))\n  rotate)\n```\n\n## Usage\n\n`samplerr` provides five high-level functions, two of which are stream functions.\n\n### Stream functions\n\n#### `(down archives \u0026 children)`\n\nThis stream function splits streams by archive and consolidation functions.\nIt conveniently passes on events to child streams, for example to send those to elasticsearch using the `persist` stream function.\n\nThe sequence `archives` should contain at least one archive. Each archive describes the aggregation that shall be performed and the target archive:\n\n```clojure\n(def archives [{:tf \"YYYY.MM.dd\" :step   20 :cfunc cfunc}\n               {:tf \"YYYY.MM\"    :step  600 :cfunc cfunc}\n               {:tf \"YYYY\"       :step 3600 :cfunc cfunc}])\n```\n\n* `:tf` time format string to be used to target the archive. This will be used by `persist` to target the corresponding elasticsearch index. This will be parsed by `clj-time.format` and must thus be valid. Example: the event `{:time 1458207113000 :metric 42}` will be indexed to elasticsearch into `.samplerr-2016.03.17`, `.samplerr-2016.03` and `.samplerr-2016` concurrently with the above config.\n* `:step` contains the consolidation time interval to be used to accumulate events to be aggregated using `cfunc`. This is the equivalent of `rrdtool`'s step, and represents the resolution of your time series.\n* `:cfunc` contains the list of consolidation functions to be used.\n\nConsolidation functions are a hash map containing two keys:\n\n```clojure\n(def cfunc [{:func samplerr/average :name avg}\n            {:func samplerr/minimum :name min}\n            {:func samplerr/maximum :name max}])\n```\n\n* The value of `:func` contains the stream function to be used for consolidation. It should accept one parameter corresponding to the `:step` interval. **the interface may change in the future**\n* The value of `:name` will be used as an attribute to the consolidated events, and subsequently be indexed using elasticsearch. Following up on the above example: the same event stream will be indexed to 9 elasticsearch documents: one per archive and per cfunc. For instance: `{\"@timestamp\": \"2016-03-17T10:31:53+01:00\", \"metric\": 42, \"cfunc\": \"avg\", \"_index\": \".samplerr-2016.03.17\"}`\n\n`samplerr` provides some commonly used cfuncs like `average`, `minimum` and `maximum` which are described in the corresponding section.\n\n#### `(persist options \u0026 children)`\n\nThis stream function sends events processed by `down` to the storage backend (elasticsearch). It is configured using the hash-map `options`:\n\n```clojure\n(def options {:index-prefix index-prefix :index-type index-type :conn es-conn-handle})\n```\n\n* `:index-prefix` points to the string to be prefixed to the elasticsearch index. The event's time formatted using the archive's `:tf` will be appended to that prefix.\n* `:index-type` elasticsearch document type\n* `:conn` connection handle to the elasticsearch REST endpoint. This can be a [`qbits.spandex/client` endpoint](https://github.com/mpenet/spandex/blob/master/src/clj/qbits/spandex.clj#L33), or our wrapped one called `connect`\n\nEvents should contain the riemann attribute `:tf` which will route them to the appropriate archive.\n\n### Other functions\n\n#### `(connect)`\n\nThis is a proxy to `qbits.spandex/client`\n\n#### `(rotate {:conn es-conn-handle :index-prefix index-prefix :alias-prefix alias-prefix :archives archives)`\n\nThis will manage elasticsearch aliases.\nAliases will be created for each `archive` by concatenating `index-prefix` with the `:tf` formatted date and will point to the first *unexpired* index (prefix `index-prefix`). Expiry is computed using the archive's `:ttl`.\nThe idea behind this is that clients will query elasticsearch using the aliases. Most high-level clients (*e.g.* [grafana], [kibana]) can only point to one time-base index pattern, *e.g.* `foo-YYYY.MM.dd`.\n\n`samplerr` will transparently position aliases pointing to the highest possible resolution archive that overlaps with it and that is not expired. The algorithm is roughly the following:\n\n* for each index matching `\u003cindex-prefix\u003e*`\n  * is the ttl expired?\n    * YES: move all its aliases to the next unexpired period\n    * NO:\n      * find archive it belongs to\n      * parse the time of the beginning of its period using `:tf`\n      * add an alias `\u003calias-prefix\u003e-\u003cparsed-time\u003e`\n\nThe usual way to use this function is either:\n\n* periodically using `periodically-rotate`\n* triggered by an event in the stream. For instance you could trigger the rotation when the day changes\n\n#### `(periodically-rotate {:interval periodicity :conn es-conn-handle :index-prefix index-prefix :alias-prefix alias-prefix :archives archives)`\n\nThis function will call `rotate` every `periodicity` time interval. The first argument should be given in terms of a `org.joda.time/PeriodType` object conventiently provided by `clj-time.core` using *e.g.* `hours`, `days`, *etc.*\n\nNote that the first rotation will not take effect immediately after riemann startup.\nAlso note that configuration reloads will work as expected.\n\n##### Example\n\nTake the example in the [synopsis](#synopsis) section. Let's say today is 2016-02-01 at 03:14 PM and\nriemann started exactly 2 days ago. `samplerr/rotate` fires up and processes the elasticsearch indices:\n\n* `.samplerr-2016.02.01` is younger than two days: create alias `samplerr-2016.02.01`\n* `.samplerr-2016.01.31` is younger than two days: create alias `samplerr-2016.01.31`\n* `.samplerr-2016.01.30` is two days old: expired! move its aliases to `.samplerr-2016.01`\n* `.sampler-2015.02` is younger than two months: create alias `sampler-2015.02`\n* `.sampler-2015.01` is younger than two months: create alias `sampler-2015.01`\n* `.sampler-2014.12` is two months old: expired! move its aliases to `.samplerr-2014`\n* …\n\n#### `(purge {:conn es-conn-handle :index-prefix index-prefix :archives archives)`\n\nThis function will **DELETE** expired indices. Use with care.\n\nThe usual way to use this function is either:\n\n* periodically using `periodically-purge`\n* triggered by an event in the stream. For instance you could trigger the purge when the disk space is full on the elasticsearch node\n\n#### `(periodically-purge {:interval periodicity :conn es-conn-handle :index-prefix index-prefix :archives archives)`\n\nThis function will call `purge` periodically.\n\n## Development\n\nAt the time of writing the contributors of this project are Fabien Wernli and some code from the elasticsearch integration was borrowed from [tnn1t1s](https://github.com/tnn1t1s/riemann-elastic) which itself borrowed from [kiries](https://github.com/threatgrid/kiries).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccin2p3%2Fsamplerr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fccin2p3%2Fsamplerr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccin2p3%2Fsamplerr/lists"}