{"id":18731016,"url":"https://github.com/mcorbin/riemann-discovery","last_synced_at":"2025-04-12T17:34:25.627Z","repository":{"id":62434456,"uuid":"91371934","full_name":"mcorbin/riemann-discovery","owner":"mcorbin","description":"Service discovery for Riemann","archived":false,"fork":false,"pushed_at":"2018-01-17T22:22:15.000Z","size":142,"stargazers_count":14,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-26T11:51:14.985Z","etag":null,"topics":["alerting","discovery","monitoring","riemann"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mcorbin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-15T18:38:09.000Z","updated_at":"2024-08-01T14:31:14.000Z","dependencies_parsed_at":"2022-11-01T21:00:54.339Z","dependency_job_id":null,"html_url":"https://github.com/mcorbin/riemann-discovery","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcorbin%2Friemann-discovery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcorbin%2Friemann-discovery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcorbin%2Friemann-discovery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcorbin%2Friemann-discovery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mcorbin","download_url":"https://codeload.github.com/mcorbin/riemann-discovery/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248605704,"owners_count":21132220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alerting","discovery","monitoring","riemann"],"created_at":"2024-11-07T14:52:13.664Z","updated_at":"2025-04-12T17:34:25.598Z","avatar_url":"https://github.com/mcorbin.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# riemann-discovery\n\n[![Clojars Project](https://img.shields.io/clojars/v/riemann-discovery.svg)](https://clojars.org/riemann-discovery)\n\nService discovery for Riemann ([riemann.io](http://riemann.io))\n\n## Why ?\n\nDetects a datasource that stopped sending events with Riemann is easy. You just need to index some events coming from the datasource and use a stream to detect expired events.\n\nBut what if a datasource hasn't emitted ? This plugin aims to solve this.\n\n## How ?\n\nThis plugin will periodically read a datasource (files, HTTP endpoint, maybe consul etc... in the future) which shows the current state of the system (hosts, services...).\nThen, it will continually generate events using the datasource content, and index these events. It will also remove outdated hosts and services, and index new ones (thus allowing the use of this plugin in a dynamic environment).\n\n## Howto\n\n### Loading the plugin\n\nFirst, you should download the .jar file, and tell Riemann how to find it in your system using the `/etc/default/riemann` or `/etc/sysconfig/riemann` file and the EXTRA_CLASSPATH option.\n\nThen, on your `riemann.config` file, add:\n\n```clojure\n(load-plugins)\n\n(require '[riemann-discovery.core :as discovery])\n```\n\nYou are now ready to use the plugin !\n\n### Files based discovery\n\nIn a directory (for example `/etc/riemann/discovery`), creates a new file, for example named `services.edn`. It will contains the informations about your system, for example:\n\n```clojure\n{:ttl 120\n :services [{:hosts [\"kafka1\" \"kafka2\"]\n             :name \"kafka\"\n             :tags [\"kafka\"]\n             :ttl 60}\n            {:hosts [\"api1\"]\n             :tags [\"api\"]\n             :name \"api\"}]}\n\n{:tags [\"production\"]\n :services [{:hosts [\"zookeeper1\"]\n             :tags [\"zookeeper\"]\n             :name \"zookeeper\"\n             :ttl 60}]}\n```\n\nHere, informations are stored in [edn format](https://github.com/edn-format/edn). We have 2 maps in this configuration. Let's take the first one.\n\n#### First map\n\nWe first have a `:ttl` key, with a value of 120. This means that all services defined in this map will have a default `:ttl` of 120. I will explain this later.\n\nWe also have a `:service` key. The value associated is a vector of map, each map describing a service.\n\nThe first map indicates that the service named `\"kafka\"` (the `:name` key) is running on hosts `\"kafka1\"` and `\"kafka2\"`. This service is also tagged `\"kafka\"`, and have a `:ttl` of 60.\n\nThe second map indicates that the service named `\"api\"` is running on host `\"api1\"`. This service is tagged `\"api\"`. Here, we don't have a `:ttl`, so this service will use the default `ttl` (120).\n\n#### Second map\n\nIn the second map, we don't have a default `:ttl` but we have default `:tags` (here `\"production\"`). Default tags and service specific tags are merged together by the plugin.\n\nhere, we only have one service named `\"zookeeper`\", running on host `\"zookeeper1\"`, tagged `\"zookeeper\"` and with a `:ttl` of 60.\n\n#### Plugin configuration\n\nYou should now configure the plugin in riemann.config:\n\n```clojure\n\n(discovery/discovery {:type :file\n                      :interval 120}\n                     {:path [\"/etc/riemann/discovery/\"]})\n\n(let [index (index)]\n  (streams\n   (discovery/discovery-stream index)))\n\n```\n\nThe `discovery/discovery` function takes 2 parameters:\n\n- First parameter: a map containing global discovery options (common to all discovery mechanisms).\n  - `:type`: The service discovery mechanism, here `:file`.\n  - `:interval`: Every `:interval` seconds (120 in our case, default 60), Riemann will call the service discovery mechanism, and inject into itself events representing added/removed hosts.\n  - `:tags`: Optional parameter, the value should be a list of tags. The plugin will only emit events tagged with all of these tags. It will emit all events if `:tags` is nil or empty. Example:\n\n```clojure\n:tags [\"production\"]\n```\n\n- Second parameter: a map containing the configuration of the discovery mechanism specified in the first parameter. For `:file`:\n  - `:path`: A list of path to directories.\n\nEmitted events are tagged `riemann-discovery`, and their states are either `added` or `removed`. **The \"-discovery\" string will be added to the service name** (see example below).\n\n- `(discovery/discovery-stream index)` is a stream taking an index as parameter. It will capture events emitted by the `discovery` function, and add/remove them from the index.\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"doc/img/riemann_discovery.png\" alt=\"Riemann discovery\" title=\"Riemann discovery\" /\u003e\n\u003c/p\u003e\n\nIn our example the `discovery` function will emit these events:\n\n```clojure\nINFO [2017-12-22 20:52:30,050] riemann task 1 - riemann.config - {:host zookeeper1, :service zookeeper-discovery, :time 1513972350029/1000, :tags [riemann-discovery production zookeeper], :state added, :ttl 60}\nINFO [2017-12-22 20:52:30,051] riemann task 1 - riemann.config - {:host api1, :service api-discovery, :time 1513972350029/1000, :tags [riemann-discovery api], :state added, :ttl 120}\nINFO [2017-12-22 20:52:30,052] riemann task 1 - riemann.config - {:host kafka2, :service kafka-discovery, :time 1513972350029/1000, :tags [riemann-discovery kafka], :state added, :ttl 60}\nINFO [2017-12-22 20:52:30,052] riemann task 1 - riemann.config - {:host kafka1, :service kafka-discovery, :time 1513972350029/1000, :tags [riemann-discovery kafka], :state added, :ttl 60}\n```\n\n**Note**: events with `:state` = \"added\" already indexed will not be reindexed, otherwise it will never expire.\n\nIf you remove hosts or services from the edn file, events with `:state \"removed\"` would be sent, and removed from the index (so no false alert).\n\nIf any of these events expire, this means that your datasource stopped or never emitted.\n\n*It's your responsability* to index events coming from your datasource to avoid expiration.\n\nIn my example i have `kafka` in the host `kafka1`. If i receive metrics from Kafka, i will index these events with `:service \"kafka-discovery\"`  to avoid expiration.\n\nThis means that you should carefully index events from your datasource and choose the right `:ttl` in your service discovery configuration, and use the right `:interval` option :\n\n- `:interval` should probably always be \u003c to your max `:ttl`, to be sure to remove removed hosts and services from the index before expiration.\n- You can also push your own events tagged `riemann-discovery` with `:state` = \"added\" or \"removed\", the `discovery-stream` will take care to add/remove the events from the index.\n\n### HTTP based discovery\n\nPeriodically send a GET request to an HTTP endpoint which must return the host/services list.\n\nPlease read the `File based discovery` section to understand how the plugin works and what are the default options.\n\n#### Plugin configuration\n\nIn your `riemann.config`:\n\n```clojure\n(discovery/discovery {:type :http\n                      :interval 120}\n                     {:url \"http://foo.foobar/discovery\"\n                      :http-options {}})\n```\n\n`:url` is the HTTP endpoint. It must returns a json body respecting this format:\n\n```javascript\n[{\"ttl\": 120,\n  \"services\": [{\"hosts\": [\"kafka1\",\n                          \"kafka2\"],\n                \"name\": \"kafka\",\n                \"ttl\": 60},\n               {\"hosts\": [\"api1\"],\n                \"name\": \"api\"}]},\n {\"services\": [{\"hosts\": [\"zookeeper1\"],\n                \"name\":\n                \"zookeeper\",\n                \"ttl\": 60}]}]\n```\n\nKeys and values are the same as the file discovery mechanism.\n\n`:http-options` could contains HTTP options from [clojure-http](https://github.com/dakrone/clj-http).\nDefault HTTP options are : `{:socket-timeout 1000 :conn-timeout 1000 :accept :json}`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcorbin%2Friemann-discovery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmcorbin%2Friemann-discovery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcorbin%2Friemann-discovery/lists"}