{"id":27353600,"url":"https://github.com/i-ky/effluence","last_synced_at":"2025-04-12T21:06:29.447Z","repository":{"id":42456952,"uuid":"147878510","full_name":"i-ky/effluence","owner":"i-ky","description":"Zabbix loadable module for real-time export of history to InfluxDB","archived":false,"fork":false,"pushed_at":"2025-01-01T19:49:06.000Z","size":63,"stargazers_count":32,"open_issues_count":8,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T21:06:23.370Z","etag":null,"topics":["hacktoberfest","influx","influxdb","influxdb-zabbix","zabbix","zabbix-loadable","zabbix-server"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/i-ky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-07T21:56:22.000Z","updated_at":"2025-01-01T19:49:10.000Z","dependencies_parsed_at":"2025-01-01T20:28:44.483Z","dependency_job_id":"c0f71ece-f875-4822-a61c-0376b5514f7e","html_url":"https://github.com/i-ky/effluence","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-ky%2Feffluence","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-ky%2Feffluence/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-ky%2Feffluence/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-ky%2Feffluence/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/i-ky","download_url":"https://codeload.github.com/i-ky/effluence/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631685,"owners_count":21136562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","influx","influxdb","influxdb-zabbix","zabbix","zabbix-loadable","zabbix-server"],"created_at":"2025-04-12T21:06:28.760Z","updated_at":"2025-04-12T21:06:29.434Z","avatar_url":"https://github.com/i-ky.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# effluence [![Travis CI Build Status](https://app.travis-ci.com/i-ky/effluence.svg?branch=master)](https://app.travis-ci.com/i-ky/effluence) [![Coverity Scan Build Status](https://img.shields.io/coverity/scan/17710.svg)](https://scan.coverity.com/projects/i-ky-effluence) [![Gitpod Ready-to-Code](https://img.shields.io/badge/Gitpod-Ready--to--Code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/i-ky/effluence)\n\n[Zabbix](http://www.zabbix.com)\n[loadable module](https://www.zabbix.com/documentation/current/manual/config/items/loadablemodules)\nfor real-time export of history to\n[InfluxDB](https://www.influxdata.com/products/influxdb-overview/)\n\n**Disclaimer**:\nAs\n[the name](https://www.urbandictionary.com/define.php?term=effluence)\nsuggests,\nthe module is not ready yet for the use in production,\nbut testing it is very much appreciated\nand any suggestion on performance tuning is welcomed!\nOne size does not fit all\nand you may find that\n[zensqlmonitor/influxdb-zabbix](https://github.com/zensqlmonitor/influxdb-zabbix)\nor\n[LMacPhail/zabbix-history-influxdb](https://github.com/LMacPhail/zabbix-history-influxdb)\nwill suit your needs better.\n\n## background\n\nFrom its inception,\nusing an SQL database for storing both configuration and collected monitoring data\nwas one of distinctive features of Zabbix.\nHowever,\nas Zabbix was gaining acceptance among large enterprises,\nthe limitations of such approach became evident.\nDespite all the optimizations made by Zabbix team over the years,\nfor large setups performance of the database often becomes a bottleneck,\nperformance of `insert` queries towards `history*` and `trends*` tables in particular.\n\nTypical ways to cure the problem are:\n- partitioning `history*` and `trends*` tables;\n- replacing HDDs with SSDs on the database server;\n- using clustered drop-in replacements of supported database backends (e.g. [Percona](https://www.percona.com))\n\nBut everyone seems to agree that the ultimate solution would be to offload historical data into a more suitable storage engine.\n\nMultiple NoSQL solutions and numerous time-series databases were named as alternatives to SQL.\nThe relevant discussion can be found\n[here](https://support.zabbix.com/browse/ZBXNEXT-714).\nSince there is no agreement among experts on which exactly NoSQL solution or time-series database Zabbix should use,\nthere were\n[attempts](https://support.zabbix.com/browse/ZBXNEXT-3661)\nfrom Zabbix side to provide a generic interface for historical data storage backend,\nleaving implementation of specific adapters to the community.\nUnfortunately, as of now, these efforts have not really \"took off\".\n\n## purpose\n\nThe goal of this module is to replicate historical data in the database of Zabbix to InfluxDB database(s).\nReplication happens in real time,\ntherefore data is available in InfluxDB at the same time as in Zabbix DB.\n\nUnfortunately, the information provided to modules by Zabbix server is very scarce: values themselves, timestamps and unique item identifiers.\nTo make sense of these data (specifically item identifiers),\none has to additionally use Zabbix API to find out item names, host names, etc.\nLuckily, there is a piece of software which does exactly that - The Magnificent\n[Zabbix plugin](https://grafana.com/plugins/alexanderzobnin-zabbix-app)\nfor\n[Grafana](https://grafana.com)!\n\nSince the large pile of historical data is mostly used in Zabbix for graphs,\noffloading this task to InfluxDB and Grafana combo will make huge `history*` and `trends*` tables redundant.\nYou will need to keep the bare minimum there,\njust enough to populate value cache upon Zabbix server restarts\nand resolve `{ITEM.VALUE}` macros.\nAs we know, small tables are fast\nand as a consequence database write operations will no longer limit the performance of Zabbix server!\n\n## quick start\n\n### get binaries\n\nCheck out\n[the latest release](https://github.com/i-ky/effluence/releases/latest)\nfor precompiled binaries or\n[compile](./COMPILE.md)\nfrom sources yourself.\n\n### install\n\n1. Install\n[`libcurl`](https://curl.haxx.se/libcurl/)\nand\n[`libyaml`](https://pyyaml.org/wiki/LibYAML)\nin case you are using precompiled binaries or\nwere compiling on a different machine,\nsort of:\n * CentOS:\n```bash\nsudo yum install libcurl libyaml\n```\n * Ubuntu:\n```bash\nsudo apt install libcurl4 libyaml-0-2\n```\n\n2. Copy `effluence.so` to a desired location.\n```bash\ncp effluence.so /path/to/zabbix/modules/\n```\n\n3. Set up necessary permissions.\n\n### configure\n\n1. Create\n[database(s)](https://docs.influxdata.com/influxdb/latest/introduction/getting-started/#creating-a-database)\nand\n[user(s)](http://docs.influxdata.com/influxdb/latest/administration/authentication_and_authorization)\nin your InfluxDB instance(s).\n\n2. Create module\n[configuration file](#configuration-file-format)\n(e.g. `/path/to/effluence/config.yaml`):\n```yaml\nurl:  http://localhost:8086\ndb:   zabbix\nuser: effluence\npass: r3a11y_$tr0n9_pa$$w0rd\n```\n\u003e This is a very simple configuration to export all data types.\nHowever, if your only goal is integration with Grafana,\nconsider exporting only numeric data types (learn\n[here](#configuration-file-for-exporting-numeric-values-only)\nhow to achieve this).\nAt the moment Zabbix plugin queries other data types from Zabbix API.\n\n3. Set `EFFLU_CONFIG` environment variable for Zabbix server with the path to module configuration file.\nAssuming Zabbix server is a service:\n```bash\nsudo systemctl edit zabbix-server\n```\nThis will open a text editor,\ntype there something like:\n```\n[Service]\nEnvironment='EFFLU_CONFIG=/path/to/effluence/config.yaml'\n```\n...and save.\nThis creates a drop-in file for `zabbix-server` unit keeping its main unit file intact.\n\n4. Set `LoadModulePath` and `LoadModule` parameters in Zabbix\n[server](https://www.zabbix.com/documentation/current/manual/appendix/config/zabbix_server)\nconfiguration file.\n```\nLoadModulePath=/path/to/zabbix/modules\nLoadModule=effluence.so\n```\n\n5. Restart Zabbix server.\n\n### enjoy\n\nBut keep in mind that there is\n[more](#boring-details)\nto learn when you come back later.\nYou may want to fine-tune\n[configuration](#configuration-file-format)\nand know how to keep\n[database size](#database-sizing)\nunder control.\nIf you are **not** planning to use Zabbix plugin for Grafana,\nthere is\n[some useful information](#database-schema)\nregarding data representation in InfluxDB.\nIt will be handy if you decide to\n[explore your data](https://docs.influxdata.com/influxdb/latest/query_language/data_exploration)\nusing InfluxQL.\n\n## boring details\n\n### configuration file format\n\nModule uses\n[YAML Ain't Markup Language](http://yaml.org)\nformat of configuration file.\nHere is the list of attributes one can specify there:\n\nattribute |           | description\n----------|-----------|------------\n`url`     | mandatory | URL to send requests to (`/write` is added automatically)\n`db`      | mandatory | database where to store data\n`user`    | optional  | username to use for authentication\n`pass`    | optional  | password to use for authentication\n\nThese attributes can be specified in the root of the document,\nthis way they will have global effect.\nWhen different types need to be stored\nin different InfluxDB instances,\ndifferent databases\nor on behalf of different users,\nsame attributes can be specified per data type\n(names should be familiar to anyone who has ever tried to\n[setup Zabbix with Elasticsearch](https://www.zabbix.com/documentation/current/manual/appendix/install/elastic_search_setup)):\n\ndata type | *Type of information*\n----------|----------------------\n`dbl`     | *Numeric (float)*\n`uint`    | *Numeric (unsigned)*\n`str`     | *Character*\n`text`    | *Text*\n`log`     | *Log*\n\nIf there is no section for a specific data type,\nmodule will use global attributes (if provided).\nIf the configuration of a particular data type lacks any one of mandatory attributes,\nthen the callback for that data type is not provided to Zabbix server\nand the data of this type are not being exported.\n\n#### configuration file examples\n\n##### minimalist configuration file\n\nWhen you have just installed InfluxDB\nand have not enabled authentication yet,\nyou can use the following configuration.\nAll data types will be exported\nand stored together.\n\n```yaml\n# one set of attributes for all data types\nurl:  http://localhost:8086\ndb:   zabbix\n```\n\n##### configuration with a special place for numeric types\n\nWith the following configuration all data types will be exported,\nbut `dbl` will be sent to a different URL,\nwhile `uint` is stored to a different database.\n\n```yaml\n# global attributes (will be used for Character, Text and Log)\nurl:  http://localhost:8086\ndb:   zabbix\nuser: effluence\npass: r3a11y_$tr0n9_pa$$w0rd\n\ndbl: # specifically for Numeric (float)\n  url:  http://very.special.place\n  db:   float_only\n  user: ecneulffe\n  pass: w34k_p4$$w0rd\n\nuint: # specifically for Numeric (unsigned)\n  url: http://only.for.unsigned\n  db:  not_that_special_but_still\n  # no authentication, global configuration does not apply\n```\n\n##### configuration file using YAML alias\n\nThis configuration file shows\nhow to avoid duplication\nand copy-paste errors\nusing YAML *aliases* and *references*.\n\n```yaml\nstr: # Character\n  url: \u0026url http://localhost:8086\n  db: shorties\n\ntext: # Text\n  url: *url # reference to \u0026url\n  db: longies\n```\n\n##### configuration file for exporting numeric values only\n\nAnother way to provide identical attributes for different data types is to use *array* of data types as *key* for type-specific attribute section.\n\n```yaml\n[dbl, uint]: # for both Numeric (float) and Numeric (unsigned)\n  url: http://localhost:8086\n  db: numeric\n  user: effluence\n  pass: r3a11y_$tr0n9_pa$$w0rd\n```\n\n### database schema\n\nRegardless of configured `url` and `database`\neach data type will be stored in separate\n[measurement](https://docs.influxdata.com/influxdb/latest/concepts/glossary/#measurement).\n\n*Type of information* | measurement\n----------------------|------------\n*Numeric (float)*     | `history`\n*Numeric (unsigned)*  | `history_uint`\n*Character*           | `history_str`\n*Text*                | `history_text`\n*Log*                 | `history_log`\n\nAs you see, measurement names are the same as table names in Zabbix DB.\nThere is a good reason for this.\nMeasurements are analogous to SQL tables and\nhaving same names simplifies writing\n[queries](https://docs.influxdata.com/influxdb/latest/query_language/data_exploration/)\nfor different backends.\n\nAll measurements will have the only [tag](https://docs.influxdata.com/influxdb/latest/concepts/glossary/#tag) named `itemid` and a [field](https://docs.influxdata.com/influxdb/latest/concepts/glossary/#field) named `value`.\nMeasurement `history_log` will additionally have `source`, `timestamp`, `logeventid` and `severity` fields.\nAnd of course every datapoint in InfluxDB has a `time` associated with it.\n\nOne important thing to note is that *Numeric (unsigned)* values are stored as floats,\nbecause unlike Zabbix which uses *unsigned* 64 bit integers,\nInfluxDB prefers *signed* 64 bit integers.\nSince it wouldn't be possible to squeeze largest Zabbix integers into InfluxDB integer type\nand InfluxDB\n[does not support fields changing type on the fly](https://docs.influxdata.com/influxdb/latest/write_protocols/line_protocol_reference/#field-type-discrepancies),\ndecision was made to store all numeric values as floats.\nYes, for largest values some precision will be lost,\nbut hopefully it won't be noticeable.\n\n### database sizing\n\nFortunately, InfluxDB provides enough\n[tools](https://docs.influxdata.com/influxdb/latest/guides/downsampling_and_retention/)\nto solve this problem.\nThis allows to keep module simple and ~~stupid~~ reliable.\nThis also provides more flexibility to the end user because no parameters are hard coded.\n\nInfluxDB has a mechanism of\n[retention policies](https://docs.influxdata.com/influxdb/latest/concepts/glossary/#retention-policy-rp)\nto keep the size of the database under control.\nPlease [create](https://docs.influxdata.com/influxdb/latest/query_language/database_management/#retention-policy-management)\none and make it `default`,\nbecause module will only write data into default retention policy of the database.\nRetention policies specify how long InfluxDB stores the data.\nWithout retention policy data is never deleted from InfluxDB and your database may grow out of control.\n\nOne more sensible thing to do is to aggregate older numeric data.\nThis will be good for both database size\nand performance of queries with longer intervals.\nIn Zabbix this concept is known as\n[history and trends](https://www.zabbix.com/documentation/current/manual/config/items/history_and_trends),\nin InfluxDB same can be achieved using\n[continuous queries](https://docs.influxdata.com/influxdb/latest/query_language/continuous_queries).\n[Create](https://docs.influxdata.com/influxdb/latest/query_language/database_management/#retention-policy-management)\none more retention policy for trends (non-`default` this time) with longer (or even infinite) storage duration.\nPopulate it with a couple of continuous queries like these two\n(assuming database is called `zabbix`\nand retention policy is `longterm`):\n```sql\nCREATE CONTINUOUS QUERY trends ON zabbix\nBEGIN\n    SELECT\n        count(value) AS num,\n        min(value) AS value_min,\n        mean(value) AS value_avg,\n        max(value) AS value_max\n    INTO\n        zabbix.longterm.:MEASUREMENT\n    FROM\n        /history(?:_uint)?$/\n    GROUP BY\n        time(1h,2h),*\nEND\n```\n\u003e Users of Zabbix plugin for Grafana should refer to plugin's documentation to get the **exact** measurement and field names.\nOtherwise plugin won't be able to find the data!\n\nInfluxDB lets user to be creative and does not limit you to one aggregation stage.\nFor example,\nyou can keep raw data for a day\nand aggregate it on an hourly basis.\nThen you can keep hourly data for a week\nand further aggregate it on a daily basis\nbefore storing it forever.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-ky%2Feffluence","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fi-ky%2Feffluence","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-ky%2Feffluence/lists"}