{"id":16847485,"url":"https://github.com/indygreg/collectd-carbon","last_synced_at":"2025-03-17T05:32:05.407Z","repository":{"id":1268737,"uuid":"1207826","full_name":"indygreg/collectd-carbon","owner":"indygreg","description":"collectd plugin to write to Carbon (frontend for Graphite's Whisper storage format)","archived":false,"fork":false,"pushed_at":"2014-02-03T16:04:39.000Z","size":181,"stargazers_count":146,"open_issues_count":3,"forks_count":29,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-02-27T18:13:30.448Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/indygreg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-12-30T06:19:24.000Z","updated_at":"2024-02-01T18:06:49.000Z","dependencies_parsed_at":"2022-08-16T12:50:32.534Z","dependency_job_id":null,"html_url":"https://github.com/indygreg/collectd-carbon","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indygreg%2Fcollectd-carbon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indygreg%2Fcollectd-carbon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indygreg%2Fcollectd-carbon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indygreg%2Fcollectd-carbon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/indygreg","download_url":"https://codeload.github.com/indygreg/collectd-carbon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243846976,"owners_count":20357294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T13:08:07.437Z","updated_at":"2025-03-17T05:32:05.044Z","avatar_url":"https://github.com/indygreg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\ncollectd-carbon is a [collectd](http://www.collectd.org/) plugin that\nwrites obtained values to Carbon.\n\nCarbon is a frontend to Whisper, which is a storage engine (similar to\nRRD). At this time, Carbon and Whisper are likely encountered alongside\n[Graphite](http://graphite.wikidot.com/start), a nifty real-time\ngraphing application.\n\nShort version: collectd-carbon is an alternative data writer to RRD.\n\n# Requirements\n\n* Collectd 4.9 or later (for the Python plugin) (A patch may be required\n  to fix the Python plugin - see below)\n* Python 2.4 or later\n* A running Carbon LineReceiver server (such as *carbon-cache.py*)\n\nIf you don't have Python configured and still wish to write Collectd\ninformation to Graphite/Carbon, see\nhttps://github.com/jssjr/collectd-write_graphite for a similar plugin\nwritten in C.\n\n# Version Information\n\ncollectd-carbon does not currently maintain release versions. The policy\n(at least on indygreg's original Github repository) is that the *master*\nbranch should always be stable and production ready.\n\nAs always, it is probably best to exercise discretion. If a commit has\nlanded recently, you may want to consider looking it over and/or waiting\na little while for others to test it for you.\n\n# Configuration\n\nThe plugin requires some configuration. This is done by passing\nparameters via the \u003cModule\u003e config section in your Collectd config. The\nfollowing parameters are recognized:\n\n* LineReceiverHost - hostname or IP address where a Carbon line receiver\n  is listening\n* LineReceiverPort - port on which line receiver is listening\n* LineReceiverProtocol - method in which you wish to send the packet to \n  the line receiver.  Options are: tcp or udp (default = tcp)\n* TypesDB - file(s) defining your Collectd types. This should be the\n  sames as your TypesDB global config parameters. If not specified, the\n  plugin will not work.\n* DifferentiateCounters - If present, the plugin will normalize COUNTER\n  and DERIVE types by recording the difference between two subsequent\n  values. See the section below.\n* DifferentiateCountersOverTime - If present, the plugin will divide\n  the values of COUNTER and DERIVE types by the interval between the\n  two subsequent values. Implies DifferentiateCounters. See the section\n  below.\n* LowercaseMetricNames - If present, all metric names will be converted\n  to lower-case.\n* MetricPrefix - If present, all metric names will contain this string\n  prefix. Do not include a trailing period.\n* HostPostfix - If present, all hostnames will contain this string\n  postfix (my_host_name.HostPostfix). Do not include a trailing period.\n* HostSeparator - If present, will change the default behavior of\n  inserting a \"_\" between the hostname (my_host_name) to this value.\n* MetricSeparator - If present, will change the default behavior of inserting\n  a \".\" within type and type_instances. This can be used to prevent deep\n  paths in Graphite/carbon for IPs and hostnames.\n\n## Example\n\nThe following is an example Collectd configuration for this plugin:\n\n    \u003cLoadPlugin \"python\"\u003e\n        Globals true\n    \u003c/LoadPlugin\u003e\n\n    \u003cPlugin \"python\"\u003e\n        # carbon_writer.py is at path /opt/collectd-plugins/carbon_writer.py\n        ModulePath \"/opt/collectd-plugins/\"\n\n        Import \"carbon_writer\"\n\n        \u003cModule \"carbon_writer\"\u003e\n            LineReceiverHost \"myhost.mydomain\"\n            LineReceiverPort 2003\n            LineReceiverProtocol \"tcp\"\n            DifferentiateCountersOverTime true\n            LowercaseMetricNames true\n            TypesDB \"/usr/share/collectd/types.db\"\n        \u003c/Module\u003e\n    \u003c/Plugin\u003e\n\n# Operational Notes\n\nIf the connection to the line receiver cannot be established or goes\nbad, the plugin will automatically attempt to reconnect. If connections\nfail, the plugin will reconnect at most once every 10 seconds. This\nprevents many likely failures from occurring when the server is down.\n\nThe plugin needs to parse Collectd type files. If there was an error\nparsing a specific type (look for log messages at Collectd startup\ntime), the plugin will fail to write values for this type. It will\nsimply skip over them and move on to the next value. It will write a log\nmessage every time this happens so you can correct the problem.\n\nThe plugin needs to perform redundant parsing of the type files because\nthe Collectd Python API does not provide an interface to the types\ninformation (unlike the Perl and Java plugin APIs). Hopefully this will\nbe addressed in a future version of Collectd.\n\n# Data Mangling\n\nCollectd data is collected/written in discrete tuples having the\nfollowing:\n\n    (host, plugin, plugin_instance, type, type_instance, time, interval, metadata, values)\n\n_values_ is itself a list of { counter, gauge, derive, absolute }\n(numeric) values. To further complicate things, each distinct _type_ has\nits own definition corresponding to what's in the _values_ field.\n\nGraphite, by contrast, deals with tuples of ( metric, value, time ). So,\nwe effectively need to mangle all those extra fields down into the\n_metric_ value.\n\nThis plugin mangles the fields to the metric name:\n\n    host.plugin[.plugin_instance].type[.type_instance].data_source\n\nWhere *data_source* is the name of the data source (i.e. ds_name) in the\ntype being written.\n\nFor example, the Collectd distribution has a built-in _df_ type:\n\n    df used:GAUGE:0:1125899906842623, free:GAUGE:0:1125899906842623\n\nThe *data_source* values for this type would be *used* and *free*\nyielding the metrics (along the lines of)\n*hostname_domain.plugin.df.used* and *hostname_domain.plugin.df.free*.\n\n## COUNTER and DERIVE Types\n\nCollectd data types, like RRDTool, differentiate between ABSOLUTE,\nCOUNTER, DERIVE, and GAUGE types. When values are stored in RRDTool,\nthese types invoke special functionality. However, they do nothing\nspecial in Carbon. And, if you are using Graphite, they complicate\nmatters because you'll want to apply a derivative function to COUNTER\nand DERIVE types to obtain any useful values.\n\nWhen the plugin is configured with the *DifferentiateCounters* flag,\nthe plugin will send the difference between two data points to Carbon.\nPlease note the following regarding behavior:\n\n* Data is sent to Carbon after receiving the 2nd data point. This is\n  because the plugin must establish an initial value from which to\n  calculate the difference.\n* The plugin is aware of the minimum and maximum values and will handle\n  overflows and wrap-arounds properly.\n* An overflow for a type with max value *U* is treated as an initial\n  value. i.e. you will lose one data point.\n* A minimum value of *U* is treated as *0*.\n\nPlease note that *DifferentiateCounters* stores the difference, not the\nactually time-derivative, between 2 values. This can lead to some\nunexpected, but still valid, behavior. For example, with a Collectd\npolling interval of 10s, values dispatches from the CPU plugin will be\nnormalized to around 1000 (100 per each second). RRD, by contrast, would\nnormalize values to around 100. If graphed, the values should form the\nsame shapes, but off by a scale of 10:1.\n\nIf *DifferentiateCountersOverTime* is configured, the actual\ntime-derivative betwen the values is stored. This results in CPU plugin\nresults that are normalized to around 100 (like RRD).\n\n*DifferentiateCountersOverTime* implies *DifferentiateCounters*.\n\n# Troubleshooting\n\nYou can see the raw values dispatched to carbon by packet sniffing:\n\n    $ sudo ngrep -qd any . tcp dst port 2003\n\nor\n\n    $ sudo ngrep -qd any . udp dst port 2003\n\n## Collectd Python Write Callback Bug\n\nCollectd versions through 4.10.2 and 4.9.4 have a bug in the Python\nplugin where Python would receive bad values for certain data sets. The\nbug would typically manifest as data values appearing to be 0. The\noriginal author of this plugin identified the bug and sent a fix to the\nCollectd development team.\n\nCollectd versions 4.9.5, 4.10.3, and 5.0.0 are the first official\nversions with a fix for this bug. If you are not running one of these\nversions or have not applied the fix (which can be seen at\n\u003chttps://github.com/indygreg/collectd/commit/31bc4bc67f9ae12fb593e18e0d3649e5d4fa13f2\u003e),\nyou will likely dispatch wrong values to Carbon.\n\n## Collectd 4.10.3 on EL5 ImportError\n\nUsing the plugin with collectd-4.10.3 from EPEL5 on RHEL or CentOS 5.x\nmay produce the following error:\n\n    Jul 20 14:54:38 mon0 collectd[2487]: plugin_load_file: The global flag is not supported, libtool 2 is required for this.\n    Jul 20 14:54:38 mon0 collectd[2487]: python plugin: Error importing module \"carbon_writer\".\n    Jul 20 14:54:38 mon0 collectd[2487]: Unhandled python exception in importing module: ImportError: /usr/lib64/python2.4/lib-dynload/_socketmodule.so: undefined symbol: PyExc_ValueError\n    Jul 20 14:54:38 mon0 collectd[2487]: python plugin: Found a configuration for the \"carbon_writer\" plugin, but the plugin isn't loaded or didn't register a configuration callback.\n    Jul 20 14:54:38 mon0 collectd[2488]: plugin_dispatch_values: No write callback has been registered. Please load at least one output plugin, if you want the collected data to be stored.\n    Jul 20 14:54:38 mon0 collectd[2488]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status 2 (ENOENT). Most likely this means you didn't load any write plugins.\n\nThis may also occur on other operating systems. It is caused by a\nlibtool/libltdl quirk described in [this mailing list thread](http://mailman.verplant.org/pipermail/collectd/2008-March/001616.html).\nAs per the workarounds detailed there, you may either:\n\n 1. Modify the init script.\n\n        @@ -25,7 +25,7 @@\n                echo -n $\"Starting $prog: \"\n                if [ -r \"$CONFIG\" ]\n                then\n        -               daemon /usr/sbin/collectd -C \"$CONFIG\"\n        +               LD_PRELOAD=/usr/lib64/libpython2.4.so daemon /usr/sbin/collectd -C \"$CONFIG\"\n                        RETVAL=$?\n                        echo\n                        [ $RETVAL -eq 0 ] \u0026\u0026 touch /var/lock/subsys/$prog\n\n 1. Modify the RPM and rebuild.\n\n        @@ -182,7 +182,7 @@\n\n\n         %build\n        -%configure \\\n        +%configure CFLAGS=-\"DLT_LAZY_OR_NOW='RTLD_LAZY|RTLD_GLOBAL'\" \\\n             --disable-static \\\n             --disable-ascent \\\n             --disable-apple_sensors \\\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findygreg%2Fcollectd-carbon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Findygreg%2Fcollectd-carbon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findygreg%2Fcollectd-carbon/lists"}