{"id":13563360,"url":"https://github.com/fluent/fluent-plugin-prometheus","last_synced_at":"2025-04-03T20:30:35.472Z","repository":{"id":31589269,"uuid":"35154080","full_name":"fluent/fluent-plugin-prometheus","owner":"fluent","description":"A fluent plugin that collects metrics and exposes for Prometheus.","archived":false,"fork":false,"pushed_at":"2024-08-09T00:40:07.000Z","size":240,"stargazers_count":257,"open_issues_count":60,"forks_count":79,"subscribers_count":31,"default_branch":"master","last_synced_at":"2024-10-29T16:21:21.947Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-06T10:45:47.000Z","updated_at":"2024-10-26T20:25:10.000Z","dependencies_parsed_at":"2024-05-31T12:06:47.567Z","dependency_job_id":"c6fc5a3a-557b-4533-849b-ad261de7de82","html_url":"https://github.com/fluent/fluent-plugin-prometheus","commit_stats":{"total_commits":201,"total_committers":35,"mean_commits":5.742857142857143,"dds":0.7064676616915423,"last_synced_commit":"ee4cfe13a247ccd4cd3adcf529730059cdc82996"},"previous_names":["kazegusuri/fluent-plugin-prometheus"],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-prometheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-prometheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-prometheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-prometheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent","download_url":"https://codeload.github.com/fluent/fluent-plugin-prometheus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222340175,"owners_count":16968841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:18.369Z","updated_at":"2024-11-04T16:30:26.379Z","avatar_url":"https://github.com/fluent.png","language":"Ruby","readme":"# fluent-plugin-prometheus, a plugin for [Fluentd](https://www.fluentd.org)\n\n[![Build Status](https://travis-ci.org/fluent/fluent-plugin-prometheus.svg?branch=master)](https://travis-ci.org/fluent/fluent-plugin-prometheus)\n\nA fluent plugin that instruments metrics from records and exposes them via web interface. Intended to be used together with a [Prometheus server](https://github.com/prometheus/prometheus).\n\n## Requirements\n\n| fluent-plugin-prometheus | fluentd    | ruby   |\n|--------------------------|------------|--------|\n| 1.x.y                    | \u003e= v1.9.1  | \u003e= 2.4 |\n| 1.[0-7].y                | \u003e= v0.14.8 | \u003e= 2.1 |\n| 0.x.y                    | \u003e= v0.12.0 | \u003e= 1.9 |\n\nSince v1.8.0, fluent-plugin-prometheus uses [http_server helper](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server) to launch HTTP server.\nIf you want to handle lots of connections, install `async-http` gem.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'fluent-plugin-prometheus'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install fluent-plugin-prometheus\n\n## Usage\n\nfluentd-plugin-prometheus includes 6 plugins.\n\n- `prometheus` input plugin\n- `prometheus_monitor` input plugin\n- `prometheus_output_monitor` input plugin\n- `prometheus_tail_monitor` input plugin\n- `prometheus` output plugin\n- `prometheus` filter plugin\n\nSee [sample configuration](./misc/fluentd_sample.conf), or try [tutorial](#try-plugin-with-nginx).\n\n### prometheus input plugin\n\nYou have to configure this plugin to expose metrics collected by other Prometheus plugins.\nThis plugin provides a metrics HTTP endpoint to be scraped by a Prometheus server on 24231/tcp(default).\n\nWith following configuration, you can access http://localhost:24231/metrics on a server where fluentd running.\n\n```\n\u003csource\u003e\n  @type prometheus\n\u003c/source\u003e\n```\n\nMore configuration parameters:\n\n- `bind`: binding interface (default: '0.0.0.0')\n- `port`: listen port (default: 24231)\n- `metrics_path`: metrics HTTP endpoint (default: /metrics)\n- `aggregated_metrics_path`: metrics HTTP endpoint (default: /aggregated_metrics)\n- `content_encoding`: encoding format for the exposed metrics (default: identity). Supported formats are {identity, gzip}\n\nWhen using multiple workers, each worker binds to port + `fluent_worker_id`.\nTo scrape metrics from all workers at once, you can access http://localhost:24231/aggregated_metrics.\n\n#### TLS setting\n\nUse `\u003ctrasnport tls\u003e`. See [transport config article](https://docs.fluentd.org/configuration/transport-section) for more details.\n\n```\n\u003csource\u003e\n  @type prometheus\n  \u003ctransport tls\u003e\n    # TLS parameters...\n  \u003c/transport\n\u003c/source\u003e\n```\n\n### prometheus_monitor input plugin\n\nThis plugin collects internal metrics in Fluentd. The metrics are similar to/part of [monitor_agent](https://docs.fluentd.org/input/monitor_agent).\n\n\n#### Exposed metrics\n\n- `fluentd_status_buffer_queue_length`\n- `fluentd_status_buffer_total_bytes`\n- `fluentd_status_retry_count`\n- `fluentd_status_buffer_newest_timekey` from fluentd v1.4.2\n- `fluentd_status_buffer_oldest_timekey` from fluentd v1.4.2\n\n#### Configuration\n\nWith following configuration, those metrics are collected.\n\n```\n\u003csource\u003e\n  @type prometheus_monitor\n\u003c/source\u003e\n```\n\nMore configuration parameters:\n\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `interval`: interval to update monitor_agent information in seconds (default: 5)\n\n### prometheus_output_monitor input plugin\n\nThis plugin collects internal metrics for output plugin in Fluentd. This is similar to `prometheus_monitor` plugin, but specialized for output plugin. There are Many metrics `prometheus_monitor` does not include, such as `num_errors`, `retry_wait` and so on.\n\n#### Exposed metrics\n\nMetrics for output\n\n- `fluentd_output_status_retry_count`\n- `fluentd_output_status_num_errors`\n- `fluentd_output_status_emit_count`\n- `fluentd_output_status_retry_wait`\n    - current retry_wait computed from last retry time and next retry time\n- `fluentd_output_status_emit_records`\n- `fluentd_output_status_write_count`\n- `fluentd_output_status_rollback_count`\n- `fluentd_output_status_flush_time_count` in milliseconds from fluentd v1.6.0\n- `fluentd_output_status_slow_flush_count` from fluentd v1.6.0\n\nMetrics for buffer\n\n- `fluentd_output_status_buffer_total_bytes`\n- `fluentd_output_status_buffer_stage_length` from fluentd v1.6.0\n- `fluentd_output_status_buffer_stage_byte_size` from fluentd v1.6.0\n- `fluentd_output_status_buffer_queue_length`\n- `fluentd_output_status_buffer_queue_byte_size` from fluentd v1.6.0\n- `fluentd_output_status_buffer_newest_timekey` from fluentd v1.6.0\n- `fluentd_output_status_buffer_oldest_timekey` from fluentd v1.6.0\n- `fluentd_output_status_buffer_available_space_ratio` from fluentd v1.6.0\n\n#### Configuration\n\nWith following configuration, those metrics are collected.\n\n```\n\u003csource\u003e\n  @type prometheus_output_monitor\n\u003c/source\u003e\n```\n\nMore configuration parameters:\n\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `interval`: interval to update monitor_agent information in seconds (default: 5)\n- `gauge_all`: Specify metric type. If `true`, use `gauge` type. If `false`, use `counter` type. Since v2, this parameter will be removed and use `counter` type.\n\n### prometheus_tail_monitor input plugin\n\nThis plugin collects internal metrics for in_tail plugin in Fluentd. in_tail plugin holds internal state for files that the plugin is watching. The state is sometimes important to monitor plugins work correctly.\n\nThis plugin uses internal class of Fluentd, so it's easy to break.\n\n#### Exposed metrics\n\n- `fluentd_tail_file_position`: Current bytes which plugin reads from the file\n- `fluentd_tail_file_inode`: inode of the file\n- `fluentd_tail_file_closed`: Number of closed files\n- `fluentd_tail_file_opened`: Number of opened files\n- `fluentd_tail_file_rotated`: Number of rotated files\n- `fluentd_tail_file_throttled`: Number of times files got throttled (only with fluentd version \u003e 1.17)\n\nDefault labels:\n\n- `plugin_id`: a value set for a plugin in configuration.\n- `type`: plugin name. `in_tail` only for now.\n- `path`: file path\n\n#### Configuration\n\nWith following configuration, those metrics are collected.\n\n```\n\u003csource\u003e\n  @type prometheus_tail_monitor\n\u003c/source\u003e\n```\n\nMore configuration parameters:\n\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `interval`: interval to update monitor_agent information in seconds (default: 5)\n\n### prometheus output/filter plugin\n\nBoth output/filter plugins instrument metrics from records. Both plugins have no impact against values of each records, just read.\n\nAssuming you have following configuration and receiving message,\n\n```\n\u003cmatch message\u003e\n  @type stdout\n\u003c/match\u003e\n```\n\n```\nmessage {\n  \"foo\": 100,\n  \"bar\": 200,\n  \"baz\": 300\n}\n```\n\nIn filter plugin style,\n\n```\n\u003cfilter message\u003e\n  @type prometheus\n  \u003cmetric\u003e\n    name message_foo_counter\n    type counter\n    desc The total number of foo in message.\n    key foo\n  \u003c/metric\u003e\n\u003c/filter\u003e\n\n\u003cmatch message\u003e\n  @type stdout\n\u003c/match\u003e\n```\n\nIn output plugin style:\n\n```\n\u003cfilter message\u003e\n  @type prometheus\n  \u003cmetric\u003e\n    name message_foo_counter\n    type counter\n    desc The total number of foo in message.\n    key foo\n  \u003c/metric\u003e\n\u003c/filter\u003e\n\n\u003cmatch message\u003e\n  @type copy\n  \u003cstore\u003e\n    @type prometheus\n    \u003cmetric\u003e\n      name message_foo_counter\n      type counter\n      desc The total number of foo in message.\n      key foo\n    \u003c/metric\u003e\n  \u003c/store\u003e\n  \u003cstore\u003e\n    @type stdout\n  \u003c/store\u003e\n\u003c/match\u003e\n```\n\nWith above configuration, the plugin collects a metric named `message_foo_counter` from key `foo` of each records.\n\nYou can access nested keys in records via dot or bracket notation (https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax), for example: `$.kubernetes.namespace`, `$['key1'][0]['key2']`. The record accessor is enable only if the value starts with `$.` or `$[`.\n\nSee Supported Metric Type and Labels for more configuration parameters.\n\n## Supported Metric Types\n\nFor details of each metric type, see [Prometheus documentation](http://prometheus.io/docs/concepts/metric_types/). Also see [metric name guide](http://prometheus.io/docs/practices/naming/).\n\n### counter type\n\n```\n\u003cmetric\u003e\n  name message_foo_counter\n  type counter\n  desc The total number of foo in message.\n  key foo\n  \u003clabels\u003e\n    tag ${tag}\n    host ${hostname}\n    foo bar\n  \u003c/labels\u003e\n\u003c/metric\u003e\n```\n\n- `name`: metric name (required)\n- `type`: metric type (required)\n- `desc`: description of this metric (required)\n- `key`: key name of record for instrumentation (**optional**)\n- `initialized`: boolean controlling initilization of metric (**optional**). See [Metric initialization](#metric-initialization)\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `\u003cinitlabels\u003e`: labels to use for initialization of ReccordAccessors/Placeholder labels (**optional**). See [Metric initialization](#metric-initialization)\n\nIf key is empty, the metric values is treated as 1, so the counter increments by 1 on each record regardless of contents of the record.\n\n### gauge type\n\n```\n\u003cmetric\u003e\n  name message_foo_gauge\n  type gauge\n  desc The total number of foo in message.\n  key foo\n  \u003clabels\u003e\n    tag ${tag}\n    host ${hostname}\n    foo bar\n  \u003c/labels\u003e\n\u003c/metric\u003e\n```\n\n- `name`: metric name (required)\n- `type`: metric type (required)\n- `desc`: description of metric (required)\n- `key`: key name of record for instrumentation (required)\n- `initialized`: boolean controlling initilization of metric (**optional**). See [Metric initialization](#metric-initialization)\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `\u003cinitlabels\u003e`: labels to use for initialization of ReccordAccessors/Placeholder labels (**optional**). See [Metric initialization](#metric-initialization)\n\n### summary type\n\n```\n\u003cmetric\u003e\n  name message_foo\n  type summary\n  desc The summary of foo in message.\n  key foo\n  \u003clabels\u003e\n    tag ${tag}\n    host ${hostname}\n    foo bar\n  \u003c/labels\u003e\n\u003c/metric\u003e\n```\n\n- `name`: metric name (required)\n- `type`: metric type (required)\n- `desc`: description of metric (required)\n- `key`: key name of record for instrumentation (required)\n- `initialized`: boolean controlling initilization of metric (**optional**). See [Metric initialization](#metric-initialization)\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `\u003cinitlabels\u003e`: labels to use for initialization of ReccordAccessors/Placeholder labels (**optional**). See [Metric initialization](#metric-initialization)\n\n### histogram type\n\n```\n\u003cmetric\u003e\n  name message_foo\n  type histogram\n  desc The histogram of foo in message.\n  key foo\n  buckets 0.1, 1, 5, 10\n  \u003clabels\u003e\n    tag ${tag}\n    host ${hostname}\n    foo bar\n  \u003c/labels\u003e\n\u003c/metric\u003e\n```\n\n- `name`: metric name (required)\n- `type`: metric type (required)\n- `desc`: description of metric (required)\n- `key`: key name of record for instrumentation (required)\n- `initialized`: boolean controlling initilization of metric (**optional**). See [Metric initialization](#metric-initialization)\n- `buckets`: buckets of record for instrumentation (optional)\n- `\u003clabels\u003e`: additional labels for this metric (optional). See [Labels](#labels)\n- `\u003cinitlabels\u003e`: labels to use for initialization of ReccordAccessors/Placeholder labels (**optional**). See [Metric initialization](#metric-initialization)\n\n## Labels\n\nSee [Prometheus Data Model](http://prometheus.io/docs/concepts/data_model/) first.\n\nYou can add labels with static value or dynamic value from records. In `prometheus_monitor` input plugin, you can't use label value from records.\n\n### labels section\n\n```\n\u003clabels\u003e\n  key1 value1\n  key2 value2\n\u003c/labels\u003e\n```\n\nAll labels sections has same format. Each lines have key/value for label.\n\nYou can access nested fields in records via dot or bracket notation (https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax), for example: `$.kubernetes.namespace`, `$['key1'][0]['key2']`. The record accessor is enable only if the value starts with `$.` or `$[`. Other values are handled as raw string as is and may be expanded by placeholder described later.\n\nYou can use placeholder for label values. The placeholders will be expanded from reserved values and records.\nIf you specify `${hostname}`, it will be expanded by value of a hostname where fluentd runs.\nThe placeholder for records is deprecated. Use record accessor syntax instead.\n\nReserved placeholders are:\n\n- `${hostname}`: hostname\n- `${worker_id}`: fluent worker id\n- `${tag}`: tag name\n  - only available in Prometheus output/filter plugin\n- `${tag_parts[N]}` refers to the Nth part of the tag.\n  - only available in Prometheus output/filter plugin\n- `${tag_prefix[N]}` refers to the [0..N] part of the tag.\n  - only available in Prometheus output/filter plugin\n- `${tag_suffix[N]}` refers to the [`tagsize`-1-N..] part of the tag.\n  - where `tagsize` is the size of tag which is splitted with `.` (when tag is `1.2.3`, then `tagsize` is 3)\n  - only available in Prometheus output/filter plugin\n\n### Metric initialization\n\nYou can configure if a metric should be initialized to its zero value before receiving any event. To do so you just need to specify `initialized true`.\n\n```\n\u003cmetric\u003e\n  name message_bar_counter\n  type counter\n  desc The total number of bar in message.\n  key bar\n  initialized true\n  \u003clabels\u003e\n    foo bar\n  \u003c/labels\u003e\n\u003c/metric\u003e\n```\n\nIf your labels contains ReccordAccessors or Placeholders, you must use `\u003cinitlabels\u003e` to specify the values your ReccordAccessors/Placeholders will take. This feature is useful only if your Placeholders/ReccordAccessors contain deterministic values. Initialization will create as many zero value metrics as `\u003cinitlabels\u003e` blocks you defined.\nPotential reserved placeholders `${hostname}` and `${worker_id}`, as well as static labels, are automatically added and should not be specified in `\u003cinitlabels\u003e` configuration.\n\n```\n\u003cmetric\u003e\n  name message_bar_counter\n  type counter\n  desc The total number of bar in message.\n  key bar\n  initialized true\n  \u003clabels\u003e\n    key $.foo\n    tag ${tag}\n    foo bar\n    worker_id ${worker_id}\n  \u003c/labels\u003e\n  \u003cinitlabels\u003e\n    key foo1\n    tag tag1\n  \u003c/initlabels\u003e\n  \u003cinitlabels\u003e\n    key foo2\n    tag tag2\n  \u003c/initlabels\u003e\n\u003c/metric\u003e\n\u003clabels\u003e\n  hostname ${hostname}\n\u003c/labels\u003e\n```\n\n### top-level labels and labels inside metric\n\nPrometheus output/filter plugin can have multiple metric section. Top-level labels section specifies labels for all metrics. Labels section inside metric section specifies labels for the metric. Both are specified, labels are merged.\n\n```\n\u003cfilter message\u003e\n  @type prometheus\n  \u003cmetric\u003e\n    name message_foo_counter\n    type counter\n    desc The total number of foo in message.\n    key foo\n    \u003clabels\u003e\n      key foo\n      data_type ${type}\n    \u003c/labels\u003e\n  \u003c/metric\u003e\n  \u003cmetric\u003e\n    name message_bar_counter\n    type counter\n    desc The total number of bar in message.\n    key bar\n    \u003clabels\u003e\n      key bar\n    \u003c/labels\u003e\n  \u003c/metric\u003e\n  \u003clabels\u003e\n    tag ${tag}\n    hostname ${hostname}\n  \u003c/labels\u003e\n\u003c/filter\u003e\n```\n\nIn this case, `message_foo_counter` has `tag`, `hostname`, `key` and `data_type` labels.\n\n\n## Try plugin with nginx\n\nCheckout repository and setup.\n\n```\n$ git clone git://github.com/fluent/fluent-plugin-prometheus.git\n$ cd fluent-plugin-prometheus\n$ bundle install --path vendor/bundle\n```\n\nDownload pre-compiled Prometheus binary and start it. It listens on 9090.\n\n```\n$ wget https://github.com/prometheus/prometheus/releases/download/v1.5.2/prometheus-1.5.2.linux-amd64.tar.gz -O - | tar zxf -\n$ ./prometheus-1.5.2.linux-amd64/prometheus -config.file=./misc/prometheus.yaml -storage.local.path=./prometheus/metrics\n```\n\nInstall Nginx for sample metrics. It listens on 80 and 9999.\n\n```\n$ sudo apt-get install -y nginx\n$ sudo cp misc/nginx_proxy.conf /etc/nginx/sites-enabled/proxy\n$ sudo chmod 777 /var/log/nginx \u0026\u0026 sudo chmod +r /var/log/nginx/*.log\n$ sudo service nginx restart\n```\n\nStart fluentd with sample configuration. It listens on 24231.\n\n```\n$ bundle exec fluentd -c misc/fluentd_sample.conf -v\n```\n\nGenerate some records by accessing nginx.\n\n```\n$ curl http://localhost/\n$ curl http://localhost:9999/\n```\n\nConfirm that some metrics are exported via Fluentd.\n\n```\n$ curl http://localhost:24231/metrics\n```\n\nThen, make a graph on Prometheus UI. http://localhost:9090/\n\n## Contributing\n\n1. Fork it ( https://github.com/fluent/fluent-plugin-prometheus/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n\n## Copyright\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eAuthor\u003c/td\u003e\u003ctd\u003eMasahiro Sano \u003csabottenda@gmail.com\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eCopyright\u003c/td\u003e\u003ctd\u003eCopyright (c) 2015- Masahiro Sano\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLicense\u003c/td\u003e\u003ctd\u003eApache License, Version 2.0\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent%2Ffluent-plugin-prometheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent%2Ffluent-plugin-prometheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent%2Ffluent-plugin-prometheus/lists"}