{"id":21056951,"url":"https://github.com/fluent-plugins-nursery/fluent-plugin-concat","last_synced_at":"2025-10-04T07:39:42.233Z","repository":{"id":51069143,"uuid":"55645054","full_name":"fluent-plugins-nursery/fluent-plugin-concat","owner":"fluent-plugins-nursery","description":"Fluentd Filter plugin to concatenate multiline log separated in multiple events.","archived":false,"fork":false,"pushed_at":"2025-06-12T07:30:09.000Z","size":163,"stargazers_count":108,"open_issues_count":32,"forks_count":34,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-09-21T16:36:29.350Z","etag":null,"topics":["fluentd","fluentd-plugin"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent-plugins-nursery.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-04-06T22:16:34.000Z","updated_at":"2025-06-12T07:28:19.000Z","dependencies_parsed_at":"2024-06-18T15:19:45.415Z","dependency_job_id":"1e826c2f-6f17-48fe-9f4a-647b07435356","html_url":"https://github.com/fluent-plugins-nursery/fluent-plugin-concat","commit_stats":{"total_commits":163,"total_committers":20,"mean_commits":8.15,"dds":0.4049079754601227,"last_synced_commit":"bf2a1f6b70e7e54bfd669237050fd58fb7f8564f"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/fluent-plugins-nursery/fluent-plugin-concat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-concat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-concat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-concat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-concat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent-plugins-nursery","download_url":"https://codeload.github.com/fluent-plugins-nursery/fluent-plugin-concat/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-concat/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278283487,"owners_count":25961309,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","fluentd-plugin"],"created_at":"2024-11-19T16:55:14.001Z","updated_at":"2025-10-04T07:39:42.225Z","avatar_url":"https://github.com/fluent-plugins-nursery.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-concat\n\n[![Build Status](https://travis-ci.org/fluent-plugins-nursery/fluent-plugin-concat.svg?branch=master)](https://travis-ci.org/fluent-plugins-nursery/fluent-plugin-concat)\n\nFluentd Filter plugin to concatenate multiline log separated in multiple events.\n\n## Requirements\n\n| fluent-plugin-concat | fluentd    | ruby   |\n|----------------------|------------|--------|\n| \u003e= 2.0.0             | \u003e= v0.14.0 | \u003e= 2.1 |\n| \u003c 2.0.0              | \u003e= v0.12.0 | \u003e= 1.9 |\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'fluent-plugin-concat'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install fluent-plugin-concat\n\n## Plugin helpers\n\n* [timer](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-timer)\n* [event_emitter](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-event_emitter)\n\n* See also: [Filter Plugin Overview](https://docs.fluentd.org/filter#overview)\n\n## Configuration\n\n### Example\n\n```\n\u003cfilter docker.log\u003e\n  @type concat\n  key loga\n  #separator \"\\n\"\n  n_lines 10\n  #multiline_start_regexp /^Start/\n  #multiline_end_regexp /^End/\n  #continuous_line_regexp nil\n  #stream_identity_key nil\n  #flush_interval 60\n  #timeout_label nil\n  #use_first_timestamp false\n  #partial_key nil\n  #partial_value nil\n  #keep_partial_key false\n  #use_partial_metadata false\n  #keep_partial_metadata false\n  #partial\\_metadata\\_format docker-fluentd\n  #use\\_partial\\_cri\\_logtag false\n  #partial\\_cri\\_logtag\\_key nil\n  #partial\\_cri\\_stream\\_key stream\n\u003c/filter\u003e\n```\n\n### Parameter\n\n|parameter|description|default|\n|---|---|---|\n|key|The key for part of multiline log||\n|separator|The separator of lines|`\"\\n\"`|\n|n\\_lines|The number of lines. This is exclusive with `multiline_start_regex`|nil|\n|multiline\\_start\\_regexp|The regexp to match beginning of multiline. This is exclusive with `n_lines`|nil|\n|multiline\\_end\\_regexp|The regexp to match ending of multiline.This is exclusive with `n_lines`|nil|\n|continuous\\_line\\_regexp|The regexp to match continuous lines.This is exclusive with `n_lines`|nil|\n|stream\\_identity\\_key|The key to determine which stream an event belongs to|nil|\n|flush\\_interval|The number of seconds after which the last received event log will be flushed.If specified 0, wait for next line forever|60|\n|timeout\\_label|The label name to handle events caused by timeout|nil|\n|use\\_first\\_timestamp|Use timestamp of first record when buffer is flushed|`false`|\n|partial\\_key|The field name that is the reference to concatenate records|nil|\n|partial\\_value|The value stored in the field specified by partial_key that represent partial log|nil|\n|keep\\_partial\\_key|If true, keep partial_key in concatenated records|`false`|\n|use\\_partial\\_metadata|Use partial metadata to concatenate multiple records|`false`|\n|keep\\_partial\\_metadata|If true, keep partial metadata|`false`|\n|partial\\_metadata\\_format|Input format of the partial metadata (fluentd or journald docker log driver) ( `docker-fluentd`, `docker-journald`, `docker-journald-lowercase`)\u003cbr\u003eConfigure based on the input plugin, that is used. \u003cbr\u003eThe docker fluentd and journald log drivers are behaving differently, so the plugin needs to know, what to look for.\u003cbr\u003eUse `docker-journald-lowercase`, if you have `fields_lowercase true` in the `journald` source config |`docker-fluentd`|\n|use\\_partial\\_cri\\_logtag|bool (optional)|Use cri log tag to concatenate multiple records||\n|partial\\_cri\\_logtag\\_key|string (optional)|The key name that is referred to concatenate records on cri log||\n|partial\\_cri\\_stream\\_key|string (optional)|The key name that is referred to detect stream name on cri log|`stream`|\n|buffer\\_limit\\_size|The max size of each buffer|`512000`|\n\n|parameter|description|available values|default|\n|---|---|---|---|\n|buffer\\_overflow\\_method|The method if overflow buffer|`ignore`, `truncate`, `drop`, `new`|`ignore`|\n\n* `ignore`\n  * Concatenate the current record in the buffer\n* `truncate`\n  * Drop the current record and flush the buffer\n* `drop`\n  * Drop the current record and clear the buffer\n* `new`\n  * Flush the buffer and store the current record in next buffer\n\n## Usage\n\nEvery 10 events will be concatenated into one event.\n\n```aconf\n\u003cfilter docker.log\u003e\n  @type concat\n  key message\n  n_lines 10\n\u003c/filter\u003e\n```\n\nSpecify first line of multiline by regular expression.\n\n```aconf\n\u003cfilter docker.log\u003e\n  @type concat\n  key message\n  multiline_start_regexp /^Start/\n\u003c/filter\u003e\n```\n\nYou can handle timeout events and remaining buffers on shutdown this plugin.\n\n```aconf\n\u003clabel @ERROR\u003e\n  \u003cmatch docker.log\u003e\n    @type file\n    path /path/to/error.log\n  \u003c/match\u003e\n\u003c/label\u003e\n```\n\nHandle timeout log lines the same as normal logs.\n\n```aconf\n\u003cfilter **\u003e\n  @type concat\n  key message\n  multiline_start_regexp /^Start/\n  flush_interval 5\n  timeout_label @NORMAL\n\u003c/filter\u003e\n\n\u003cmatch **\u003e\n  @type relabel\n  @label @NORMAL\n\u003c/match\u003e\n\n\u003clabel @NORMAL\u003e\n  \u003cmatch **\u003e\n    @type stdout\n  \u003c/match\u003e\n\u003c/label\u003e\n```\n\nHandle single line JSON from Docker containers.\n\n```aconf\n\u003cfilter **\u003e\n  @type concat\n  key message\n  multiline_end_regexp /\\n$/\n\u003c/filter\u003e\n```\n\nHandle Docker logs splitted in several parts (using `partial_message`), and do not add new line between parts.\n\n```aconf\n\u003cfilter\u003e\n  @type concat\n  key log\n  partial_key partial_message\n  partial_value true\n  separator \"\"\n\u003c/filter\u003e\n```\n\n(Docker v19.03+) Handle Docker logs splitted in several parts (using `use_partial_metadata`), and do not add new line between parts.\n\n```aconf\n\u003cfilter\u003e\n  @type concat\n  key log\n  use_partial_metadata true\n  separator \"\"\n\u003c/filter\u003e\n```\n\n(Docker v20.10+) Handle Docker logs splitted in several parts (using `use_partial_metadata`), and do not add new line between parts.\n\nDocker v20.10 improved partial message handling by adding better metadata in the journald log driver, this works now similarily to the fluentd log driver, but requires one additional setting\n\n```aconf\n\u003cfilter\u003e\n  @type concat\n  key log\n  use_partial_metadata true\n  partial_metadata_format docker-journald\n  separator \"\"\n\u003c/filter\u003e\n```\n\nHandle Docker logs splitted in several parts (using newline detection), and do not add new line between parts (prior to Docker 18.06).\n\n```aconf\n\u003cfilter **\u003e\n  @type concat\n  key log\n  multiline_end_regexp /\\\\n$/\n  separator \"\"\n\u003c/filter\u003e\n```\n\nHandle containerd/cri in Kubernetes.\n\n```aconf\n\u003csource\u003e\n  @type tail\n  path /var/log/containers/*.log\n  \u003cparse\u003e\n    @type regexp\n    expression /^(?\u003ctime\u003e[^ ]+) (?\u003cstream\u003estdout|stderr) (?\u003clogtag\u003e[^ ]*) (?\u003cmessage\u003e.*)$/\n    time_format %Y-%m-%dT%H:%M:%S.%L%z\n  \u003c/parse\u003e\n  tag k8s\n  @label @CONCAT\n\u003c/source\u003e\n\n\u003clabel @CONCAT\u003e\n  \u003cfilter k8s\u003e\n    @type concat\n    key message\n    use_partial_cri_logtag true\n    partial_cri_logtag_key logtag\n    partial_cri_stream_key stream\n  \u003c/filter\u003e\n  \u003cmatch k8s\u003e\n    @type relabel\n    @label @OUTPUT\n  \u003c/match\u003e\n\u003c/label\u003e\n\n\u003clabel @OUTPUT\u003e\n  \u003cmatch\u003e\n    @type stdout\n  \u003c/match\u003e\n\u003c/label\u003e\n```\n\n## Contributing\n\n1. Fork it\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create new Pull Request\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-concat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-concat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-concat/lists"}