{"id":21056959,"url":"https://github.com/fluent-plugins-nursery/fluent-plugin-systemd","last_synced_at":"2025-04-04T22:07:51.359Z","repository":{"id":50868242,"uuid":"45683446","full_name":"fluent-plugins-nursery/fluent-plugin-systemd","owner":"fluent-plugins-nursery","description":"This is a fluentd input plugin. It reads logs from the systemd journal.","archived":false,"fork":false,"pushed_at":"2025-02-28T01:59:39.000Z","size":320,"stargazers_count":155,"open_issues_count":17,"forks_count":42,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-28T21:06:32.610Z","etag":null,"topics":["fluentd","journal","libsystemd","logging","systemd","systemd-journal"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent-plugins-nursery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-11-06T13:25:23.000Z","updated_at":"2025-02-28T01:55:42.000Z","dependencies_parsed_at":"2024-02-21T14:29:58.826Z","dependency_job_id":"422db826-f2a8-42e1-b4d8-29a776ee187b","html_url":"https://github.com/fluent-plugins-nursery/fluent-plugin-systemd","commit_stats":null,"previous_names":["reevoo/fluent-plugin-systemd","fluent-plugin-systemd/fluent-plugin-systemd"],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-systemd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-systemd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-systemd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-systemd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent-plugins-nursery","download_url":"https://codeload.github.com/fluent-plugins-nursery/fluent-plugin-systemd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247256112,"owners_count":20909240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","journal","libsystemd","logging","systemd","systemd-journal"],"created_at":"2024-11-19T16:55:15.748Z","updated_at":"2025-04-04T22:07:51.326Z","avatar_url":"https://github.com/fluent-plugins-nursery.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# systemd plugin for [Fluentd](http://github.com/fluent/fluentd)\n\n[![Build Status](https://github.com/fluent-plugin-systemd/fluent-plugin-systemd/actions/workflows/ci.yml/badge.svg)](https://github.com/fluent-plugin-systemd/fluent-plugin-systemd/actions/workflows/ci.yml) [![Maintainability](https://api.codeclimate.com/v1/badges/5c773a4c1c6a6964fa4b/maintainability)](https://codeclimate.com/github/fluent-plugin-systemd/fluent-plugin-systemd/maintainability) [![Gem Version](https://badge.fury.io/rb/fluent-plugin-systemd.svg)](https://rubygems.org/gems/fluent-plugin-systemd)\n\n## Overview\n\n* **systemd** input plugin to read logs from the systemd journal\n* **systemd** filter plugin for basic manipulation of systemd journal entries\n\n## Support\n\nJoin the #plugin-systemd channel on the [Fluentd Slack](http://slack.fluentd.org/)\n\n## Requirements\n\n|fluent-plugin-systemd|fluentd|td-agent|ruby|\n|----|----|----|----|\n| \u003e 0.1.0 | \u003e= 0.14.11, \u003c 2 | 3 | \u003e= 2.1 |\n| 0.0.x | ~\u003e 0.12.0       | 2 | \u003e= 1.9  |\n\n|fluent-plugin-systemd|fluentd|fluent-package|ruby|\n|----|----|----|----|\n| \u003e= 1.1.0 | \u003e= 0.14.11, \u003c 2 | 5 | \u003e= 3.0 |\n\n* The 1.x.x series is developed from this branch (master)\n* The 0.0.x series (compatible with fluentd v0.12, and td-agent 2) is maintained on the [0.0.x branch](https://github.com/reevoo/fluent-plugin-systemd/tree/0.0.x)\n\n## Installation\n\nSimply use RubyGems:\n\n    gem install fluent-plugin-systemd -v 1.0.3\n\nor\n\n    td-agent-gem install fluent-plugin-systemd -v 1.0.3\n\n## Upgrading\n\nIf you are upgrading to version 1.0 from a previous version of this plugin take a look at the [upgrade documentation](docs/upgrading.md). A number of deprecated config options were removed so you might need to update your configuration.\n\n## Input Plugin Configuration\n\n    \u003csource\u003e\n      @type systemd\n      tag kubelet\n      path /var/log/journal\n      matches [{ \"_SYSTEMD_UNIT\": \"kubelet.service\" }]\n      read_from_head true\n\n      \u003cstorage\u003e\n        @type local\n        path /var/log/fluentd-journald-kubelet-cursor.json\n      \u003c/storage\u003e\n\n      \u003centry\u003e\n        fields_strip_underscores true\n        fields_lowercase true\n      \u003c/entry\u003e\n    \u003c/source\u003e\n\n    \u003cmatch kubelet\u003e\n      @type stdout\n    \u003c/match\u003e\n\n    \u003csystem\u003e\n      root_dir /var/log/fluentd\n    \u003c/system\u003e\n\n**`path`**\n\nPath to the systemd journal, defaults to `/var/log/journal`\n\n**`filters`**\n\n_This parameter name is depreciated and should be renamed to `matches`_\n\n**`matches`**\n\nExpects an array of hashes defining desired matches to filter the log\nmessages with. When this property is not specified, this plugin will default to\nreading all logs from the journal.\n\nSee [matching details](docs/matching.md) for a more exhaustive\ndescription of this property and how to use it.\n\n**`storage`**\n\nConfiguration for a [storage plugin](https://docs.fluentd.org/storage) used to store the journald cursor.\n\n**`read_from_head`**\n\nIf true reads all available journal from head, otherwise starts reading from tail,\n ignored if cursor exists in storage (and is valid). Defaults to false.\n\n**`entry`**\n\nOptional configuration for an embedded systemd entry filter. See the  [Filter Plugin Configuration](#filter-plugin-configuration) for config reference.\n\n**`tag`**\n\n_Required_\n\nA tag that will be added to events generated by this input.\n\n\n## Filter Plugin Configuration\n\n```\n\u003cfilter kube-proxy\u003e\n  @type systemd_entry\n  field_map {\"MESSAGE\": \"log\", \"_PID\": [\"process\", \"pid\"], \"_CMDLINE\": \"process\", \"_COMM\": \"cmd\"}\n  field_map_strict false\n  fields_lowercase true\n  fields_strip_underscores true\n\u003c/filter\u003e\n```\n\n_Note that the following configurations can be embedded in a systemd source block, within an entry block, you only need to use a filter directly for more complicated workflows._\n\n**`field_map`**\n\nObject / hash defining a mapping of source fields to destination fields. Destination fields may be existing or new user-defined fields. If multiple source fields are mapped to the same destination field, the contents of the fields will be appended to the destination field in the order defined in the mapping. A field map declaration takes the form of:\n\n    {\n      \"\u003csrc_field1\u003e\": \"\u003cdst_field1\u003e\",\n      \"\u003csrc_field2\u003e\": [\"\u003cdst_field1\u003e\", \"\u003cdst_field2\u003e\"],\n      ...\n    }\nDefaults to an empty map.\n\n**`field_map_strict`**\n\nIf true, only destination fields from `field_map` are included in the result. Defaults to false.\n\n**`fields_lowercase`**\n\nIf true, lowercase all non-mapped fields. Defaults to false.\n\n**`fields_strip_underscores`**\n\nIf true, strip leading underscores from all non-mapped fields. Defaults to false.\n\n### Filter Example\n\nGiven a systemd journal source entry:\n```\n{\n  \"_MACHINE_ID\": \"bb9d0a52a41243829ecd729b40ac0bce\"\n  \"_HOSTNAME\": \"arch\"\n  \"MESSAGE\": \"this is a log message\",\n  \"_PID\": \"123\"\n  \"_CMDLINE\": \"login -- root\"\n  \"_COMM\": \"login\"\n}\n```\nThe resulting entry using the above sample configuration:\n```\n{\n  \"machine_id\": \"bb9d0a52a41243829ecd729b40ac0bce\"\n  \"hostname\": \"arch\",\n  \"msg\": \"this is a log message\",\n  \"pid\": \"123\"\n  \"cmd\": \"login\"\n  \"process\": \"123 login -- root\"\n}\n```\n\n## Common Issues\n\n\u003e ### When I look at fluentd logs, everything looks fine but no journal logs are read ?\n\nThis is commonly caused when the user running fluentd does not have the correct permissions\nto read the systemd journal.\n\nAccording to the [systemd documentation](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html):\n\u003e Journal files are, by default, owned and readable by the \"systemd-journal\" system group but are not writable. Adding a user to this group thus enables her/him to read the journal files.\n\n\u003e ### How can I deal with multi-line logs ?\n\nIdeally you want to ensure that your logs are saved to the systemd journal as a single entry regardless of how many lines they span.\n\nIt is possible for applications to naively support this (but only if they have tight integration with systemd it seems) see: https://github.com/systemd/systemd/issues/5188.\n\nTypically you would not be able to this, so another way is to configure your logger to replace newline characters with something else. See this blog post for an example configuring a Java logging library to do this https://fabianlee.org/2018/03/09/java-collapsing-multiline-stack-traces-into-a-single-log-event-using-spring-backed-by-logback-or-log4j2/\n\nAnother strategy would be to use a plugin like [fluent-plugin-concat](https://github.com/fluent-plugins-nursery/fluent-plugin-concat) to combine multi line logs into a single event, this is more tricky though because you need to be able to identify the first and last lines of a multi line message with a regex.\n\n\u003e ### How can I use this plugin inside of a docker container ?\n\n* Install the [systemd dependencies](#dependencies) if required\n* You can use an [offical fluentd docker](https://github.com/fluent/fluentd-docker-image) image as a base, (choose the debian based version, as alpine linux doesn't support systemd).\n* Bind mount `/var/log/journal` into your container.\n\n\u003e ### I am seeing lots of logs being generated very rapidly!\n\nThis commonly occurs when a loop is created when fluentd is logging to STDOUT, and the collected logs are then written to the systemd journal. This could happen if you run fluentd as a systemd serivce, or as a docker container with the systemd log driver.\n\nWorkarounds include:\n\n* Use another fluentd output\n* Don't read every message from the journal, set some `matches` so you only read the messages you are interested in.\n* Disable the systemd log driver when you launch your fluentd docker container, e.g. by passing `--log-driver json-file`\n\n### Example\n\nFor an example of a full working setup including the plugin, take a look at [the fluentd kubernetes daemonset](https://github.com/fluent/fluentd-kubernetes-daemonset)\n\n## Dependencies\n\nThis plugin depends on libsystemd\n\nOn Debian or Ubuntu you might need to install the libsystemd0 package:\n\n```\napt-get install libsystemd0\n```\n\nOn AlmaLinux or RHEL you might need to install the systemd package:\n\n```\nyum install -y systemd\n```\n\nIf you want to do this in a AlmaLinux docker image you might first need to remove the `fakesystemd` package.\n\n```\nyum remove -y fakesystemd\n```\n\n## Running the tests\n\nTo run the tests with docker on several distros simply run `rake`\n\nFor systems with systemd installed you can run the tests against your installed libsystemd with `rake test`\n\n## License\n\n[Apache-2.0](LICENCE)\n\n## Contributions\n\nIssues and pull requests are very welcome.\n\nIf you want to make a contribution but need some help or advice feel free to message me @errm on the [Fluentd Slack](http://slack.fluentd.org/), or send me an email edward-robinson@cookpad.com\n\nWe have adopted the [Contributor Covenant](CODE_OF_CONDUCT.md) and thus expect anyone interacting with contributors, maintainers and users of this project to abide by it.\n\n## Maintainer\n\n* [Ed Robinson](https://github.com/errm)\n\n## Contributors\n\nMany thanks to our fantastic contributors\n\n* [Erik Maciejewski](https://github.com/emacski)\n* [Ernie Hershey](https://github.com/ehershey)\n* [Hiroshi Hatake](https://github.com/cosmo0920)\n* [Jesus Rafael Carrillo](https://github.com/jescarri)\n* [Joel Gerber](https://github.com/Jitsusama)\n* [John Thomas Wile II](https://github.com/jtwile2)\n* [Kazuhiro Suzuki](https://github.com/ksauzz)\n* [Marius Grigoriu](https://github.com/mariusgrigoriu)\n* [Masahiro Nakagawa](https://github.com/repeatedly)\n* [Mike Kaplinskiy](https://github.com/mikekap)\n* [Richard Megginson](https://github.com/richm)\n* [Sadayuki Furuhashi](https://github.com/frsyuki)\n* [Seiichi Nishikata](https://github.com/neko-neko)\n* [Kentaro Hayashi](https://github.com/kenhys)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-systemd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-systemd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-systemd/lists"}