{"id":35544389,"url":"https://github.com/zebrium/ze-fluentd-plugin","last_synced_at":"2026-01-04T06:01:04.100Z","repository":{"id":132413579,"uuid":"204803800","full_name":"zebrium/ze-fluentd-plugin","owner":"zebrium","description":"Zebrium's fluentd output plugin sends the logs you collect with fluentd to Zebrium for automated anomaly detection.","archived":false,"fork":false,"pushed_at":"2024-05-21T18:54:35.000Z","size":250,"stargazers_count":3,"open_issues_count":4,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-21T20:05:41.637Z","etag":null,"topics":["anomaly-detection","log-analytics","log-collector","log-management","structured-data"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zebrium.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-27T22:53:45.000Z","updated_at":"2024-05-21T18:54:32.000Z","dependencies_parsed_at":"2024-05-21T20:02:36.354Z","dependency_job_id":"8130c417-9e70-4e92-b0fe-f9f3bbade6df","html_url":"https://github.com/zebrium/ze-fluentd-plugin","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/zebrium/ze-fluentd-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebrium%2Fze-fluentd-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebrium%2Fze-fluentd-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebrium%2Fze-fluentd-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebrium%2Fze-fluentd-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zebrium","download_url":"https://codeload.github.com/zebrium/ze-fluentd-plugin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebrium%2Fze-fluentd-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28198653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-04T02:00:06.065Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anomaly-detection","log-analytics","log-collector","log-management","structured-data"],"created_at":"2026-01-04T06:00:18.116Z","updated_at":"2026-01-04T06:01:04.058Z","avatar_url":"https://github.com/zebrium.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LINUX COLLECTOR DETAILS\n\nZebrium's linux log collector sends the logs you collect with [fluentd](https://www.fluentd.org/) on linux to Zebrium for automated anomaly detection.\nOur github repository is located [here](https://github.com/zebrium/ze-fluentd-plugin).  Zebium's linux log collector leverages our [fluentd output plugin](https://github.com/zebrium/fluentd-output-zebrium) and is distributed leveraging fluent-package version 5.  Because of this, we are limited to the installation platforms that are supported by the fluent-package installation.  For more information, see the package [documentation](https://docs.fluentd.org/quickstart/fluent-package-v5-vs-td-agent).  Below are instructions on utilizing our installation script for installing and configuring fluent-package alongside our plugins.\n\nFor instructions on deploying our fluentd collector for [docker](https://github.com/zebrium/fluentd-output-zebrium) environments, please see docker setup [here](https://docs.zebrium.com/docs/setup/docker)\n\n## Installation and Configuration\n\nBelow is the installation and configuration instructions for the zebrium log collector.  The default configuration will collect any logs inside of `/var/log/*.log` and send them to Zebrium.  For advanced configuration and usages, please see the relevant section [here](#advanced-configurations).\n\n### System Requirements\n\nThe following OS distributions are supported:\n\nDEB Packages:\n\n1. Ubuntu: Jammy, Focal, Bionic, Xenial\n2. Debian: Bullseye, Buster\n\nRPM Package:\n\n1. CentOS/RHEL 7/8/9\n2. Oracle Linux 7/8/9\n3. Amazon Linux 2\n\n### Installing\n\n1. If the environment uses a proxy server please the section \"Operation with a Proxy Server\" below.\n2. Get Zebrium API server URL and authentication token from [Zebrium](https://www.zebrium.com).\n3. Determine what deployment name to use.\n4. Run the following command in a shell on host:\n\n   ``` bash\n   curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=\u003cZAPI_URL\u003e ZE_LOG_COLLECTOR_TOKEN=\u003cAUTH_TOKEN\u003e ZE_HOST_TAGS=\"ze_deployment_name=\u003cdeployment_name\u003e\" /bin/bash\n   ```\n\n   The default system log file paths are defined by the ZE_LOG_PATHS environment variable. Its default value is\n\n   ``` bash\n   \"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure\"\n   ```\n\n   The ZE_USER_LOG_PATHS environment variable can be used to add more user specific log file paths. For example, to add app log files at `/app1/log/app1.log` and `/app2/log/\\*.log`, you can set ZE_USER_LOG_PATHS to:\n\n   ``` bash\n   \"/app1/log/app1.log,/app2/log/*.log\"\n   ```\n\n### Upgrading\n\nThe upgrade command is similar to the installation command:\n\n``` bash\ncurl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=\u003cZAPI_URL\u003e ZE_LOG_COLLECTOR_TOKEN=\u003cAUTH_TOKEN\u003e ZE_HOST_TAGS=\"ze_deployment_name=\u003cdeployment_name\u003e\" OVERWRITE_CONFIG=1 /bin/bash\n```\n\nPlease note setting `OVERWRITE_CONFIG` to 1 will cause `/etc/fluent/fluentd.conf` to be upgraded to latest version.\n\n### Uninstalling\n\n``` bash\ncurl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_OP=uninstall /bin/bash\n```\n\n### Installing on Hosts with Existing fluent Configuration\n\nIt is possible to add Zebrium output plugin on a host with existing fluent configuration without running Zebrium log collector installer.  Zebrium output plugin is provided through RubyGems [here](https://rubygems.org/gems/fluent-plugin-zebrium_output)\n\n1. Run the following command:\n\n   ``` bash\n   sudo fluent-gem install fluent-plugin-zebrium_output\n   ```\n\n3. Add Zebrium output configuration to `/etc/fluent/fluent.conf`. Below is an example configuration which duplicates log messages and sends one copy to Zebrium.\n\n   ``` bash\n   \u003cmatch **\u003e\n     @type copy\n     # Zebrium log collector\n     \u003cstore\u003e\n       @type zebrium\n       ze_log_collector_url \"ZE_LOG_COLLECTOR_URL\"\n       ze_log_collector_token \"ZE_LOG_COLLECTOR_TOKEN\"\n       ze_host_tags \"ze_deployment_name=#{Socket.gethostname},myapp=test2\"\n       @log_level \"info\"\n       \u003cbuffer tag\u003e\n         @type file\n         path \"/var/fluent/zebrium\"\n         flush_mode \"interval\"\n         flush_interval \"60s\"\n       \u003c/buffer\u003e\n     \u003c/store\u003e\n     \u003cstore\u003e\n         @type OTHER_OUTPUT_PLUGIN\n         ...\n     \u003c/store\u003e\n   \u003c/match\u003e\n   ```\n\n### Configurations\n\nThere are several configurations options for the log collector. The configuration file for fluent-package is at `/etc/fluent/fluentd.conf`.\n\n#### Parameters\n\nThe following parameters must be configured for your instance:\n| Parameter | Description | Note |\n| ----------| ----------| ----------|\n| ze_log_collector_url | Zebrium log host URL | Provided by Zebrium once your account has been created.|\n| ze_log_collector_token | Authentication token | Provided by Zebrium once your account has been created.|\n| path | Log files to read | Both files and file patterns are allowed. Files should be separated by comma. The default value is `\"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure\"`|\n| ze_host_tags|Host meta data | This parameter is optional. You can pass meta data in key-value pairs, the format is: \"key1=value1,key2=value2\". We suggest at least set one tag for deployment name: \"ze_deployment_name=\u0026lt;your_deployment_name\u0026gt;\"|\n| ze_host_in_logpath | Log path component for remote host name | This parameter is optional. For situations where a remote host name is embedded in the log file directory path structure, e.g. \"/var/log/remote/\u0026lt;host\u0026gt;/...\", this can be used as the originating host for the log by setting this parameter to the path component to be used for the hostname. The value should be an integer, 1-based. In this example the configuration would be \"ze_host_in_logpath=4\".|\n| ze_forward_tag | Tag to specify log-forwarded sources | This parameter is optional. It can be used to indicate sources that are being used for remote log forwarding, by specifying a specific fluentd \"tag\" to one or more sources.  The default tag value is \"ze_forwarded_logs\".|\n| ze_path_map_file| Path mapping file | This parameter is optional. It allows embedded semantic data (ids, tags,configs) in logfile paths to be parsed and added to Zebrium log data. Set to the full path of a JSON file containing mapping information. Default is empty string. See below under Log Path Mapping|\n\n#### Advanced Configurations\n\n##### User Log Paths\n\nIn some configurations, you may wish to create a dynamic configuration from a file to be loaded on the start of fluent-package.  This can be accomplished by creating a log-file-map.conf json file as seen below.  This file will need to reside in the following location `/etc/fluent/log-file-map.conf`. During log collector service startup, if `/etc/fluent/log-file-map.conf` exists, log collector service script writes log paths defined in `/etc/fluent/log-file-map.conf` to `/etc/fluent/conf.d/user.conf`.\n\nPlease note any user log paths configured at installation time via ZE_USER_LOG_PATHS must be added to `/etc/fluent/log-file-map.conf` to avoid being overwritten.\n\n``` json\n{\n  \"mappings\": [\n    {\n      \"file\": \"/app1/log/error.log\",\n      \"alias\": \"app1_error\"\n    },\n    {\n      \"file\": \"/app2/log/error.log\",\n      \"alias\": \"app2_error\"\n    },\n    {\n      \"file\": \"/var/log/*.log\",\n      \"exclude\": \"/var/log/my_debug.log,/var/log/my_test.log\"\n    }\n  ]\n}\n```\n\n##### Filtering Specific Log Events\n\nIf you wish to exclude certain sensitive or noisy events from being sent to Zebrium, you can filter them at the source collection point by doing the following:\n\n1. Add the following in /etc/fluent/fluentd.conf after other \"@include\":\n\n   ``` configuration\n   @include conf.d/log_msg_filters.conf\n   ```\n\n2. Create a config file /etc/fluent/conf.d/log_msg_filters.conf containing:\n\n   ``` configuration\n   \u003cfilter TAG_FOR_LOG_FILE\u003e\n     @type grep\n     \u003cexclude\u003e\n       key message\n       pattern /\u003cPATTERN_FOR_LOG_MESSAGES\u003e/\n   \u003c/exclude\u003e\n   \u003c/filter\u003e\n   ```\n\n3. Restart fluentd: sudo systemctl restart fluentd\n\n###### Example\n\nBelow is an example `log_msg_filters.conf` for filtering out specific messages from a Vertica log file at `/fast1/vertica_catalog/zdb/v_zdb_node0001_catalog/vertica.log`\n\nIn this example, the Fluentd tag for file is node.logs.\u003cFILE_NAME_REPLACE_/_WITH_DOT\u003e (i.e replace all slashes with dots in the file path).\n\n``` configuration\n\u003cfilter node.logs.fast1.vertica_catalog.zdb.v_zdb_node0001_catalog.vertica.log\u003e\n  @type grep\n  \u003cexclude\u003e\n    key message\n    pattern /^[^2]|^.[^0]|TM Merge|Authenticat|[Ll]oad *[Bb]alanc[ei]|\\[Session\\] \u003cINFO\u003e|\\[Catalog\\] \u003cINFO\u003e|\\[Txn\\] \u003cINFO\u003e|Init Session.*\u003cLOG\u003e/\n  \u003c/exclude\u003e\n\u003c/filter\u003e\n```\n\n##### Log Path Mapping\n\nLog path mapping allows semantic information (ids, configs and tags) to be extracted from log paths\nand passed to the Zebrium backend. For example, log-file specific host information or business-related\ntags that are embedded in the path of the log file can be extracted..\n\nLog path mapping is configured using a JSON file, with format:\n\n``` json\n{\n  \"mappings\": {\n    \"patterns\": [\n      \"regex1\", ...\n    ],\n    \"tags\": [\n      \"tag_name\", ...\n    ],\n    \"ids\": [\n      \"id_name\",...\n    ],\n    \"configs\": [\n       \"config_name\",...\n    ]\n  }\n}\n```\n##### Configuring Multiple Zebrium Service Groups Within a Single Collector\n\nIt is possible to use a single fluentd installation to send log files to multiple Zebrium service groups. Knowlege about advanced fluentd configuration is required. It is recommended to review the official documentation at https://docs.fluentd.org/configuration/config-file \n\nThe following are required:\n- each service group needs to have its own source block and match block defenitions\n- in each source block, the path should be as specific as possible\n- paths in source blocks should not overlap\n- each source block needs a unique pos_file (fluentd will create the file if it does not exist)\n- each source block should include a unique tag to specify which match block/service group should pick up the log events\n- each match block should match on the tag in its corresponding source block\n- ze_log_collector_url, ze_log_collector_token, and ze_log_collector_type will probably be the same in all match blocks\n- ze_host_tags specifies the service group name with \"ze_deployment_name=\u003cservice group name\u003e\"\n- each match block requires a unique buffer path, which will be created if the specified path does not exist\n\nHere's an example of how this could be configured in /etc/fluent/fluentd.conf:\n```\n\u003csource\u003e\n  @type tail\n  path \"/var/log/auth.log\"\n  format none\n  path_key tailed_path\n  pos_file /var/log/fluent/position_file_1.pos\n  tag seamus1\n  read_from_head true\n\u003c/source\u003e\n\n\u003csource\u003e\n  @type tail\n  path \"/var/log/syslog\"\n  format none\n  path_key tailed_path\n  pos_file /var/log/fluent/position_file_2.pos\n  tag seamus2\n  read_from_head true\n\u003c/source\u003e\n\n@include conf.d/user.conf\n@include conf.d/containers.conf\n@include conf.d/systemd.conf\n\n\u003cmatch seamus1\u003e\n  @type zebrium\n  ze_log_collector_url \"https://trial.zebrium.com\"\n  ze_log_collector_token \"\u003cyour token here\u003e\"\n  ze_log_collector_type \"linux\"\n  ze_host_tags \"ze_deployment_name=seamusfirstservicegroup\"\n  \u003cbuffer tag\u003e\n    @type file\n    path /var/log/fluent/buffer1/out_zebrium.*.buffer\n    chunk_limit_size \"1MB\"\n    chunk_limit_records \"4096\"\n    flush_mode \"interval\"\n    flush_interval \"60s\"\n  \u003c/buffer\u003e\n\u003c/match\u003e\n\n\u003cmatch seamus2\u003e\n  @type zebrium\n  ze_log_collector_url \"https://trial.zebrium.com\"\n  ze_log_collector_token \"\u003cyour token here, should be the same as above\u003e\"\n  ze_log_collector_type \"linux\"\n  ze_host_tags \"ze_deployment_name=seamussecondservicegroup\"\n  \u003cbuffer tag\u003e\n    @type file\n    path /var/log/fluent/buffer2/out_zebrium.*.buffer\n    chunk_limit_size \"1MB\"\n    chunk_limit_records \"4096\"\n    flush_mode \"interval\"\n    flush_interval \"60s\"\n  \u003c/buffer\u003e\n\u003c/match\u003e\n```\n\n\n\nSet \"patterns\" to regular expressions to match the log file path. Each regex named capture in a matching regular expression will be compared to the \"tags\", \"ids\" and \"configs\" sections and added to the corresponding record section(s).\nUse the ze_path_map_file configuration parameter to specify the path to the JSON file.\n\n##### Proxy Configuration\n\nIf the agent environment requires a non-transparent proxy server to be configured this should be done at two points:\n\n* The standard http\\_proxy and https\\_proxy environment variables must be set in the local environment when the installer is run. This allows the installer to access the Internet to download necessary components.\n\n* After installation is run the system service also needs to have the same environment variables available. This allows the Zebrium agent to communicate with the log host to send logs.\n\n###### Setting proxy server in a systemd environment\n\nIf the agent service is run from systemd and a proxy server is in use, the service needs to have the appropriate proxy configuration added to systemd. (This may not be needed if your system is already configured so that all systemd services globally use a proxy.) To do this, after the installation is performed edit the file /etc/systemd/system/fluentd.service.d/override.conf to add environment configuration lines for the proxy server, for example:\n\n``` bash\nEnvironment=http_proxy=myproxy.example.com:8080\n```\n\nAfter this is done the systemd daemon should be reloaded, and then the service started:\n\n``` bash\nsudo systemctl daemon-reload\n\nsudo systemctl restart fluentd\n```\n\n## Usage\n\n### Start/stop Fluentd\n\nFluentd agent can be started or stopped with the command:\n\n``` bash\nsudo systemctl \u003cstart | stop\u003e fluentd\n```\n\n## Testing your installation\n\nOnce the collector has been deployed in your environment, your logs and anomaly detection will be available in the Zebrium UI.\n\n## Troubleshooting\n\nIn the event that Zebrium requires the collector logs for troubleshooting, logs are located here:\n\n1. Collector installation log:  `/tmp/zlog-collector-install.log.*`\n\n2. Collector runtime log: `/var/log/fluent/fluentd.log`\n\nIn case of an HTTP connection error, please check the spelling of the Zebrium host URL. Also check that any network proxy servers are configured appropriately.\n\nPlease contact Zebrium Support at \u003csupport@zebrium.com\u003e if you need any assistance.  When reaching out to support, please be sure to include the following information\nand files to more efficiently resolve your issue.\n\n1. Description of the problem and relevant environment specific information\n2. Collector installation log:  `/tmp/zlog-collector-install.log.*`\n3. Collector runtime log: `/var/log/fluent/fluentd.log`\n4. Collector configurations: `/etc/fluent/fluentd.conf`, `/etc/fluent/conf.d/*.conf`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzebrium%2Fze-fluentd-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzebrium%2Fze-fluentd-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzebrium%2Fze-fluentd-plugin/lists"}