{"id":18143872,"url":"https://github.com/magnusbaeck/logstash-filter-verifier","last_synced_at":"2025-04-09T21:12:57.539Z","repository":{"id":38360645,"uuid":"46632418","full_name":"magnusbaeck/logstash-filter-verifier","owner":"magnusbaeck","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-13T00:29:47.000Z","size":721,"stargazers_count":197,"open_issues_count":42,"forks_count":27,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-09T21:12:46.918Z","etag":null,"topics":["hacktoberfest","logstash","testing-tools"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/magnusbaeck.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-11-21T20:25:32.000Z","updated_at":"2025-01-21T20:35:24.000Z","dependencies_parsed_at":"2024-05-31T06:28:43.613Z","dependency_job_id":"ad3ae235-48e0-4ae7-87ef-55e3e2e4d976","html_url":"https://github.com/magnusbaeck/logstash-filter-verifier","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnusbaeck%2Flogstash-filter-verifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnusbaeck%2Flogstash-filter-verifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnusbaeck%2Flogstash-filter-verifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnusbaeck%2Flogstash-filter-verifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/magnusbaeck","download_url":"https://codeload.github.com/magnusbaeck/logstash-filter-verifier/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248111973,"owners_count":21049578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","logstash","testing-tools"],"created_at":"2024-11-01T19:09:05.648Z","updated_at":"2025-04-09T21:12:57.502Z","avatar_url":"https://github.com/magnusbaeck.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Logstash Filter Verifier\n\n![build](https://github.com/magnusbaeck/logstash-filter-verifier/actions/workflows/test.yml/badge.svg?event=push)\n[![GoReportCard](https://goreportcard.com/badge/github.com/magnusbaeck/logstash-filter-verifier)](https://goreportcard.com/report/github.com/magnusbaeck/logstash-filter-verifier)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://raw.githubusercontent.com/magnusbaeck/logstash-filter-verifier/master/LICENSE)\n\n* [Introduction](#introduction)\n* [Installing](#installing)\n* [Standalone and Daemon mode (since Version 2.0)](#standalone-and-daemon-mode-since-version-20)\n* [Examples](#examples)\n  * [Syslog messages](#syslog-messages)\n  * [Beats messages](#beats-messages)\n  * [JSON messages](#json-messages)\n  * [Version 2.0 (Daemon mode only)](#version-20-daemon-mode-only)\n* [Test case file reference](#test-case-file-reference)\n  * [Standalone mode / Logstash Filter Verifier before version 2.0](#standalone-mode--logstash-filter-verifier-before-version-20)\n  * [Daemon mode](#daemon-mode)\n    * [Plugin mock](#plugin-mock)\n* [Migrating to the current test case file format](#migrating-to-the-current-test-case-file-format)\n* [Notes](#notes)\n  * [The \\-\\-sockets flag](#the---sockets-flag)\n  * [The \\-\\-logstash\\-arg flag](#the---logstash-arg-flag)\n  * [Logstash compatibility](#logstash-compatibility)\n    * [Standalone mode](#standalone-mode)\n    * [Daemon mode](#daemon-mode)\n  * [Windows compatibility](#windows-compatibility)\n  * [Plugin ID (Daemon mode)](#plugin-id-daemon-mode)\n  * [Logstash Plugins](#logstash-plugins)\n  * [@metadata field](#metadata-field)\n* [Development](#development)\n  * [Dependencies](#dependencies)\n  * [Run Integration Tests](#run-integration-tests)\n* [Known limitations and future work](#known-limitations-and-future-work)\n* [License](#license)\n\n\n## Introduction\n\nThe [Logstash](https://www.elastic.co/products/logstash) program for\ncollecting and processing logs from is popular and commonly used to\nprocess e.g. syslog messages and HTTP logs.\n\nApart from ingesting log events and sending them to one or more\ndestinations it can transform the events in various ways, including\nextracting discrete fields from flat blocks of text, joining multiple\nphysical lines into singular logical events, parsing JSON and XML, and\ndeleting unwanted events. It uses its own domain-specific\nconfiguration language to describe both inputs, outputs, and the\nfilters that should be applied to events.\n\nWriting the filter configurations necessary to parse events isn't\ndifficult for someone with basic programming skills, but verifying\nthat the filters do what you expect can be tedious; especially when\nyou tweak existing filters and want to make sure that all kinds of\nlogs will continue to be processed as before. If you get something\nwrong you might have millions of incorrectly parsed events before you\nrealize your mistake.\n\nThis is where Logstash Filter Verifier comes in. It lets you define\ntest case files containing lines of input together with the expected\noutput from Logstash. Pass one of more such test case files to\nLogstash Filter Verifier together with all of your Logstash filter\nconfiguration files and it'll run Logstash for you and verify that\nLogstash actually returns what you expect.\n\nBefore you can run Logstash Filter Verifier you need to install\nit. After covering that, let's start with a simple example and follow\nup with reference documentation.\n\n\n## Installing\n\nAll releases of Logstash Filter Verifier are published in binary form\nfor the most common platforms at\n[github.com/magnusbaeck/logstash-filter-verifier/releases](https://github.com/magnusbaeck/logstash-filter-verifier/releases).\n\nIf you need to run the program on other platforms or if you want to\nmodify the program yourself you can build and use it on any platform\nfor which a recent [Go](https://golang.org/) compiler is\navailable. Pretty much any platform where Logstash runs should be\nfine, including Windows.\n\nMany Linux distributions make some version of the Go compiler easily\ninstallable, but otherwise you can [download and install the latest\nversion](https://golang.org/dl/). The source code is written to use\n[Go modules](https://github.com/golang/go/wiki/Modules) for dependency\nmanagement. You need at least least Go 1.16.x.\n\nTo just build an executable file you don't need anything but the Go\ncompiler; just clone the Logstash Filter Verifier repository and run\n`go build` from the root directory of the cloned repostiory. If\nsuccessful you'll find an executable in the current directory.\n\nOne drawback of this is that the program won't get stamped with the\ncorrect version number, so `logstash-filter-verifier --version` will\nsay \"unknown\"). To address this and make it easy to run tests and\nstatic checks you need GNU make and other GNU tools.\n\nThe makefile can also be used to install Logstash Filter Verifier\ncentrally, by default in /usr/local/bin but you can change that by\nmodifying the PREFIX variable. For example, to install it in $HOME/bin\n(which is probably in your shell's path) you can issue the following\ncommand:\n\n    $ make install PREFIX=$HOME\n\n\n## Standalone and Daemon mode (since Version 2.0)\n\nSince version 2.0, there are two different modes, Logstash Filter Verifier\ncan be operated in.\n\n1. **Standalone**: In this mode, for each test run a fresh instance of Logstash\n   is started in the background by Logstash Filter Verifier. If a user wants to\n   frequently execute test cases, this might be slow and tedious.  \n   This has been to only mode available in versions prior to 2.0.\n2. **Daemon**: In this mode, Logstash Filter Verifier is executed twice in\n   parallel (preferably in two different shells). One instance is the daemon.\n   The daemon starts and controls the Logstash instances (there might be\n   multiple). This daemon process is normally left running for the time the user\n   is working on the Logstash configuration and testing it with Logstash Filter\n   Verifier.  \n   For each execution of the test cases, another instance Logstash Filter\n   Verifier is started (client). The client collects the current state of the\n   Logstash configuration as well as the test cases and passes them to the\n   daemon. The daemon reloads the configuration in one of the running Logstash\n   instances, executes the test cases and returns the result back to the client.\n   The client shows the results to the user and exits, while the daemon\n   continues to run and waits for the next client to submit a test execution\n   job.\n\n\n## Examples\n\nThe examples that follow build upon each other and do not only show\nhow to use Logstash Filter Verifier to test that particular kind of\nlog. They also highlight how to deal with different features in logs.\n\n\n### Syslog messages\n\nLogstash is often used to parse syslog messages, so let's use that as\na first example.\n\nTest case files are in JSON or YAML format and contain a single object\nwith about a handful of supported properties.\n\nSample with JSON format:\n```json\n{\n  \"fields\": {\n    \"type\": \"syslog\"\n  },\n  \"testcases\": [\n    {\n      \"input\": [\n        \"Oct  6 20:55:29 myhost myprogram[31993]: This is a test message\"\n      ],\n      \"expected\": [\n        {\n          \"@timestamp\": \"2015-10-06T20:55:29.000Z\",\n          \"host\": \"myhost\",\n          \"message\": \"This is a test message\",\n          \"pid\": 31993,\n          \"program\": \"myprogram\",\n          \"type\": \"syslog\"\n        }\n      ]\n    }\n  ]\n}\n```\n\nSample with YAML format:\n```yaml\nfields:\n  type: \"syslog\"\ntestcases:\n  - input:\n      - \"Oct  6 20:55:29 myhost myprogram[31993]: This is a test message\"\n    expected:\n      - \"@timestamp\": \"2015-10-06T20:55:29.000Z\"\n        host: \"myhost\"\n        message: \"This is a test message\"\n        pid: 31993\n        program: \"myprogram\"\n        type: \"syslog\"\n```\n\nMost Logstash configurations contain filters for multiple kinds of\nlogs and uses conditions on field values to select which filters to\napply. Those field values are typically set in the input plugins. To\nmake Logstash treat the test events correctly we can \"inject\"\nadditional field values to make the test events look like the real\nevents to Logstash. In this example, `fields.type` is set to \"syslog\"\nwhich means that the input events in the test cases in this file will\nhave that in their `type` field when they're passed to Logstash.\n\nNext, in `input`, we define a single test string that we want to feed\nthrough Logstash, and the `expected` array contains a one-element\narray with the event we expect Logstash to emit for the given input.\n\nThe `testcases` array can contain multiple objects with `input` and\n`expected` keys. For example, if we change the example above to\n\n```yaml\nfields:\n  type: \"syslog\"\ntestcases:\n  - input:\n      - \"Oct  6 20:55:29 myhost myprogram[31993]: This is a test message\"\n    expected:\n      - \"@timestamp\": \"2015-10-06T20:55:29.000Z\"\n        host: \"myhost\"\n        message: \"This is a test message\"\n        pid: 31993\n        program: \"myprogram\"\n        type: \"syslog\"\n  - input:\n      - \"Oct  6 20:55:29 myhost myprogram: This is a test message\"\n    expected:\n      - \"@timestamp\": \"2015-10-06T20:55:29.000Z\"\n        host: \"myhost\"\n        message: \"This is a test message\"\n        program: \"myprogram\"\n        type: \"syslog\"\n```\n\nwe also test syslog messages that lack the bracketed pid after the\nprogram name.\n\nNote that UTC is the assumed timezone for input events to avoid\ndifferent behavior depending on the timezone of the machine where\nLogstash Filter Verifier happens to run. This won't affect time\nformats that include a timezone.\n\nThis command will run this test case file through Logstash Filter\nVerifier (replace all \"path/to\" with the actual paths to the files,\nobviously):\n\n    $ path/to/logstash-filter-verifier standalone path/to/syslog.json path/to/filters\n\nIf the test is successful, Logstash Filter Verifier will terminate\nwith a zero exit code and (almost) no output. If the test fails it'll\nrun `diff -u` (or some other command if you use the `--diff-command`\nflag) to compare the pretty-printed JSON representation of the\nexpected and actual events.\n\nThe actual event emitted by Logstash will contain a `@version` field,\nbut since that field isn't interesting it's ignored by default when\nreading the actual event. Hence we don't need to include it in the\nexpected event either. Additional fields can be ignored with the\n`ignore` array property in the test case file (see details below).\n\n### Beats messages\n\nIn [Beats](https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html)\nyou can also specify fields to control the behavior of the Logstash pipeline.  \nAn example in Beats config might look like this:\n\n```yaml\n- input_type: log\n  paths: [\"/var/log/work/*.log\"]\n  fields:\n    type: openlog\n- input_type: log\n  paths: [\"/var/log/trace/*.trc\"]\n  fields:\n    type: trace\n```\n\nThe Logstash configuration would then look like this to check the\ngiven field:\n\n```none\nif ([fields][type] == \"openlog\") {\n   Do something for type openlog\n```\n\nBut, in order to test the behavior with LFV you have to give it like so:\n\n```none\n{\n  \"fields\": {\n    \"[fields][type]\": \"openlog\"\n  },\n```\n\nThe reason is, that Beats is inserting by default declared fields under a\nroot element `fields`, while the LFV is just considering it as a configuration\noption.  \nAlternatively you can tell Beats to insert the configured fields on root:\n\n```yaml\nfields_under_root: true\n```\n\n### JSON messages\n\nI always prefer to configure applications to emit JSON objects\nwhenever possible so that I don't have to write complex and/or\nambiguous grok expressions. Here's an example:\n\n```json\n{\"message\": \"This is a test message\", \"client\": \"127.0.0.1\", \"host\": \"myhost\", \"time\": \"2015-10-06T20:55:29Z\"}\n```\n\nWhen you feed events like this to Logstash it's likely that the\ninput used will have its codec set to \"json_lines\". This is something we\nshould mimic on the Logstash Filter Verifier side too. Use `codec` for\nthat:\n\nSample with JSON format:\n\n```json\n{\n  \"fields\": {\n    \"type\": \"app\"\n  },\n  \"codec\": \"json_lines\",\n  \"ignore\": [\"host\"],\n  \"testcases\": [\n    {\n      \"input\": [\n        \"{\\\"message\\\": \\\"This is a test message\\\", \\\"client\\\": \\\"127.0.0.1\\\", \\\"time\\\": \\\"2015-10-06T20:55:29Z\\\"}\"\n      ],\n      \"expected\": [\n        {\n          \"@timestamp\": \"2015-10-06T20:55:29.000Z\",\n          \"client\": \"localhost\",\n          \"clientip\": \"127.0.0.1\",\n          \"message\": \"This is a test message\",\n          \"type\": \"app\"\n        }\n      ]\n    }\n  ]\n}\n```\n\nSample with YAML format:\n\n```yaml\nfields:\n  type: \"app\"\ncodec: \"json_lines\"\nignore:\n  - \"host\"\ntestcases:\n  - input:\n      - \u003e\n        {\n          \"message\": \"This is a test message\",\n          \"client\": \"127.0.0.1\",\n          \"time\": \"2015-10-06T20:55:29Z\"\n        }\n    expected:\n      - \"@timestamp\": \"2015-10-06T20:55:29.000Z\"\n        client: \"localhost\"\n        clientip: \"127.0.0.1\"\n        message: \"This is a test message\"\n        type: \"app\"\n```\n\nThere are a few points to be made here:\n\n* The double quotes inside the string must be escaped when using JSON format.\n  YAML files sometimes require quoting too; for example if the value starts\n  with `[` or `{` or if a numeric value should be forced to be parsed as a\n  string.\n* Together with the lack of a need to escape double quotes inside JSON\n  strings, the use of `\u003e` to create folded lines in the YAML representation\n  makes the input JSON much easier to read.\n* The filters being tested here use Logstash's [dns\n  filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dns.html)\n  to transform the IP address in the `client` field into a hostname\n  and copy the original IP address into the `clientip` field. To avoid\n  future problems and flaky tests, pick a hostname or IP address for\n  the test case that will always resolve to the same thing. As in this\n  example, localhost and 127.0.0.1 should be safe picks.\n* If the input event doesn't contain a `host` field, Logstash will add\n  such a field containing the name of the current host. To avoid test\n  cases that behave differently depending on the host where they're\n  run, we ignore that field with the `ignore` property.\n\n\n### Version 2.0 (Daemon mode only)\n\nWith version 2.0 of Logstash Filter Verifier (Daemon mode) some new features\nhave been added:\n\n* **Export of @metadata**:  \n  There is out of the box support to let Logstash Filter Verifier export\n  the values in the (otherwise hidden) `@metadata` field of the event.\n  This allows to write test cases, which take the values in the `@metadata`\n  field into account. (see `export_metadata` in test case file reference)\n* **Pipeline configuration**:  \n  Logstash Filter Verifier in Daemon mode accepts complete Logstash pipelines\n  as configuration. This includes the localization of the Logstash configuration\n  files through the paths provided in the `pipelines.yml` file and replacing all\n  input and output filters with the respective parts to execute the tests.\n* **Multiple pipelines**  \n  The pipeline configuration may consist of multiple pipelines, that might be\n  linked ([pipeline to pipeline communication](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html))\n  or independent pipelines.\n* **Plugin mock**  \n  Plugin mock allows to replace (or remove) plugins in the Logstash\n  configuration under test, that do not work during or that would potentially\n  not produce the expected results test execution. Examples for such filter\n  plugins are mainly plugins, that perform some sort of call out to a third\n  party system, for example to look up data ([elasticsearch](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html),\n  [http](https://www.elastic.co/guide/en/logstash/current/plugins-filters-http.html),\n  [jdbc](https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_static.html),\n  [memcached](https://www.elastic.co/guide/en/logstash/current/plugins-filters-memcached.html)).\n  In order to to be able to produce reproducible results in the test cases,\n  these plugins can be replaced with mocks. In particular the [mutate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)\n  and the [translate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html)\n  filters have proven to be helpful as replacements.\n  Other use cases for mocks are:\n  * replace filters, that operate on environment variables, that need to be changed\n    between different test cases (see [#120](https://github.com/magnusbaeck/logstash-filter-verifier/issues/120)).\n  * replace pipeline input and output plugins in order to test pipelines in isolation.\n\nIn order to execute a test case in daemon mode, first the daemon needs to be\nstarted (e.g. in its own terminal or shell):\n\n    $ path/to/logstash-filter-verifier daemon start\n\nNext, a single test case run can be launched with (in second terminal/shell):\n\n    $ path/to/logstash-filter-verifier daemon run --pipeline path/to/pipelines.yml --pipeline-base base/path/of/logstash-configuration --testcase-dir path/to/testcases\n\nThe flag `--pipeline-base` is required, if the `pipelines.yml` file does use\nrelative paths for the actual logstash pipeline configuration.\n\nIf the Logstash configuration under test does not contain `id` attributes for\nall plugins, the `--add-missing-id` flag instructs Logstash Filter Verifier to\nadd the missing `id` attributes on the fly.\n\nAs an example, we can execute the `basic_pipeline` test case from this\nrepository.\n\nLet us assume, the following setup:\n\n* The logstash filter verifier binary is available at `/usr/local/bin/logstash-fitler-verifier`.\n* This repository is available at `/tmp/logstash-filter-verifier` (e.g. with `git clone https://github.com/magnusbaeck/logstash-filter-verifier `).\n\nThe command to run the `basic_pipeline` example would look like this (the daemon\nneeds to be started beforehand):\n\n    $ /usr/local/bin/logstash-fitler-verifier daemon run --pipeline /tmp/logstash-filter-verifier/testdata/basic_pipeline.yml --pipeline-base /tmp/logstash-filter-verifier/testdata/basic_pipeline --testcase-dir /tmp/logstash-filter-verifier/testdata/testcases/basic_pipeline --add-missing-id\n\nMore examples (e.g. multiple pipelines and plugin mock) can be found in the\n`testcases/` folder of this repository.\n\n\n## Test case file reference\n\n### Standalone mode / Logstash Filter Verifier before version 2.0\n\nTest case files are JSON files containing a single object. That object\nmay have the following properties:\n\n* `codec`: A string with the codec configuration of the input plugin used\n  when executing the tests. This string will be included verbatim in the\n  Logstash configuration so it could either be just the name of the codec\n  plugin (normally `line` or `json_lines`) or include additional codec\n  options like e.g. `plain { charset =\u003e \"ISO-8859-1\" }`.\n* `fields`: An object containing the fields that all input messages\n  should have. This is vital since filters typically are configured\n  based on the event's type and/or tags. Scalar values (strings,\n  numbers, and booleans) are supported, as are objects (containing\n  scalars, arrays and nested objects), arrays of scalars and nested arrays.\n  The only combination which is not allowed are objects within arrays.\n  A shorthand for defining nested fields is to use the Logstash's field\n  reference syntax (`[field][subfield]`), i.e.\n  `fields: {\"[log][file][path]\": \"/tmp/test.log\"}` is equivalent to\n  `fields: {\"log\": {\"file\": {\"path\": \"/tmp/test.log\"}}}`.\n* `ignore`: An array with the names of the fields that should be\n  removed from the events that Logstash emit. This is for example\n  useful for dynamically generated fields whose contents can't be\n  predicted and hardwired into the test case file. If you need to exclude\n  individual subfields you can use Logstash's field reference syntax,\n  i.e. `[log][file][path]` will exclude that field but keep other subfields\n  of `log` like e.g. `[log][level]` and `[log][file][line]`.\n* `testcases`: An array of test case objects, each having the following\n  contents:\n  * `input`: An array with the lines of input (each line being a string)\n    that should be fed to the Logstash process. If you use `json_lines` codec\n    you can use Logstash's syntax reference syntax for fields in the JSON\n    object, making\n    `{\"message\": \"my message\", \"[log][file][path]\": \"/tmp/test.log\"}`\n    equivalent to\n    `{\"message\": \"my message\", \"log\": {\"file\": {\"path\": \"/tmp/test.log\"}}}`.\n  * `expected`: An array of JSON objects with the events to be\n    expected. They will be compared to the actual events produced by the\n    Logstash process.\n  * `description`: An optional textual description of the test case, e.g.\n    useful as documentation. This text will be included in the program's\n    progress messages.\n\n\n### Daemon mode\n\nTest case files for the Daemon mode have the same fields as for Standalone mode\nwith the following changes/additions\n\nAdditional fields:\n\n* `input_plugin`: The unique [ID](https://www.elastic.co/guide/en/logstash/7.10/plugins-inputs-file.html#plugins-inputs-file-id)\n  of the input plugin in the tested configuration, where the test input is\n  coming from. This is necessary, if a setup with multiple inputs is tested,\n  which either have different codecs or are part of different pipelines.\n* `export_metadata`: Controls if the metadata of the event processed by Logstash\n  is returned. The metadata is contained in the field `[@metadata]` in the\n  Logstash event. If the metadata is exported, the respective fields are\n  compared with the expected result of the testcase as well. (default: false)\n* `export_outputs`: Controls if the ID of the output, a particular event has\n  emitted by, is kept in the event or not. If this is enabled, the expected\n  event needs to contain a field named `_lfv_out_passed` which contains the ID\n  of the Logstash output.\n* `testcases`:\n  * `fields`: Local fields, only added to the events of this test case. These\n    fields overwrite global fields.\n\nIgnored / obsolete fields:\n\n* `codec`\n\n\n#### Plugin mock\n\nThe plugin mock config file (yaml) consists of an array of plugin mock elements.\nEach plugin mock element consists for the plugin id that should be replaced as\nwell as the Logstash configuration string that should be used as the\nreplacement. This string might be empty. In this case, the mocked plugin is just\nremoved from the Logstash configuration.\n\nExample:\n\n```yaml\n- id: removeme\n- id: mockme\n  mock: |\n    mutate {\n      replace =\u003e {\n        \"[message]\" =\u003e \"mocked\"\n      }\n    }\n```\n\nGiven the above plugin mock configuration, the plugin with the ID `removeme` is\nremoved from the Logstash configuration. The plugin with the ID `mockme` is\nreplaced with the given Logstash configuration.\n\n## Migrating to the current test case file format\n\nOriginally the `input` and `expected` configuration keys were at the\ntop level of the test case file. They were later moved into the\n`testcases` key.\n\nTo migrate test case files from the old to the new file format the\nfollowing command using [jq](https://stedolan.github.io/jq/) can be\nused (run it in the directory containing the test case files):\n\n```\nfor f in *.json ; do\n    jq '{ codec, fields, ignore, testcases:[[.input[]], [.expected[]]] | transpose | map({input: [.[0]], expected: [.[1]]})} | with_entries(select(.value != null))' $f \u003e $f.migrated \u0026\u0026 mv $f.migrated $f\ndone\n```\n\nThis command only works for test case files where there's a one-to-one\nmapping between the elements of the `input` array and the elements of\nthe `expected` array. If you e.g. have drop and/or split filters in\nyour Logstash configuration you'll have to patch the converted test\ncase file by hand afterwards.\n\n\n## Notes\n\n### The `--sockets` flag (Standalone mode)\n\nThe command line flag `--sockets` allows to use unix domain sockets instead of\nstdin to send the input to Logstash. The advantage of this approach is, that\nit allows to process test case files in parallel to Logstash, instead of\nstarting a new Logstash instance for every test case file. Because Logstash\nis known to start slowly, this increases the time needed significantly,\nespecially if there are lots of different test case files.\n\nFor the test cases to work properly together with the unix domain socket input,\nthe test case files need to include the property `codec` set to the value `line`\n(or `json_lines`, if json formatted input should be processed).\n\n\n### The `--logstash-arg` flag\n\nThe `--logstash-arg` flag is used to supply additional command line\narguments or flags for Logstash. Those arguments are not processed by\nLogstash Filter Verifier other than just forwarding them to Logstash.\nFor flags consisting of a flag name and a value, for both a seperate\n`--logstash-arg` in the correct order has to be provided.  Because\nvalues, starting with one or two dashes (`-`) are treated as flag by\nLogstash Filter Verifier, for those flags the value _must_ not be\nseparated using a space but they have to be separated from the flag\nwith the equal sign (`=`).\n\nFor example to set the Logstash node name the following arguments have\nto be provided to Logstash Filter Verifier:\n\n    --logstash-arg=--node.name --logstash-arg MyInstanceName\n\n\n### Logstash compatibility\n\n#### Standalone mode\n\nDifferent versions of Logstash behave slightly differently and changes\nin Logstash may require changes in Logstash Filter Verifier. Upon\nstartup, the program will attempt to auto-detect the version of\nLogstash used and will use this information to adapt its own behavior.\n\nStarting with Logstash 5.0 finding out the Logstash version is very\nquick but in previous versions the version string was printed by Ruby\ncode in the JVM so it took several seconds. To avoid this you can use\nthe `--logstash-version` flag to tell Logstash Filter Verifier which\nversion of Logstash it should expect. Example:\n\n    logstash-filter-verifier standalone ... --logstash-version 2.4.0\n\n\n#### Daemon mode\n\nIn order to use Logstash Filter Verifier in Daemon mode, at least Logstash\nversion 6.7.x is required. Older versions of Logstash are not supported and do\nnot work with Daemon mode.\n\n\n### Windows compatibility\n\nLogstash Filter Verifier has been reported to work on Windows, but\nthis isn't tested by the author and it's not guaranteed to work. There\nare a couple of known quirks that are easy to work around:\n\n* It won't guess the location of your Logstash executable so you'll have\n  to manually provide it with the `--logstash-path` flag.\n* The default value of the `--diff-command` is `diff -u` which won't work\n  on typical Windows machines. You'll have to explicitly select which diff\n  tool to use.\n\n\n### Plugin ID (Daemon mode)\n\nThe Daemon mode of Logstash Filter Verifier expects each plugin in the Logstash\nconfiguration to have a unique [ID](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-id).\nIn order to test an existing Logstash configuration, which lacks these ID, there\nare two options:\n\n1. Permanently add the missing ID to the configuration. This can either be done\n   by hand or with the help of [`mustache`](https://github.com/breml/logstash-config).\n\n       mustache lint --auto-fix-id \u003cLogstash config files\u003e\n\n2. Let Logstash Filter Verifier add the ID temporarily just for the execution\n   of the test cases by adding the flag `--add-missing-id`.\n\n\n### Logstash Plugins\n\n#### Clone (Filter)\n\nLFV configures Logstash such that the ordering of the events while being\nprocessed by the pipeline is ensured. In this regard, the [`clone` filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html)\nis special, because it injects additional events into the processing pipeline.\n\nThe cloned events can be expected in the following order:\n\n1. original event\n2. clones of the event in the order of appearance in the `clones` attribute\n   of the `clone` filter.\n\n\n### @metadata field\n\nEvents in Logstash have a special field called [`@metadata`](https://www.elastic.co/guide/en/logstash/master/event-dependent-configuration.html#metadata).\nThe contents of `@metadata` are not part of any of your events at output time.\nLFV does make use of `@metadata` to store information, which is required to\ncontrol the processing of the events. To ensure the correct operation of LFV,\nthe Logstash configuration under test is not allowed to alter any field within\nthe `@metadata`, which start with `__lfv_`. For example the following fields\nare used by LFV (not conclusive):\n\n* `[@metadata][__lfv_id]`\n* `[@metadata][__lfv_out_passed]`\n\nThese fields are also removed from `@metadata` even when the test case definition\ndoes include `export_metadata`.\n\nAdditionally, all fields within `@metadata` starting with `__tmp_` are removed.\n\nFor debug purposes, the `--debug` flag may be passed, which prevents the\nremoval of the above mentioned fields.\n\n\n## Development\n\n### Dependencies\n\nFor a fully working development environment, the following tooling needs to be\npresent:\n\n* Go compiler\n* `make` command\n* Proto buffer compiler (`protobuf-compiler`)\n\n\n### Run Integration Tests\n\nIn order to run the integration tests, the following preparation is needed:\n\n1. Run `go run . setup 8.12.1` to download Logstash version 8.12.1\n2. Prepare a `logstash-filter-verifier.yml` config file, which points to the\n   downloaded Logstash version, e.g.\n\n   ```yaml\n   ---\n\n   loglevel: debug\n   logstash:\n     path: ./3rdparty/logstash-8.12.1-linux-x86_64/bin/logstash\n   keep-envs:\n     - TESTMODE\n   metadata-key: __metadata\n   ```\n\n3. Execute the integration tests by providing the respective env var:\n\n   `INTEGRATION_TEST=1 go test -v .`\n\n   or\n\n   ```shell\n   export INTEGRATION_TEST=1\n   go test -v .\n   ```\n\n\n## Known limitations and future work\n\n* Some log formats don't include all timestamp components. For\n  example, most syslog formats don't include the year. This should be\n  dealt with somehow.\n* Logstash configurations, which remove the `@timestamp` field can not be tested\n  using the daemon mode (see [#151](https://github.com/magnusbaeck/logstash-filter-verifier/issues/151)).\n* The maximum size of events, that can be processed is ~64KB (see [#169](https://github.com/magnusbaeck/logstash-filter-verifier/issues/169)).\n\n\n## License\n\nThis software is copyright 2015–2021 by Magnus Bäck \u003c\u003cmagnus@noun.se\u003e\u003e and\nother contributors and licensed under the Apache 2.0 license. See the LICENSE\nfile for the full license text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmagnusbaeck%2Flogstash-filter-verifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmagnusbaeck%2Flogstash-filter-verifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmagnusbaeck%2Flogstash-filter-verifier/lists"}