{"id":13783242,"url":"https://github.com/TomasKoutek/logstash-pipeline-parser","last_synced_at":"2025-05-11T18:31:35.835Z","repository":{"id":210448669,"uuid":"726461106","full_name":"TomasKoutek/logstash-pipeline-parser","owner":"TomasKoutek","description":"Parsing expression grammar and Abstract syntax tree for Logstash pipeline syntax.","archived":false,"fork":false,"pushed_at":"2023-12-29T20:52:35.000Z","size":3466,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-07-19T15:12:04.609Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TomasKoutek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-02T13:21:06.000Z","updated_at":"2024-01-16T00:58:37.000Z","dependencies_parsed_at":"2024-01-17T21:26:58.919Z","dependency_job_id":"4e7a7b63-91be-4d06-8b7d-f67374fc125e","html_url":"https://github.com/TomasKoutek/logstash-pipeline-parser","commit_stats":null,"previous_names":["tomaskoutek/logstash-pipeline-parser"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomasKoutek%2Flogstash-pipeline-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomasKoutek%2Flogstash-pipeline-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomasKoutek%2Flogstash-pipeline-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomasKoutek%2Flogstash-pipeline-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TomasKoutek","download_url":"https://codeload.github.com/TomasKoutek/logstash-pipeline-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":213844383,"owners_count":15646637,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T19:00:16.959Z","updated_at":"2024-08-03T19:01:06.729Z","avatar_url":"https://github.com/TomasKoutek.png","language":"Python","funding_links":[],"categories":["Elasticsearch developer tools and utilities"],"sub_categories":["Development and debugging"],"readme":"## About The Project\n\n### How Logstash Pipeline Works\n\u003e The Logstash event processing pipeline has three stages: inputs → filters → outputs.\\\n\u003e Inputs generate events, filters modify them, and outputs ship them elsewhere.\\\n\u003e Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.\n\nThe pipeline configuration file is a custom format developed by the Logstash folks using [Treetop](https://cjheath.github.io/treetop/syntactic_recognition.html).\nThe grammar itself is described in the source file [grammar.treetop](https://github.com/elastic/logstash/tree/v8.11.1/logstash-core/lib/logstash/config/grammar.treetop) and compiled using Treetop into the custom [grammar.rb](https://github.com/elastic/logstash/blob/v8.11.1/logstash-core/lib/logstash/config/grammar.rb) parser.\nThat parser is then used to set up the pipeline from the Logstash configuration.\n\nSee also:\n- [How Logstash Works](https://www.elastic.co/guide/en/logstash/current/pipeline.html)\n- [Creating a Logstash pipeline](https://www.elastic.co/guide/en/logstash/current/configuration.html)\n\n\n## Documentation\n\nThe latest documentation is at https://tomaskoutek.github.io/logstash-pipeline-parser/ and contains description of all methods, classes with individual examples and their results. \nOr you can display description directly with the help function.\n\nFor example:\n\n```python\nfrom logstash_pipeline_parser import Pipeline\n\nhelp(Pipeline.parse)\n```\n\n## Getting Started\n\n![graph](./doc/graphviz.png \"Plugins tree\")\n\n### Installing\n\nInstall from [pip](https://pypi.org/project/logstash-pipeline-parser/):\n\n```\npip install logstash-pipeline-parser\n```\n\n### Dependencies\n- [pyparsing](https://github.com/pyparsing/pyparsing) for creating [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) parser\n\n## Quick Example\n\nFor full documentation with examples please see [documentation](https://tomaskoutek.github.io/logstash-pipeline-parser/).\n\nLet's try pipeline with two input plugins: \n\n```python\nfrom logstash_pipeline_parser import Pipeline\n\ndata = r\"\"\"\n       input {\n         beats {\n           host =\u003e \"0.0.0.0\"\n           port =\u003e 5044\n           client_inactivity_timeout =\u003e 3600\n           include_codec_tag =\u003e true\n           enrich =\u003e [source_metadata, ssl_peer_metadata]\n           ssl =\u003e true\n           ssl_key =\u003e \"/some/path/my.key\"\n           id =\u003e \"input_beats\"\n         }\n         \n         udp {\n           port =\u003e 5045\n           host =\u003e \"0.0.0.0\"\n         }\n       }\n\"\"\"\n\npipeline = Pipeline(data)\nast = pipeline.parse()\n```\n\nwill produce [Abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree):\n\n```python\nfrom ipaddress import IPv4Address\nfrom pathlib import Path\n\n[\n    [\"input\", [\n        [\"beats\", [\n            [\"host\", [IPv4Address(\"0.0.0.0\")]], \n            [\"port\", [5044]], \n            [\"client_inactivity_timeout\", [3600]], \n            [\"include_codec_tag\", [True]], \n            [\"enrich\", [\n                [\"source_metadata\", \"ssl_peer_metadata\"]\n            ]], \n            [\"ssl\", [True]], \n            [\"ssl_key\", [Path(\"/some/path/my.key\")]], \n            [\"id\", [\"input_beats\"]]\n        ]], \n        [\"udp\", [\n            [\"port\", [5045]], \n            [\"host\", [IPv4Address(\"0.0.0.0\")]]\n        ]]\n    ]]\n]\n\n```\nOf course, it is possible to parse all kinds of plugins, conditions and data types.\nYou can also search in the pipeline. The searched key can also contain the wildcard `*`, for example **\"output.\\*.hosts\"** will return\n(if the pipeline definition contains them):\n\n- `(\"output.elasticsearch.hosts\", [[\"127.0.0.1:9200\",\"127.0.0.2:9200\"]])`\n- `(\"output.logstash.hosts\", [\"127.0.0.1:9801\"])`\n\n\n```python\nresults = pipeline.search(\"input.*.port\")\n\nprint(list(results))\n# [\n#   (\"input.beats.port\", [5044]), \n#   (\"input.udp.port\", [5045])\n# ]\n```\n\nThe `search` method returns a generator, so we can easily iterate:\n\n```python\nfor key, value in pipeline.search(\"*.port\"):\n    print(f\"key: {key}, value: {value[0]}\")\n\n# key: input.beats.port, value: 5044\n# key: input.udp.sub.port, value: 5045\n```\n\nThe return value can be any element from the tree (integer, string, field, plugin,...):\n\n```python\nresults = pipeline.search(\"input.beats.enrich\")\n\nprint(list(results))\n# [\n#   (\"input.beats.enrich\", [[\"source_metadata\", \"ssl_peer_metadata\"]])\n# ]\n```\n\n\n## License\n\nDistributed under the MIT License. See LICENSE for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTomasKoutek%2Flogstash-pipeline-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTomasKoutek%2Flogstash-pipeline-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTomasKoutek%2Flogstash-pipeline-parser/lists"}