{"id":19872041,"url":"https://github.com/saltstack-formulas/logstash-formula","last_synced_at":"2025-05-02T09:30:59.823Z","repository":{"id":19894272,"uuid":"23159111","full_name":"saltstack-formulas/logstash-formula","owner":"saltstack-formulas","description":null,"archived":false,"fork":false,"pushed_at":"2022-02-04T13:51:54.000Z","size":23,"stargazers_count":24,"open_issues_count":20,"forks_count":71,"subscribers_count":36,"default_branch":"master","last_synced_at":"2023-03-11T09:43:43.580Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html","language":"SaltStack","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saltstack-formulas.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-20T18:16:28.000Z","updated_at":"2023-03-11T09:43:43.580Z","dependencies_parsed_at":"2022-08-26T18:51:16.461Z","dependency_job_id":null,"html_url":"https://github.com/saltstack-formulas/logstash-formula","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltstack-formulas%2Flogstash-formula","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltstack-formulas%2Flogstash-formula/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltstack-formulas%2Flogstash-formula/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltstack-formulas%2Flogstash-formula/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saltstack-formulas","download_url":"https://codeload.github.com/saltstack-formulas/logstash-formula/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224305531,"owners_count":17289446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T16:14:20.448Z","updated_at":"2024-11-12T16:14:21.236Z","avatar_url":"https://github.com/saltstack-formulas.png","language":"SaltStack","funding_links":[],"categories":[],"sub_categories":[],"readme":"================\nlogstash formula\n================\n\nInstall and configure Logstash for Debian and RedHat based systems using\npillar data.\n\n.. note::\n\n    See the full `Salt Formulas installation and usage instructions\n    \u003chttp://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html\u003e`_.\n\n    Logstash requires Java, either the Oracle implementation or OpenJDK.  Since\n    that is outside the scope of this formula, you must ensure that Java is installed before applying this formula.\n\nAvailable states\n================\n\n.. contents::\n    :local:\n\n``logstash``\n------------\n\nInstall the ``logstash`` package, set up input/filter/output configuration\nfiles, and enable the service.  Compatible only with Salt 2014.1.10+, due to\nrequirement for \"mapping\" test in jinja 2.6.\n\nUsage\n=====\n\nSee pillar.example for an example configuration.\n\nExample\n=======\nThe easiest way to understand the formula is to look at an example.  The following is example pillar data:\n\n::\n    \n    logstash:\n        inputs:\n            -   \n                plugin_name: file\n                path:\n                    - /var/log/syslog\n                    - /var/log/authlog\n                type: syslog\n        filters:\n            -\n                plugin_name: grok\n                match:\n                    message: '%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}'\n                add_field:\n                    received_at: '%{@timestamp}'\n                    received_from: '%{host}'\n        outputs:\n            -\n                plugin_name: lumberjack\n                hosts:\n                    - logs.example.com\n                port: 5000\n                ssl_certificate: /etc/ssl/certs/lumberjack.crt\n\nThat would result in this logstash config (the three separate files it would create are concatenated here):\n\n::\n\n    input {\n        file { \n            path =\u003e [\n                \"/var/log/syslog\",\n                \"/var/log/auth.log\"\n            ]\n            type =\u003e \"syslog\"\n        }\n    }\n    filter {\n        grok {  \n            match =\u003e {\n                message =\u003e \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\"\n            }\n            add_field =\u003e {\n                received_at =\u003e \"%{@timestamp}\"\n                received_from =\u003e \"%{host}\"\n            }\n        }\n        if [log][file][path] and ([log][file][path] == \"/var/log/nginx/admin_access.log\" {\n            mutate {\n                add_field =\u003e {\n                    \"[@metadata][target_pipeline]\" =\u003e \"nginx.pipeline\"\n                    \"[@metadata][target_index]\" =\u003e \"access-nginx\"\n                }\n            }\n        }\n  }\n    }\n    output {\n        lumberjack { \n            hosts =\u003e [\n                \"logs.example.com\"\n            ]\n            port =\u003e \"5000\"\n            ssl_certificate =\u003e \"/etc/ssl/certs/lumberjack.crt\"\n        }\n        if [@metadata][target_pipeline] and [@metadata][target_index] {\n            elasticsearch {\n                pipeline =\u003e \"%{[@metadata][target_pipeline]}\"\n                hosts =\u003e \"elasticsearch.example.com\"\n                index =\u003e \"%{[@metadata][target_index]}-%{+YYYY.MM.dd}\"\n                ssl =\u003e true\n                ssl_certificate_verification =\u003e true\n            }\n        }\n    }\n\n\nFor a more complicated example, including conditionals, see pillar.example.\n\n\nPillar Data Explained\n---------------------\n\nThe pillar data is structured as a dictionary with key 'logstash', followed\nby three optional keys:\n\n* inputs: A list of input plugins, to be rendered in-order to \n  /etc/logstash/conf.d/01-inputs.conf\n* filters: A list of filter plugins, to be rendered in-order to \n  /etc/logstash/conf.d/02-filters.conf\n* outputs: A list of output plugins, to be rendered in-order to \n  /etc/logstash/conf.d/03-outputs.conf\n\nEach list item for any of the three plugin types contains arbitrary\nattributes of type string, number, dictionary, or list which will \nbe rendered into Logstash's configuration syntax.  For a list of plugins\nand their configuration attributes,see \u003chttp://logstash.net/docs/1.4.2/\u003e.\n\nUsing Conditionals\n------------------\nThe only plugin attributes that are unique for this formula is the \"cond\" \nattribute, which is used to set up conditionals.  For example you may want\nto filter a logstash entry only if it meets certain criteria, such as being of\na certain type.  This formula supports if/else if/else by embedding the \nconditional to be used in the \"cond\" attribute of the plugin.  For this reason,\nthis formula does not support nested conditionals at this time.  See\npillar.example for an example of the conditional functionality.\n\nOverriding Defaults\n-------------------\nThis formula sets up certain defaults in map.jinja, specifically:\n\n* Name of the logstash package is logstash\n* Name of the logstash service is logstash\n* The latest version of logstash available will be installed  \n  and kept up to date, instead of a one-time install of the latest version\n  (e.g. use states.pkg.latest instead of states.pkg.installed)\n* The configuration files will use an indentation of four spaces\n\nThese settings can be overridden by adding the appropriate keys to your\npillar data, for example::\n    logstash:\n        pkg: logstash-altversion\n        svc: logstash-alterversion\n        pkgstate: installed # instead of latest\n        indent: 2\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaltstack-formulas%2Flogstash-formula","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaltstack-formulas%2Flogstash-formula","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaltstack-formulas%2Flogstash-formula/lists"}