{"id":19788341,"url":"https://github.com/graylog2/graylog-guide-syslog-amqp","last_synced_at":"2025-02-28T08:38:48.061Z","repository":{"id":137547621,"uuid":"59660241","full_name":"Graylog2/graylog-guide-syslog-amqp","owner":"Graylog2","description":"How to use send Syslog messages via AMQP to Graylog","archived":false,"fork":false,"pushed_at":"2020-04-21T10:23:19.000Z","size":9,"stargazers_count":9,"open_issues_count":0,"forks_count":4,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-01-11T03:49:05.788Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Graylog2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-25T12:11:35.000Z","updated_at":"2022-10-30T13:41:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"60f7ca7f-aa4d-42f2-96ab-d0ca876d9263","html_url":"https://github.com/Graylog2/graylog-guide-syslog-amqp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graylog2%2Fgraylog-guide-syslog-amqp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graylog2%2Fgraylog-guide-syslog-amqp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graylog2%2Fgraylog-guide-syslog-amqp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graylog2%2Fgraylog-guide-syslog-amqp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Graylog2","download_url":"https://codeload.github.com/Graylog2/graylog-guide-syslog-amqp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241125008,"owners_count":19913839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T06:26:56.303Z","updated_at":"2025-02-28T08:38:48.028Z","avatar_url":"https://github.com/Graylog2.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sending Syslog via AMQP into Graylog\n\nIf your setup does not allow direct communication from all hosts to the Graylog Server or your Graylog Server is located inside a private Network you could use AMQP as Transport. You will need to have an AMQP Server like rabbitMQ reachable by all Hosts. But it's easy to secure this communication if transport is over an insecure wire. Forget more Information please read [the rabbitMQ SSL Guide](https://www.rabbitmq.com/ssl.html).\n\n```\nThis Guide will not give you a complete copy\u0026paste how-to,\nbut it will guide you and provide additional information.\n\nPlease do not follow the steps if you did not know how to deal\nwith common issues yourself.   \n```\n\n\nIn this scenario a Syslog message will have the following stages:\n\n- transformed into JSON by [rsyslog](http://www.rsyslog.com)\n- send from rsyslog to [logstash](https://www.elastic.co/products/logstash) via TCP/UDP\n- send from logstash to [rabbitMQ](https://www.rabbitmq.com)\n- consumed by graylog from rabbitMQ\n- Syslog extracted from JSON by Graylog\n\nWe will assume that you have a rabbitMQ running on **amqp.ext.example.org (203.0.113.2)** and your Graylog Instance is running on **graylog.int.example.org (192.168.0.10)**. Additional we have the Linux System **syslog.o1.example.org (198.51.100.1)** and **syslog.o2.example.org (192.0.2.1)** that will send Syslog Data. All Systems are running *ubuntu* so you might need to adjust some configuration path settings.\n\n## prepare rabbitMQ\nIf no AMQP Broker is present, [install rabbitMQ](https://www.rabbitmq.com/install-debian.html) on **amqp.ext.example.org** and create a user for log delivery on CLI.\n\n```\nrabbitmqctl add_user my_rabbite_mq_user_here my_super_secure_password_rabbit_mq_password\nrabbitmqctl set_permissions -p / my_rabbite_mq_user_here \".*\" \".*\" \".*\"\n```\n\nIf this Server is available *in the wild* please enable SSL  in your Setup. A Management GUI can be [installed with a few commands](https://www.rabbitmq.com/management.html) and an [admin User is Created ](http://stackoverflow.com/questions/22850546/cant-access-rabbitmq-web-management-interface-after-fresh-install) similar to the User creation above.\n```\nlisteners.ssl.1 = 5671                  # this is the secure port for rabbitmq\n#ssl_options.verify               = verify_peer\n#ssl_options.fail_if_no_peer_cert = true\nssl_options.cacertfile           = /path/ssl/ca.crt\nssl_options.certfile             = /path/ssl/ssl/yourserver.crt\nssl_options.keyfile              = /path/ssl/yourserver.key\n```\n\n\n## send messages on rsyslog\nWith rsyslog, you can use templates to format how messages should look like. Formatting the messages direct at the source will help to have a clean message from the source to the destination.\n\nTo identify the messages with the Full Qualified Domain Name of the System that has created the message we use the Option ``PreserveFQDN`` - but you will need to have a clean working hostname resolution.\n\nrsyslog will send the message via UDP to the local running logstash.\n\n```\n$PreserveFQDN on\ntemplate(name=\"ls_json\"\n         type=\"list\"\n         option.json=\"on\") {\n           constant(value=\"{\")\n             constant(value=\"\\\"@timestamp\\\":\\\"\")     property(name=\"timereported\" dateFormat=\"rfc3339\")\n             constant(value=\"\\\",\\\"@version\\\":\\\"1\")\n             constant(value=\"\\\",\\\"message\\\":\\\"\")     property(name=\"msg\")\n             constant(value=\"\\\",\\\"host\\\":\\\"\")        property(name=\"hostname\")\n             constant(value=\"\\\",\\\"severity\\\":\\\"\")    property(name=\"syslogseverity-text\")\n             constant(value=\"\\\",\\\"facility\\\":\\\"\")    property(name=\"syslogfacility-text\")\n             constant(value=\"\\\",\\\"programname\\\":\\\"\") property(name=\"programname\")\n             constant(value=\"\\\",\\\"procid\\\":\\\"\")      property(name=\"procid\")\n           constant(value=\"\\\"}\\n\")\n         }\n\n*.* @127.0.0.1:5514;ls_json\n```\n\nThe configuration above need to be placed inside the ``/etc/rsyslog.d/90-logstash.conf`` on **syslog.01.example.org** and **syslog.o2.example.org** in our example and rsyslog need to be restarted (``service rsyslog restart``).\n\n\n## route messages with logstash\nAs of writing this, rsyslog was not able to send messages direct to AMQP on Ubuntu, so we need to use logstash for the transport.\n\nLogstash will listen on *localhost* port *udp/5514* for the messages that are coming from rsyslog and forward them to the rabbitMQ Server.\n\n```\ninput {\n    UDP {\n        port =\u003e 5514\n        host =\u003e \"127.0.0.1\"\n        type =\u003e syslog\n        codec =\u003e \"json\"\n        }\n}\n\nfilter {\n  # This replaces the host field (UDP source) with the host that generated the message (sysloghost)\n  if [sysloghost] {\n      mutate {\n          replace =\u003e [ \"host\", \"%{sysloghost}\" ]\n          remove_field =\u003e \"sysloghost\" # prune the field after successfully replacing \"host\"\n        }\n      }\n}\n\noutput {\n    rabbitmq {\n      exchange =\u003e \"log-messages\"\n        exchange_type =\u003e \"fanout\"\n        key =\u003e \"log-messages\"\n        host =\u003e \"amqp.ext.example.org\"\n        durable =\u003e true\n        persistent =\u003e true\n        port =\u003e 5672\n        user =\u003e \"my_rabbite_mq_user_here\"\n        password =\u003e \"my_super_secure_password_rabbit_mq_password\"\n        verify_ssl =\u003e true  # we assume that you have a valid certificate!\n      }\n    }\n```\n\n## consume messages with graylog\nNow the Data need to be consumed by graylog. Create an [input](http://docs.graylog.org/en/2.0/pages/getting_started/config_input.html) with the Input *Syslog AMQP*. Add the Information that is configured in the former steps (exchange, username, password, hostname). Set the Option *Allow overwrite date*.\n\nStart the Input to consume the first messages and create [a JSON extractor](http://docs.graylog.org/en/2.0/pages/extractors.html#using-the-json-extractor). Additional create a second extractor on the field `host` and the type `copy input` and store it in the field `source`. You might want a third `copy input` to store `@timestamp` in `timestamp`.\n\n## what's next?\nUse the *rsyslog* Systems as Syslog Proxies for every possible source in the same network, add more systems to your setup.\n\n\n# Credits\n- untergeek for [rsyslog / json template](https://gist.github.com/untergeek/0373ee85a41d03ae1b78) and the [blogpost](http://untergeek.com/2012/10/11/using-rsyslog-to-send-pre-formatted-json-to-logstash/)\n- IETF for [documentation ips](https://tools.ietf.org/html/rfc5737)\n- StackOverflow User Gabriele for the answer [how to create User on CLI](http://stackoverflow.com/questions/22850546/cant-access-rabbitmq-web-management-interface-after-fresh-install)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraylog2%2Fgraylog-guide-syslog-amqp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraylog2%2Fgraylog-guide-syslog-amqp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraylog2%2Fgraylog-guide-syslog-amqp/lists"}