{"id":27643420,"url":"https://github.com/shrunbr/graylog_pfsense_barnyard2","last_synced_at":"2025-04-24T00:13:55.803Z","repository":{"id":201644133,"uuid":"266237679","full_name":"shrunbr/graylog_pfsense_barnyard2","owner":"shrunbr","description":"A method for parsing Snort Barnyard2 logs from pfSense in Graylog","archived":false,"fork":false,"pushed_at":"2020-05-26T00:48:18.000Z","size":514,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2023-10-19T23:45:49.097Z","etag":null,"topics":["elasticsearch","graylog","json","pfsense","snort"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shrunbr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-05-23T01:03:09.000Z","updated_at":"2023-10-20T01:35:12.957Z","dependencies_parsed_at":null,"dependency_job_id":"59a2cd08-feb7-40dc-8686-b5a43ad92442","html_url":"https://github.com/shrunbr/graylog_pfsense_barnyard2","commit_stats":null,"previous_names":["shrunbr/graylog_pfsense_barnyard2"],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrunbr%2Fgraylog_pfsense_barnyard2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrunbr%2Fgraylog_pfsense_barnyard2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrunbr%2Fgraylog_pfsense_barnyard2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrunbr%2Fgraylog_pfsense_barnyard2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shrunbr","download_url":"https://codeload.github.com/shrunbr/graylog_pfsense_barnyard2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250535112,"owners_count":21446509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","graylog","json","pfsense","snort"],"created_at":"2025-04-24T00:13:55.157Z","updated_at":"2025-04-24T00:13:55.777Z","avatar_url":"https://github.com/shrunbr.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Graylog Barnyard2 Log Parsing\n\nInspired by [devopstales pfsense parser](https://github.com/devopstales/pfsense-graylog).\n\nUsing this guide we are able to take logs generated from Snort Barnyard2 (within pfSense) and parse them in Graylog to be able to use the information to pipe into Grafana.\n\n## Prerequisites\n\n1. pfSense with Snort running\n2. Graylog (Version 3.2.0+)\n3. Grafana (Optional, but recommended, see Grafana section for requirements)\n \nIf you don't have those 3 running, you'll need to get them setup in your environment before continuing.\n\nNote: *If you're running a separate elasticsearch backend other than what Graylog uses you cannot run the OSS version with this.*\n\n## Graylog Pre-Configuration\n\nI call this the \"pre-configuration\" because it's what we need to do before we get into the real meat and potatoes of this. Follow the steps below to get Graylog ready to parse logs from Snort within pfSense.\n\n1. Create a new index set with the settings below\n\n\t![Image of Barnyard2 index set](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/barnyard2_index_config.PNG)\n\n2. Download the `snort_barnyard2_graylog_content_pack.json` from this repository and go to **System -\u003e Content Packs** click \"Upload\" in the top right and upload the JSON file.\n\n\u003e This content pack will create the inputs, streams, pipelines, pipeline rules, lookup tables, lookup caches and lookup tables needed to properly parse the needed logs.\n3. Go to **Streams** and edit the `Snort Barnyard2 Logs` stream, check `Remove matches from ‘All messages’ stream` and set the index set to the one you just created. We check `Remove matches from ‘All messages’ stream` so that we don't store messages twice and we set the index set so it works with our pipelines.\n\nAwesome, Graylog is now \"pre-configured\" for what we need to do. Lets move onto the next section.\n\n## GeoLite2 DB Installation\n\n\nNow that you have the content pack installed to fully utilize it and get IP Geo-location you'll need to download the MaxMind GeoLite2 Database (MMDB format) and place the file on your Graylog server.\n1. Go to [MaxMind](https://dev.maxmind.com/geoip/geoip2/geolite2/) and click **Sign Up For GeoLite2** at the bottom.\n2. Create an account with MaxMind and sign in\n3. Once you're signed in, click \"Download Databases\"\n\n![Screenshot of Maxmind download](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/maxmind_download_databases.PNG)\n\n4. Click **Download GZIP** next to **GeoLite2 City** (DO NOT DOWNLOAD THE CSV FORMAT)\n\n![Screenshot of GeoLite2 City download](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/maxmind_geolite2_download.PNG)\n\n5. Extract the zip file and place the `GeoLite2-City.mmdb` file in **/etc/graylog/server/** on your graylog server.\n\t\u003e Note: You may need to chmod `GeoLite2-City.mmdb` to 744 and chown the file to whoever owns the rest of the files in that directory for your install\n6. In Graylog go to **System -\u003e Configurations** and click **Update** under **Geo-Location Processor**\n7. Set `/etc/graylog/server/GeoLite2-City.mmdb` as the path and choose **City Database** as the type and click **Save**\n8. Scroll to the top and click **Update** under **Message Processors Configuration** and change the order to what is below\n\n![Screenshot of message processors configuration](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/graylog_message_processors_configuration.PNG)\n\n\n## Elasticsearch Configuration\n\nUnderneath the hood of Graylog runs Elasticsearch. Elasticsearch is what is storing our logs in \"indexes\". We need to use a tool called [Cerebro](https://github.com/lmenezes/cerebro) to modify our `Barnyard2 Logs` index so that it templates the coordinates properly.\n\n You'll need to download Cerebro and be able to run it from a Linux box. I personally run this from my Graylog server when needed. So go to the Cerebro link above and `git clone` the repository to your home directory.\n\nNow that we have the repository cloned we're going to go into `cerebro-*/bin` folder and run Cerebro. I launch it using a few custom variables to not allow it to run on its default port of 9000. Use the command below to run Cerebro. \n\n` ./cerebro -Dhttp.port=9091 -Dhttp.address=X.X.X.X`\n\nChange X.X.X.X to the primary IP of the server you're running Cerebro on. \n\nOnce you have Cerebro running navigate to the web interface in your browser by going to https://X.X.X.X:9091 and target your graylog server IP and port for Elasticsearch (default 9200). `Ex. http://10.1.1.1:9200` and then click \"Connect\".\n\nNow that you're in Cerebro we need to create an index template. Go to `More -\u003e Index Templates` and on the right-hand side you can Create a New Template. We're going to call this template `Barnyard2-Custom` and then copy and paste the contents of `elasticsearch_custom_template.json` into the Template section. Once you've done that click **Create** at the bottom. \n\nNow that we've created the template we need to stop the Graylog service by running `systemctl stop graylog-server` on your Graylog server. Once that is stopped we need to delete the `barnyard_0` index visible under **Overview** in Cerebro.\n\n![Image of sub-menu on barnyard0 index](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/cerebro_delete_barnyard_0.PNG)\n\nNow that it is deleted we can start graylog-server again using `systemctl start graylog-server`. \n\n## Snort Configuration\n\nOkay, we have Graylog completely configured. The last step is to now pipe logs from Snort into Graylog. Follow the steps below to complete this.\n\n1. Login to pfSense and go to **Services -\u003e Snort**\n2. Edit the interface you want to get logs from (most likely your WAN interface)\n3. Navigate to **WAN Barnyard2**\n4. Check the top box `Enable barnyard2 for this interface. You will also need to enable at least one logging destination below.`\n5. Check `Enable logging of alerts to a local or remote syslog receiver.` under `Syslog Output Settings`\n6. Set the remote host to your Graylog server IP and set the port to 10001 (Barnyard2 Graylog Input Port)\n7. Click `Save` at the bottom\n\n    ![Screenshot of Snort Syslog Config](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/snort_pfsense_logging_configuration.PNG)\n\n## Confirm Logging\n\nWe now need to confirm that Graylog is receiving all the logs for Snort. We can do this by going to **Streams -\u003e Snort Barnyard2 Logs** and making sure we're receiving messages. If you click into a message you should see variables such as `src_addr`, `src_addr_geo_location`, `dst_addr`, `dst_addr_geo_location`, etc. \n\n![Screenshot of Graylog Message Example](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/graylog_snort_message_example.PNG)\n\n## Grafana Configuration\n\nNow that we have logs within Graylog for Snort and we're receiving Geo Location coordinates we can map those coordinates to World Maps within Grafana. To complete this section you'll need:\n1. Grafana already running in your environment\n2. An understanding of how Grafana Panels work\n3. World Map Grafana Plugin\n\nFirst things first, we need to add Graylog as a source to Grafana. We can do this by adding a new Elasticsearch data source and configuring it like the image below. In the `URL` box put https://X.X.X.X:9200 (replace X.X.X.X with the IP of your Graylog/Elasticsearch server).\n\n![Screenshot of Grafana Datasource Config](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/grafana_elasticsearch_datasource.PNG)\n\nNow that we have our data source we can import the `snort_grafana_dashboard.json` file in this repository to Grafana. This will give you a very basic starting dashboard for Snort that shows an Incoming connection map, top city, top country, top source ip, top classification, top attack and top destination port. \n\nTo import the dashboard:\n1. Go to **Dashboards -\u003e Manage**\n2. Click **Import** in the top-ish right\n3. Click **Upload .JSON** and select the JSON file you downloaded from the repo\n4. Click **Load**\n\nYou have now uploaded the dashboard but you'll need to edit each panel to target the newly created Elasticsearch data source. Once you've changed the data source for each panel you should be off to the races!\n\n![Screenshot of Grafana Snort Dashboard](https://github.com/shrunbr/graylog_pfsense_barnyard2/blob/master/screenshots/grafana_snort_dashboard.PNG)\n\nEnjoy your new parsed Snort logs and Grafana dashboard!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshrunbr%2Fgraylog_pfsense_barnyard2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshrunbr%2Fgraylog_pfsense_barnyard2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshrunbr%2Fgraylog_pfsense_barnyard2/lists"}