{"id":20827650,"url":"https://github.com/bonifield/logstash-to-splunk","last_synced_at":"2025-05-07T21:04:58.259Z","repository":{"id":212220916,"uuid":"420538020","full_name":"bonifield/logstash-to-splunk","owner":"bonifield","description":"writeup about sending Logstash data to Splunk using the HTTP Event Collector","archived":false,"fork":false,"pushed_at":"2022-02-10T02:13:32.000Z","size":25,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T21:04:52.949Z","etag":null,"topics":["elasticsearch","filebeat","kafka","logstash","metricbeat","splunk","splunk-enterprise","splunk-hec","splunk-universal-forwarder","winlogbeat"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bonifield.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-10-23T22:39:32.000Z","updated_at":"2024-10-14T06:35:09.000Z","dependencies_parsed_at":"2023-12-13T06:23:41.273Z","dependency_job_id":"e2e49ae0-931c-4f3a-a47d-04dd531670e1","html_url":"https://github.com/bonifield/logstash-to-splunk","commit_stats":null,"previous_names":["bonifield/logstash-to-splunk"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonifield%2Flogstash-to-splunk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonifield%2Flogstash-to-splunk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonifield%2Flogstash-to-splunk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonifield%2Flogstash-to-splunk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bonifield","download_url":"https://codeload.github.com/bonifield/logstash-to-splunk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252954432,"owners_count":21830903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","filebeat","kafka","logstash","metricbeat","splunk","splunk-enterprise","splunk-hec","splunk-universal-forwarder","winlogbeat"],"created_at":"2024-11-17T23:12:36.659Z","updated_at":"2025-05-07T21:04:58.237Z","avatar_url":"https://github.com/bonifield.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Logstash to Splunk\nThis is a writeup about sending Logstash data to Splunk using the HTTP Event Collector. Logstash provides an immense filtering capability which can be used to screen and reduce data before it hits the Splunk indexers. Additionally, Logstash can accept a vast array of inputs, and provides a middleware capability when translating Elastic Beats into Splunk logs. Using the Elastic Common Schema inside Splunk works perfectly well, as dotted fields (ex. source.ip) function without issue in the pipeline, and standard field renames (ex. turn periods into underscores, etc) can be applied either in Splunk or Logstash. Further decoupling of services, and increased data durability, can be achieved by using a Kafka cluster between Logstash and Splunk, but that is beyond the scope of this writeup.\n\n# Considerations\n| Issue | Logstash | Splunk |\n| -- | -- | -- |\n| Transformations | LS permanently alters data before reaching Splunk | SP can update values dynamically due to search-time extraction |\n| Infrastructure | LS is not cluster-aware and requires orchestration management | SP can deploy policies directly to the ingest and indexing tiers |\n| Migration | Existing Beats+LS infrastructure still works up until it reaches Splunk | SP cannot manage Logstash, Beats, or other non-Splunk agents directly* |\n\n\\* = Splunk Deployment Servers can manage endpoint software in some circumstances\n\n## 1. (Splunk) Create a new HTTP Event Collector token\n\n[Official Documentation: Configure HTTP Event Collector on Splunk Enterprise](https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/UsetheHTTPEventCollector)\n\nSettings --\u003e Add Data --\u003e Monitor --\u003e HTTP Event Collector\n\n\tName: \"webtoken\" (or any desired name)\n\tUNCHECK: Enable indexer acknowledgement\n\nclick **Next**\n\n*this is where you will decide which index this token is written to upon being received*\n\n\tCreate a new index\n\t\tIndex Name: logstash\n\t\t(leave rest defaults for this demo)\n\t\tSave\n\tSelect Allowed Indexes --\u003e click logstash to \"whitelist\" the index for this token\n\nclick **Review** --\u003e **Submit**\n\n\tcopy the token and save it for later\n\t# note this token was used for this writeup, then deleted\n\t\tc6012558-7817-45e0-a3a5-7dfc876e1bf3\n\n## 2. (Splunk) Enable Tokens Globally\n\nSettings --\u003e Data inputs --\u003e HTTP Event Collector --\u003e Global Settings (top right)\n\n*these default source type settings will not affect which actual source type this token is written to, as we will change that in the following step (note you cannot create a new source type from this panel)*\n\n\t# note HTTP Event Collector settings screen is where the token can be retrieved again if needed\n\tAll Tokens: Enabled\n\tDefault Source Type: Structured --\u003e _json\n\tDefault Index: logstash\n\tUNCHECK: Use Deployment Server\n\tUNCHECK: Enable SSL (will be added later in the instructions)\n\tHTTP Port Number: 8088 (default)\n\n## 3. (Splunk) Create Source Type\n\nSettings --\u003e Source types --\u003e New Source Type (top right)\n\n*it is very important to set Indexed Extractions to \"none\", otherwise Splunk will display duplicated data as it double-parses the JSON*\n\n\tName: winlogbeat\n\tDescription: Windows endpoint data\n\tDestination app: Search \u0026 Reporting\n\tCategory: Operating System\n\tIndexed Extractions: none\n\t# additional configurations should be made to set \"@timestamp\" as the document time field with the appropriate time zone, but that is not critical to this demo as both _time and @timestamp can be utilized for the same purpose\n\n## 4. (Splunk) Set Source Type for Token\n\nSettings --\u003e Data inputs --\u003e HTTP Event Collector --\u003e Edit (on appropriate source type name)\n\n\tSource Type: winlogbeat\n\n## 5. (Logstash) Output Statement\n\n*note the usage of the /services/collector/raw endpoint; the event (vs raw) endpoint is unreliable when sending POST data from Logstash*\n\n\toutput {\n\t\thttp {\n\t\t\tcontent_type =\u003e \"application/json\"\n\t\t\thttp_method =\u003e \"post\"\n\t\t\turl =\u003e \"http://your-splunk-server:8088/services/collector/raw\"\n\t\t\theaders =\u003e [\"Authorization\", \"Splunk c6012558-7817-45e0-a3a5-7dfc876e1bf3\"]\n\t\t}\n\t}\n\n## Considerations\n- use one token per index+sourcetype, and name/describe them accordingly\n- harden and firewall your assets; do not allow the entire world, or even the whole corporate network, to reach the ingest ports\n- certificates don't last forever; include the Splunk keys in your key management processes (and don't forget the Logstash stores)\n- Logstash --\u003e Kafka \u003c-- Splunk: this setup is more durable and reduces data loss if the Splunk cluster goes down for some reason\n\n## TO DO\n- generate SSL/TLS certificates\n- configure the Logstash truststore and keystore\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbonifield%2Flogstash-to-splunk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbonifield%2Flogstash-to-splunk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbonifield%2Flogstash-to-splunk/lists"}