{"id":15780015,"url":"https://github.com/rgl/beats-windows-vagrant","last_synced_at":"2026-05-07T06:32:52.149Z","repository":{"id":139750296,"uuid":"163690639","full_name":"rgl/beats-windows-vagrant","owner":"rgl","description":"Elastic Beats playground","archived":false,"fork":false,"pushed_at":"2019-04-26T11:05:59.000Z","size":34,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-31T11:05:56.979Z","etag":null,"topics":["beats","elasticsearch","vagrant","windows"],"latest_commit_sha":null,"homepage":"","language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rgl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-31T18:42:39.000Z","updated_at":"2020-04-17T18:23:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"1eae4d17-541c-435b-b8c8-7264416a290c","html_url":"https://github.com/rgl/beats-windows-vagrant","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rgl/beats-windows-vagrant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fbeats-windows-vagrant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fbeats-windows-vagrant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fbeats-windows-vagrant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fbeats-windows-vagrant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rgl","download_url":"https://codeload.github.com/rgl/beats-windows-vagrant/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fbeats-windows-vagrant/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261789274,"owners_count":23209775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beats","elasticsearch","vagrant","windows"],"created_at":"2024-10-04T18:40:21.642Z","updated_at":"2026-05-07T06:32:47.113Z","avatar_url":"https://github.com/rgl.png","language":"PowerShell","funding_links":[],"categories":[],"sub_categories":[],"readme":"this is a [Beats](https://www.elastic.co/products/beats) playground\n\n# Usage\n\n[Build and install the Windows 2019 base image](https://github.com/rgl/windows-2016-vagrant).\n\nLaunch the `beats` machine:\n\n```bash\nvagrant up beats --provider=virtualbox # or --provider=libvirt\n```\n\nLogon at the Windows console.\n\nElasticsearch is available at:\n\n  [https://elasticsearch.example.com](https://elasticsearch.example.com)\n\nKibana is available at:\n\n  [https://kibana.example.com](https://kibana.example.com)\n\n# Filebeat\n\nFilebeat creates the following Elasticsearch artifacts:\n\n| type     | name           | url                                                                                        | notes                                        |\n|----------|----------------|--------------------------------------------------------------------------------------------|----------------------------------------------|\n| template | filebeat-7.0.0 | [_template/filebeat-7.0.0](https://elasticsearch.example.com/_template/filebeat-7.0.0) | contains all the modules definitions |\n| pipeline | filebeat-7.0.0-iis-error-default | [_ingest/pipeline/filebeat-7.0.0-iis-error-default](https://elasticsearch.example.com/_ingest/pipeline/filebeat-7.0.0-iis-error-default) | IIS HTTPERR log ingest pipeline |\n| pipeline | filebeat-7.0.0-iis-access-default | [_ingest/pipeline/filebeat-7.0.0-iis-access-default](https://elasticsearch.example.com/_ingest/pipeline/filebeat-7.0.0-iis-access-default) | IIS access log ingest pipeline |\n\nFilebeat will tail all the log files for log messages, batch them, then send them in [Elasticsearch bulk requests](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docs-bulk.html) to a index with the pattern `filebeat-%{[beat.version]}-%{+yyyy.MM.dd}` (e.g. `filebeat-7.0.0-2018.12.30`).\n\n## IIS logs\n\nIIS logs are normally stored at:\n\n* `C:\\inetpub\\logs\\LogFiles\\*\\*.log`\n* `C:\\Windows\\System32\\LogFiles\\HTTPERR\\*.log`\n\nThe IIS logs are buffered for some time (one minute by default) before being flushed/written to disk. You can manually force a flush with the command `netsh http flush logbuffer` (flushes the HTTP.sys log buffer to disk).\n\nThe IIS access logs can be customized by selecting which fields will be logged, **BUT, by default, filebeat only supports five configurations**. The five supported configurations are defined as [elasticsearch grok expressions](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/grok-processor.html) inside the `C:\\filebeat\\module\\iis\\access\\ingest\\default.json` file (or the [online version source code](https://github.com/elastic/beats/blob/v7.0.0/filebeat/module/iis/access/ingest/default.json)).\n\nThe main ones are:\n\n| #1 | #2 | #3 | IIS field       | filebeat field            | grok expression                          |\n|----|----|----|-----------------|---------------------------|------------------------------------------|\n| X  | X  | X  | date time       | @timestamp                | %{TIMESTAMP_ISO8601:iis.access.time}     |\n|    | X  | X  | s-sitename      | iis.access.site_name      | %{NOTSPACE:iis.access.site_name}         |\n|    |    | X  | s-computername  | iis.access.server_name    | %{NOTSPACE:iis.access.server_name}       |\n| X  |    | X  | s-ip            | destination.address       | %{IPORHOST:destination.address}          |\n| X  | X  | X  | cs-method       | http.request.method       | %{WORD:http.request.method}              |\n| X  | X  | X  | cs-uri-stem     | url.path                  | %{URIPATH:url.path}                      |\n| X  | X  | X  | cs-uri-query    | url.query                 | %{NOTSPACE:url.query}                    |\n| X  | X  | X  | s-port          | destination.port          | %{NUMBER:destination.port:long}          |\n| X  | X  | X  | cs-username     | user.name                 | %{NOTSPACE:user.name}                    |\n| X  | X  | X  | c-ip            | source.address            | %{IPORHOST:source.address}               |\n|    |    | X  | cs-version      | http.version              | HTTP/%{NUMBER:http.version}              |\n| X  | X  | X  | cs(User-Agent)  | user_agent.original       | %{NOTSPACE:user_agent.original}          |\n|    | X  | X  | cs(Cookie)      | iis.access.cookie         | %{NOTSPACE:iis.access.cookie}            |\n| X  | X  | X  | cs(Referer)     | http.request.referrer     | %{NOTSPACE:http.request.referrer}        |\n|    | X  | X  | cs-host         | destination.domain        | %{NOTSPACE:destination.domain}           |\n| X  | X  | X  | sc-status       | http.response.status_code | %{NUMBER:http.response.status_code:long} |\n| X  | X  | X  | sc-substatus    | iis.access.sub_status     | %{NUMBER:iis.access.sub_status:long}     |\n| X  | X  | X  | sc-win32-status | iis.access.win32_status   | %{NUMBER:iis.access.win32_status:long}   |\n|    | X  | X  | sc-bytes        | http.response.body.bytes  | %{NUMBER:http.response.body.bytes:long}  |\n|    | X  | X  | cs-bytes        | http.request.body.bytes   | %{NUMBER:http.request.body.bytes:long}   |\n| X  | X  | X  | time-taken      | event.duration            | %{NUMBER:temp.duration:long}             |\n\n* **NB** the #1 configuration corresponds to the default IIS logging configuration.\n* **NB** the #2 configuration corresponds to the default Azure IIS logging configuration.\n* **NB** the #3 configuration corresponds to all the available fields in IIS (as-of IIS 10 that ships with Windows 2019), and this is the one we configure and use in this vagrant environment (see [provision-iis.ps1](provision-iis.ps1)).\n\n### IIS access log example\n\nThe following example shows how filebeat sends a IIS access log (**using configuration type #3**) line to Elasticsearch and how its transformed by the pipeline.\n\nFilebeat sends/receives the following bulk request/response:\n\n**NB** these were captured with fiddler by setting `proxy_url` in `filebeat.yml` and accessing `http://localhost/four-oh-four`.\n\n```json\nPOST /_bulk HTTP/1.1\nHost: elasticsearch.example.com:443\nUser-Agent: Go-http-client/1.1\nContent-Length: 123\nAccept: application/json\nAccept-Encoding: gzip\nContent-Type: application/json; charset=UTF-8\n\n{\n    \"index\": {\n        \"_index\": \"filebeat-7.0.0-2019.04.26\",\n        \"pipeline\": \"filebeat-7.0.0-iis-access-default\"\n    }\n}\n{\n    \"@timestamp\": \"2019-04-26T09:50:43.361Z\",\n    \"message\": \"2019-04-26 09:50:34 W3SVC1 beats 127.0.0.1 GET /four-oh-four - 80 - 127.0.0.1 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.108+Safari/537.36 - - localhost 404 0 2 5044 427 1970\",\n    \"service\": {\n        \"type\": \"iis\"\n    },\n    \"event\": {\n        \"module\": \"iis\",\n        \"dataset\": \"iis.access\"\n    },\n    \"host\": {\n        \"name\": \"beats\"\n    },\n    \"agent\": {\n        \"type\": \"filebeat\",\n        \"ephemeral_id\": \"242e7b2e-be7f-434b-9ed0-200b225306cf\",\n        \"hostname\": \"beats\",\n        \"id\": \"5cf266e8-b541-49b6-8ee5-f18d6c7f55c2\",\n        \"version\": \"7.0.0\"\n    },\n    \"log\": {\n        \"offset\": 335,\n        \"file\": {\n            \"path\": \"C:\\\\inetpub\\\\logs\\\\LogFiles\\\\W3SVC1\\\\u_ex190426.log\"\n        }\n    },\n    \"input\": {\n        \"type\": \"log\"\n    },\n    \"fileset\": {\n        \"name\": \"access\"\n    },\n    \"ecs\": {\n        \"version\": \"1.0.0\"\n    }\n}\n\nHTTP/1.1 200 OK\ncontent-type: application/json; charset=UTF-8\ncontent-encoding: gzip\ncontent-length: 123\n\n{\n    \"took\": 14,\n    \"ingest_took\": 5,\n    \"errors\": false,\n    \"items\": [{\n        \"index\": {\n            \"_index\": \"filebeat-7.0.0-2019.04.26\",\n            \"_type\": \"_doc\",\n            \"_id\": \"ZpMNWWoBQLxjzhVU5ryQ\",\n            \"_version\": 1,\n            \"result\": \"created\",\n            \"_shards\": {\n                \"total\": 1,\n                \"successful\": 1,\n                \"failed\": 0\n            },\n            \"_seq_no\": 93,\n            \"_primary_term\": 1,\n            \"status\": 201\n        }\n    }]\n}\n```\n\nWhich, after being transformed by the filebeat elasticsearch ingest pipeline, will be stored in elasticsearch as this request/response shows:\n\n```json\nGET /filebeat-7.0.0-2019.04.26/_doc/ZpMNWWoBQLxjzhVU5ryQ HTTP/1.1\nHost: localhost:9200\nConnection: close\n\nHTTP/1.1 200 OK\ncontent-type: application/json; charset=UTF-8\ncontent-length: 123\n\n{\n    \"_index\": \"filebeat-7.0.0-2019.04.26\",\n    \"_type\": \"_doc\",\n    \"_id\": \"ZpMNWWoBQLxjzhVU5ryQ\",\n    \"_version\": 1,\n    \"_seq_no\": 93,\n    \"_primary_term\": 1,\n    \"found\": true,\n    \"_source\": {\n        \"agent\": {\n            \"hostname\": \"beats\",\n            \"id\": \"5cf266e8-b541-49b6-8ee5-f18d6c7f55c2\",\n            \"type\": \"filebeat\",\n            \"ephemeral_id\": \"242e7b2e-be7f-434b-9ed0-200b225306cf\",\n            \"version\": \"7.0.0\"\n        },\n        \"temp\": {},\n        \"log\": {\n            \"file\": {\n                \"path\": \"C:\\\\inetpub\\\\logs\\\\LogFiles\\\\W3SVC1\\\\u_ex190426.log\"\n            },\n            \"offset\": 335\n        },\n        \"destination\": {\n            \"address\": \"127.0.0.1\",\n            \"port\": 80,\n            \"domain\": \"localhost\",\n            \"ip\": \"127.0.0.1\"\n        },\n        \"source\": {\n            \"address\": \"127.0.0.1\",\n            \"ip\": \"127.0.0.1\"\n        },\n        \"fileset\": {\n            \"name\": \"access\"\n        },\n        \"url\": {\n            \"path\": \"/four-oh-four\",\n            \"query\": \"-\"\n        },\n        \"input\": {\n            \"type\": \"log\"\n        },\n        \"iis\": {\n            \"access\": {\n                \"site_name\": \"W3SVC1\",\n                \"server_name\": \"beats\",\n                \"cookie\": \"-\",\n                \"sub_status\": 0,\n                \"win32_status\": 2\n            }\n        },\n        \"@timestamp\": \"2019-04-26T09:50:34.000Z\",\n        \"ecs\": {\n            \"version\": \"1.0.0\"\n        },\n        \"service\": {\n            \"type\": \"iis\"\n        },\n        \"host\": {\n            \"name\": \"beats\"\n        },\n        \"http\": {\n            \"request\": {\n                \"referrer\": \"-\",\n                \"method\": \"GET\",\n                \"body\": {\n                    \"bytes\": 427\n                }\n            },\n            \"response\": {\n                \"status_code\": 404,\n                \"body\": {\n                    \"bytes\": 5044\n                }\n            },\n            \"version\": \"1.1\"\n        },\n        \"event\": {\n            \"duration\": 1970000000,\n            \"created\": \"2019-04-26T09:50:43.361Z\",\n            \"module\": \"iis\",\n            \"dataset\": \"iis.access\"\n        },\n        \"user\": {\n            \"name\": \"-\"\n        },\n        \"user_agent\": {\n            \"original\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36\",\n            \"os\": {\n                \"name\": \"Windows 10\"\n            },\n            \"name\": \"Chrome\",\n            \"device\": {\n                \"name\": \"Other\"\n            },\n            \"version\": \"74.0.3729\"\n        }\n    }\n}\n```\n\nThe pipeline (the most import part is the [grok processor](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/grok-processor.html) block) is defined as (see the `C:\\filebeat\\module\\iis\\access\\ingest\\default.json` file or the [online version source code](https://github.com/elastic/beats/blob/v7.0.0/filebeat/module/iis/access/ingest/default.json)):\n\n```json\nGET _ingest/pipeline/filebeat-7.0.0-iis-access-default\n\n{\n    \"filebeat-7.0.0-iis-access-default\": {\n        \"processors\": [{\n            \"grok\": {\n                \"field\": \"message\",\n                \"patterns\": [\n                    \"%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:destination.address} %{WORD:http.request.method} %{URIPATHWITHBRACKET:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:http.request.referrer} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:temp.duration:long}\",\n                    \"%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:http.request.referrer} %{NOTSPACE:destination.domain} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:http.response.body.bytes:long} %{NUMBER:http.request.body.bytes:long} %{NUMBER:temp.duration:long}\",\n                    \"%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:destination.address} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} HTTP/%{NUMBER:http.version} %{NOTSPACE:user_agent.original} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:http.request.referrer} %{NOTSPACE:destination.domain} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:http.response.body.bytes:long} %{NUMBER:http.request.body.bytes:long} %{NUMBER:temp.duration:long}\",\n                    \"%{TIMESTAMP_ISO8601:iis.access.time} \\\\[%{IPORHOST:destination.address}\\\\]\\\\(http://%{IPORHOST:destination.address}\\\\) %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} \\\\[%{IPORHOST:source.address}\\\\]\\\\(http://%{IPORHOST:source.address}\\\\) %{NOTSPACE:user_agent.original} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:temp.duration:long}\",\n                    \"%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:destination.address} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:temp.duration:long}\"\n                ],\n                \"pattern_definitions\": {\n                    \"URIPATHWITHBRACKET\": \"(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%\u0026_\\\\-\\\\[\\\\]]*)+\"\n                },\n                \"ignore_missing\": true\n            }\n        }, {\n            \"remove\": {\n                \"field\": \"message\"\n            }\n        }, {\n            \"rename\": {\n                \"field\": \"@timestamp\",\n                \"target_field\": \"event.created\"\n            }\n        }, {\n            \"date\": {\n                \"field\": \"iis.access.time\",\n                \"target_field\": \"@timestamp\",\n                \"formats\": [\"yyyy-MM-dd HH:mm:ss\"]\n            }\n        }, {\n            \"remove\": {\n                \"field\": \"iis.access.time\"\n            }\n        }, {\n            \"script\": {\n                \"params\": {\n                    \"scale\": 1000000.0\n                },\n                \"if\": \"ctx.temp?.duration != null\",\n                \"lang\": \"painless\",\n                \"source\": \"ctx.event.duration = Math.round(ctx.temp.duration * params.scale)\"\n            }\n        }, {\n            \"remove\": {\n                \"field\": \"temp.duration\",\n                \"ignore_missing\": true\n            }\n        }, {\n            \"urldecode\": {\n                \"field\": \"user_agent.original\"\n            }\n        }, {\n            \"user_agent\": {\n                \"field\": \"user_agent.original\"\n            }\n        }, {\n            \"grok\": {\n                \"field\": \"destination.address\",\n                \"ignore_failure\": true,\n                \"patterns\": [\"%{NOZONEIP:destination.ip}\"],\n                \"pattern_definitions\": {\n                    \"NOZONEIP\": \"[^%]*\"\n                }\n            }\n        }, {\n            \"grok\": {\n                \"field\": \"source.address\",\n                \"ignore_failure\": true,\n                \"patterns\": [\"%{NOZONEIP:source.ip}\"],\n                \"pattern_definitions\": {\n                    \"NOZONEIP\": \"[^%]*\"\n                }\n            }\n        }, {\n            \"geoip\": {\n                \"field\": \"source.ip\",\n                \"target_field\": \"source.geo\",\n                \"ignore_missing\": true\n            }\n        }],\n        \"on_failure\": [{\n            \"set\": {\n                \"field\": \"error.message\",\n                \"value\": \"{{ _ingest.on_failure_message }}\"\n            }\n        }],\n        \"description\": \"Pipeline for parsing IIS access logs. Requires the geoip and user_agent plugins.\"\n    }\n}\n```\n\n# Reference\n\n* https://www.elastic.co/guide/en/beats/filebeat/7.0/defining-processors.html\n* [Transition Beats to ECS](https://github.com/elastic/beats/issues/8655)\n* [Elastic Common Schema (ECS)](https://github.com/elastic/ecs)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frgl%2Fbeats-windows-vagrant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frgl%2Fbeats-windows-vagrant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frgl%2Fbeats-windows-vagrant/lists"}