{"id":13846073,"url":"https://github.com/H21lab/tsharkVM","last_synced_at":"2025-07-12T04:30:52.203Z","repository":{"id":49815975,"uuid":"267039368","full_name":"H21lab/tsharkVM","owner":"H21lab","description":"tshark + ELK analytics virtual machine","archived":false,"fork":false,"pushed_at":"2021-12-29T15:03:52.000Z","size":1323,"stargazers_count":66,"open_issues_count":0,"forks_count":18,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-08-05T17:45:39.067Z","etag":null,"topics":["analytics","elasticsearch","elk","kibana","logstash","ndjson","nids","protocols","tshark","tshark-vm","vagrant","virtual-machine","vm","wireshark"],"latest_commit_sha":null,"homepage":"https://www.h21lab.com/tools/tshark-elasticsearch","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/H21lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-26T12:40:52.000Z","updated_at":"2024-07-22T03:42:07.000Z","dependencies_parsed_at":"2022-09-15T18:20:35.644Z","dependency_job_id":null,"html_url":"https://github.com/H21lab/tsharkVM","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FtsharkVM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FtsharkVM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FtsharkVM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FtsharkVM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/H21lab","download_url":"https://codeload.github.com/H21lab/tsharkVM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225791533,"owners_count":17524803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","elasticsearch","elk","kibana","logstash","ndjson","nids","protocols","tshark","tshark-vm","vagrant","virtual-machine","vm","wireshark"],"created_at":"2024-08-04T17:04:19.442Z","updated_at":"2025-07-12T04:30:52.189Z","avatar_url":"https://github.com/H21lab.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# tshark ELK VM appliance\n\nThis project builds virtual machine which can be used for analytics of tshark -T ek (ndjson) output.\nThe virtual appliance is built using vagrant, which builds Debian with pre-installed and pre-configured ELK stack. \n\nAfter the VM is up, the process is simple:\n* decoded pcaps (`tshark -T ek output` / ndjson) are sent over `TCP/17570` to the VM\n* ELK stack in VM will process and index the data\n* Kibana is running in VM and can be accessed on `http://127.0.0.1:15601/app/kibana#/dashboards`\n\n## Instuctions to build VM from Ubuntu desktop\n### Clone source code\n```bash\ngit clone https://github.com/H21lab/tsharkVM.git\n```\n\n### Build tshark VM\n```bash\nsudo apt update\nsudo apt install tshark virtualbox vagrant\nvagrant plugin install vagrant-disksize\nvagrant plugin install vagrant-scp\nbash ./build.sh\n```\n\n### Upload pcaps to VM\n```bash\n# copy your pcaps into ./Trace\n\n# upload the pcaps (with filenames)\nbash upload_pcaps_with_filenames.sh\n\n# or use vagrant scp to copy the ndjson files into /home/vagrant/input\n\n# or upload the pcaps (without filenames)\nbash upload_pcaps.sh\n\n# or use tshark directly towards 127.0.0.1 17570/tcp\ntshark -r trace.pcapng -x -T ek \u003e /dev/tcp/localhost/17570\n\n```\n\n### Open Kibana with browser\n```bash\nfirefox http://127.0.0.1:15601/app/kibana#/dashboards\n```\nOpen Main Dashboard and increase time window to e.g. last 100 years to see there the sample pcaps.\n\n![](res/tshark_vm_dashboard.png?raw=true \"Kibana Dashboard\")\n![](res/tshark_vm_discover.png?raw=true \"Kibana Discover\")\n\n### SSH to VM\n```bash\ncd ./VM\nvagrant ssh\n```\n\n### Delete VM\n```bash\ncd ./VM\nvagrant destroy default\n```\n\n### Start VM\n```bash\ncd ./VM\nvagrant up\n```\n\n### Stop VM\n```bash\ncd ./VM\nvagrant halt\n```\n\n### SSH into VM and check if ELK is running correctly\n```bash\ncd ./VM\nvagrant ssh\nsudo systemctl status kibana.service\nsudo systemctl status elasticsearch.service\nsudo systemctl status logstash.service\n```\n\n# Elasticsearch mapping template\nIn the project is included simple Elasticsearch mapping template generated for the ``frame,eth,ip,udp,tcp,dhcp`` protocols.\nTo handle additional protocols efficiently it can be required to update the mapping template in the following way:\n\n```\n# 1. Create custom mapping, by selecting required protocols\ntshark -G elastic-mapping --elastic-mapping-filter frame,eth,ip,udp,tcp,dns \u003e ./Kibana/custom_tshark_mapping.json\n\n# 2. Deduplicate and post-process the mapping to fit current Elasticsearch version\nruby ./Public/process_tshark_mapping_json.rb\n\n# 3. Upload file to vagrant VM\ncd VM\nvagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.json\ncd ..\n\n# 4. Connect to VM and upload template in the Elasticsearch\ncd VM\nvagrant ssh\ncd tsharkVM/Kibana\ncurl -X PUT \"localhost:9200/_index_template/packets_template\" -H 'Content-Type: application/json' -d@custom_tshark_mapping_deduplicated.json\n```\n\nAlternative can be using the dynamic mapping. See template ``./Kibana/template_tshark_mapping_dynamic.json``. And consider setting the numeric_detection parameter true/false depending on the mapping requirements and pcaps used. Upload the template into Elasticsearch in similar way as described above.\n\n## Limitations\ntshark -G elastic-mapping --elastic-mapping-filter mapping could be outdated, it is not following properly the Elasticsearch changes and the output can be duplicated. The manual configuration and post-processing of the mapping template is required.\n\nProgram is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY.\n\n## License\nThe default license of source codes provided inside this project is the Apache License v2.0. \n\nsimple-NIDS is licensed under the AGPLv3 (Free Open Source GNU Affero GPL v3.0).\n\nAdditionally refer to individual licenses and terms of used of installed software (see licenses for Wireshark, Elastic and other). \n\n## Attribution\nSpecial thanks to people who helped with the Wireshark development or otherwise contributed to this work:\n* Anders Broman\n* [Alexis La Goutte](https://twitter.com/alagoutte)\n* Christoph Wurm \n* [Dario Lombardo](https://twitter.com/crondaemon1)\n* [Vic Hargrave](https://twitter.com/vichargrave)\n\nExample pcap in ./Traces subfolder was downloaded from https://wiki.wireshark.org/SampleCaptures\n\nCreated by Martin Kacer\n\nCopyright 2021 H21 lab, All right reserved, https://www.h21lab.com\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FH21lab%2FtsharkVM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FH21lab%2FtsharkVM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FH21lab%2FtsharkVM/lists"}