{"id":20261331,"url":"https://github.com/parsiya/extract-sni","last_synced_at":"2025-04-11T01:45:00.907Z","repository":{"id":57571076,"uuid":"262662770","full_name":"parsiya/extract-sni","owner":"parsiya","description":"Extracts SNIs from a pcap and generates output usable in `etc/hosts` file and Burp config for proxying non-proxy-aware thick clients using HTTPs.","archived":false,"fork":false,"pushed_at":"2020-06-23T03:09:47.000Z","size":907,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T22:51:20.614Z","etag":null,"topics":["automation","golang","proxy"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/parsiya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-09T21:33:27.000Z","updated_at":"2024-11-10T11:36:04.000Z","dependencies_parsed_at":"2022-09-13T22:00:43.146Z","dependency_job_id":null,"html_url":"https://github.com/parsiya/extract-sni","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsiya%2Fextract-sni","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsiya%2Fextract-sni/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsiya%2Fextract-sni/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsiya%2Fextract-sni/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/parsiya","download_url":"https://codeload.github.com/parsiya/extract-sni/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248327863,"owners_count":21085258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","golang","proxy"],"created_at":"2024-11-14T11:25:17.526Z","updated_at":"2025-04-11T01:45:00.889Z","avatar_url":"https://github.com/parsiya.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# extract-sni \u003c!-- omit in toc --\u003e\r\nExtracts SNIs from a pcap and generates output usable in `etc/hosts` file and\r\nBurp config for proxying non-proxy-aware thick clients using HTTPs.\r\n\r\n## Quickstart\r\n\r\n1. `go get github.com/parsiya/extract-sni`\r\n2. Capture traffic for an application and store it in a pcap file.\r\n3. `go run extract-sni.go whatever.pcap -output report`\r\n4. Open `report.html` to view the instructions.\r\n    1. Some text should be copied to `etc/hosts` to redirect traffic to Burp.\r\n5. Import `report.json` in Burp as the project config to setup proxy listeners\r\n   and hostname resolution.\r\n6. ???\r\n7. Profit\r\n\r\n## Table of Contents \u003c!-- omit in toc --\u003e\r\n- [Quickstart](#quickstart)\r\n- [No, Really? How do I Use This?](#no-really-how-do-i-use-this)\r\n    - [Why?](#why)\r\n- [Pre-requisites](#pre-requisites)\r\n    - [npcap Must be Installed](#npcap-must-be-installed)\r\n- [Usage](#usage)\r\n    - [pcap file](#pcap-file)\r\n    - [DNS -d/-dns](#dns--d-dns)\r\n    - [Redirect IP -r/-redirectip](#redirect-ip--r-redirectip)\r\n    - [Output -o/-output](#output--o-output)\r\n- [Limitations](#limitations)\r\n- [Questions](#questions)\r\n- [Troubleshooting](#troubleshooting)\r\n    - [My Output File is Noisy](#my-output-file-is-noisy)\r\n    - [Error `Couldn't load wpcap.dll`](#error-couldnt-load-wpcapdll)\r\n- [Samples](#samples)\r\n- [License](#license)\r\n\r\n## No, Really? How do I Use This?\r\nYou have a non-proxy-aware thickclient that uses HTTPs. This means it does not\r\nlike the Windows proxy settings or any other proxy settings. It does not\r\n[send the CONNECT request][proxy-aware].\r\n\r\n[proxy-aware]: https://parsiya.net/blog/2016-07-28-thick-client-proxying-part-6-how-https-proxies-work/#5-proxy-aware-clients\r\n\r\nRead these blog posts to get started:\r\n\r\n* https://parsiya.net/blog/2020-06-22-thick-client-proxying-part-11-gog-galaxy-and-extract-sni/\r\n* https://parsiya.net/blog/2020-05-09-thick-client-proxing-part-10-the-hosts-file/\r\n\r\nIf the application uses HTTPs then we can process SNI\r\n\r\n### Why?\r\nBecause [Manual Work is a Bug][manual-link]. This program automates some of the\r\nprocess.\r\n\r\n[manual-link]: https://queue.acm.org/detail.cfm?id=3197520\r\n\r\n## Pre-requisites\r\nYou need Burp. **Both the pro and community versions work**. Other proxy tools\r\n(Fiddler, ZAP, etc.) might work with modification but the config file is created\r\nfor Burp.\r\n\r\n### npcap Must be Installed\r\nOn Windows, you need to install [npcap](https://nmap.org/npcap/#download) for\r\nthe Golang's pcap library to work. Be sure to check\r\n`Install Npcap in WinPcap API-compatible Mode` in the installer\r\n(it's enabled by default).\r\n\r\nI have not tested this tool on other operating systems.\r\n\r\n## Usage\r\nThe only required parameter is the pcap file. Running `extract-sni traffic.pcap`\r\nwill create a report named `traffic.html` and a Burp config file name\r\n`traffic.json` in the same path as the pcap file.\r\n\r\nUse `-h` or `--help` to see the complete usage.\r\n\r\n### pcap file\r\nPass the pcap file. This is a positional and the only required parameter.\r\n\r\n* `extract-sni.go whatever.pcap`\r\n\r\n### DNS -d/-dns\r\nOptional DNS for domain lookup. If this parameter is not provided then the\r\napplication uses the destination IP address from the pcap file for each domain.\r\n\r\nIf provided, the DNS server will be used to do lookups. The value can be a\r\ncomplete `server:port` like `dns.google:53`, or `IP:port` like `8.8.8.8:53`.\r\nPort is optional and will default to `53`. Both `8.8.8.8` and\r\n`dns.google` are both valid values.\r\n\r\nThe application does some validation checks here but it's mostly the\r\nresponsibility of the user to pass a valid and reachable DNS server.\r\n\r\n### Redirect IP -r/-redirectip\r\nUse the IP address that you want the traffic to be redirected to. This is used\r\nin two places:\r\n\r\n1. The `hosts` file.\r\n2. The Burp proxy listener will listen on this IP address.\r\n\r\nFor most uses cases this parameter does not need to be provided and the default\r\nvalue `127.0.0.1` is good enough.\r\n\r\nNote: Only use IP addresses here, not domains like `example.net` or `localhost`.\r\n\r\n### Output -o/-output\r\nThe path of the report. The report contains the instructions on how to set up.\r\nThe Burp config file will appear in the same path but with the `json` extension.\r\n\r\nAny extension here will be ignored. For example, `-o report.txt` will result in two\r\nfiles:\r\n\r\n1. `report.html`: Report with instructions.\r\n2. `report.json`: Burp config file.\r\n\r\nIf this parameter is not used, the name of the pcap file is used. For example\r\n`extract-sni app-capture.pcap` creates: `app-capture.html` and\r\n`app-capture.json` in the same path as the pcap file.\r\n\r\n## Limitations\r\nThe tools does not do everything.\r\n\r\n1. You need to create the pcap. Use Wireshark or Netmon.\r\n2. You need to filter the noisy traffic. In Wireshark, you can filter the\r\n   traffic and in Netmon you can capture traffic for specific processes.\r\n3. You need to bypass any certificate pinning.\r\n    1. Usually, [adding Burp's CA to the Windows certificate store][burp-cert]\r\n       is enough.\r\n4. The config is for Burp. You could probably replicate it for other proxies but\r\n   I only use Burp.\r\n5. If there is no SNI header in the `ClientHello` the tool cannot determine the\r\n   endpoint. This is usually the case when the application uses an IP address to\r\n   target the endpoint.\r\n    1. Similarly, the `hosts` file cannot redirect IP addresses, only domains.\r\n\r\n[burp-cert]: https://parsiya.net/blog/2016-02-21-installing-burp-certificate-authority-in-windows-certificate-store/\r\n\r\n## Questions\r\n\r\n1. Why doesn't it capture pcaps, too?\r\n    1. It's a small tool that does one thing. Feature creep is bad.\r\n    2. It's easier to filter the traffic and reduce noise with Wireshark or\r\n       Netmon.\r\n\r\n## Troubleshooting\r\n\r\n### My Output File is Noisy\r\nYour pcap is noisy. Try to filter as much unrelated traffic as you can. I use\r\nthe techniques described in `Network Traffic Attribution on Windows`:\r\n\r\n* https://parsiya.net/blog/2015-08-01-network-traffic-attribution-on-windows/\r\n\r\nI usually use [Microsoft Network Monitor][netmon] or Netmon. With Netmon you can\r\nfilter traffic by process. However, this adds an extra step. Netmon's cap file\r\nmust be converted to pcap using Wireshark (or other tools). Keep in mind that\r\nsometimes Wireshark cannot [convert cap files to pcap][cap-to-pcap].\r\n\r\n[netmon]: https://www.microsoft.com/en-ca/download/details.aspx?id=4865\r\n[cap-to-pcap]: https://parsiya.net/cheatsheet/#open-a-network-monitor-cap-file-in-wireshark-and-save-is-disabled\r\n\r\n### Error `Couldn't load wpcap.dll`\r\n`npcap` is not installed. See the [npcap](#npcap) section above for more info.\r\n\r\n## Samples\r\nSee a report sample:\r\n\r\n* [Report](report-sample/file.html)\r\n* [Burp config](report-sample/file.json)\r\n\r\nThe screenshot from a report:\r\n\r\n![report screenshot](.github/screenshot.png)\r\n\r\n## License\r\nOpensourced under the MIT license. See the [LICENSE](LICENSE) file for details.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparsiya%2Fextract-sni","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparsiya%2Fextract-sni","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparsiya%2Fextract-sni/lists"}