{"id":13454478,"url":"https://github.com/tomnomnom/meg","last_synced_at":"2025-05-14T18:01:44.576Z","repository":{"id":41190304,"uuid":"75330190","full_name":"tomnomnom/meg","owner":"tomnomnom","description":"Fetch many paths for many hosts - without killing the hosts","archived":false,"fork":false,"pushed_at":"2024-02-03T20:03:53.000Z","size":68,"stargazers_count":1643,"open_issues_count":41,"forks_count":270,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-04-13T10:04:32.634Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomnomnom.png","metadata":{"files":{"readme":"README.mkd","changelog":null,"contributing":"CONTRIBUTING.mkd","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-01T20:30:57.000Z","updated_at":"2025-04-11T14:08:34.000Z","dependencies_parsed_at":"2024-06-18T19:27:03.995Z","dependency_job_id":null,"html_url":"https://github.com/tomnomnom/meg","commit_stats":{"total_commits":78,"total_committers":8,"mean_commits":9.75,"dds":0.08974358974358976,"last_synced_commit":"5cc1d344ce2c29f8c88f183dd70072a17f3d759a"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fmeg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fmeg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fmeg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fmeg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomnomnom","download_url":"https://codeload.github.com/tomnomnom/meg/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254198452,"owners_count":22030964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:00:54.506Z","updated_at":"2025-05-14T18:01:44.536Z","avatar_url":"https://github.com/tomnomnom.png","language":"Go","readme":"# meg\n\nmeg is a tool for fetching lots of URLs but still being 'nice' to servers.\n\nIt can be used to fetch many paths for many hosts; fetching one path\nfor all hosts before moving on to the next path and repeating.\n\nYou get lots of results quickly, but non of the individual hosts get\nflooded with traffic.\n\n## Install\n\nmeg is written in Go and has no run-time dependencies. If you have Go 1.9\nor later installed and configured you can install meg with `go install`:\n\n```\n▶ go install github.com/tomnomnom/meg@latest\n```\n\nOr [download a binary](https://github.com/tomnomnom/meg/releases) and\nput it somewhere in your `$PATH` (e.g. in `/usr/bin/`).\n\n### Install Errors\n\nIf you see an error like this it means your version of Go is too old:\n\n```\n# github.com/tomnomnom/rawhttp\n/root/go/src/github.com/tomnomnom/rawhttp/request.go:102: u.Hostname undefined (\ntype *url.URL has no field or method Hostname)\n/root/go/src/github.com/tomnomnom/rawhttp/request.go:103: u.Port undefined (type\n *url.URL has no field or method Port)\n /root/go/src/github.com/tomnomnom/rawhttp/request.go:259: undefined: x509.System\n CertPool\n```\n\nYou should either update your version of Go, or use a binary release\nfor your platform.\n\n## Basic Usage\n\nGiven a file full of paths:\n\n```\n/robots.txt\n/.well-known/security.txt\n/package.json\n```\n\nAnd a file full of hosts (with a protocol):\n\n```\nhttp://example.com\nhttps://example.com\nhttp://example.net\n```\n\n`meg` will request each *path* for every *host*:\n\n```\n▶ meg --verbose paths hosts\nout/example.com/45ed6f717d44385c5e9c539b0ad8dc71771780e0 http://example.com/robots.txt (404 Not Found)\nout/example.com/61ac5fbb9d3dd054006ae82630b045ba730d8618 https://example.com/robots.txt (404 Not Found)\nout/example.net/1432c16b671043271eab84111242b1fe2a28eb98 http://example.net/robots.txt (404 Not Found)\nout/example.net/61deaa4fa10a6f601adb74519a900f1f0eca38b7 http://example.net/.well-known/security.txt (404 Not Found)\nout/example.com/20bc94a296f17ce7a4e2daa2946d0dc12128b3f1 http://example.com/.well-known/security.txt (404 Not Found)\n...\n```\n\nAnd save the output in a directory called `./out`:\n\n```\n▶ head -n 20 ./out/example.com/45ed6f717d44385c5e9c539b0ad8dc71771780e0\nhttp://example.com/robots.txt\n\n\u003e GET /robots.txt HTTP/1.1\n\u003e Host: example.com\n\n\u003c HTTP/1.1 404 Not Found\n\u003c Expires: Sat, 06 Jan 2018 01:05:38 GMT\n\u003c Server: ECS (lga/13A2)\n\u003c Accept-Ranges: bytes\n\u003c Cache-Control: max-age=604800\n\u003c Content-Type: text/*\n\u003c Content-Length: 1270\n\u003c Date: Sat, 30 Dec 2017 01:05:38 GMT\n\u003c Last-Modified: Sun, 24 Dec 2017 06:53:36 GMT\n\u003c X-Cache: 404-HIT\n\n\u003c!doctype html\u003e\n\u003chtml\u003e\n\u003chead\u003e\n```\n\nWithout any arguments, meg will read paths from a file called `./paths`,\nand hosts from a file called `./hosts`. There will also be no output:\n\n```\n▶ meg\n▶\n```\n\nBut it will save an *index* file to `./out/index`:\n\n```\n▶ head -n 2 ./out/index\nout/example.com/538565d7ab544bc3bec5b2f0296783aaec25e756 http://example.com/package.json (404 Not Found)\nout/example.com/20bc94a296f17ce7a4e2daa2946d0dc12128b3f1 http://example.com/.well-known/security.txt (404 Not Found)\n```\n\nYou can use the index file to find where the response is stored, but it's\noften easier to find what you're looking for with `grep`:\n\n```\n▶ grep -Hnri '\u003c Server:' out/\nout/example.com/61ac5fbb9d3dd054006ae82630b045ba730d8618:14:\u003c Server: ECS (lga/13A2)\nout/example.com/bd8d9f4c470ffa0e6ec8cfa8ba1c51d62289b6dd:16:\u003c Server: ECS (lga/13A3)\n```\n\nIf you want to request just one path, you can specify it directly as an argument:\n\n```\n▶ meg /admin.php\n```\n\n## Detailed Usage\n\nmeg's help output tries to actually be helpful:\n\n```\n▶ meg --help\nRequest many paths for many hosts\n\nUsage:\n  meg [options] [path|pathsFile] [hostsFile] [outputDir]\n\nOptions:\n  -c, --concurrency \u003cval\u003e    Set the concurrency level (defaut: 20)\n  -d, --delay \u003cval\u003e          Milliseconds between requests to the same host (default: 5000)\n  -H, --header \u003cheader\u003e      Send a custom HTTP header\n  -r, --rawhttp              Use the rawhttp library for requests (experimental)\n  -s, --savestatus \u003cstatus\u003e  Save only responses with specific status code\n  -v, --verbose              Verbose mode\n  -X, --method \u003cmethod\u003e      HTTP method (default: GET)\n\nDefaults:\n  pathsFile: ./paths\n  hostsFile: ./hosts\n  outputDir:  ./out\n\nPaths file format:\n  /robots.txt\n  /package.json\n  /security.txt\n\nHosts file format:\n  http://example.com\n  https://example.edu\n  https://example.net\n\nExamples:\n  meg /robots.txt\n  meg -s 200 -X HEAD\n  meg -c 30 /\n  meg hosts.txt paths.txt output\n```\n\n### Concurrency\n\nBy default meg will attempt to make 20 concurrent requests. You can change that\nwith the `-c` or `--concurrency` option:\n\n```\n▶ meg --concurrency 5\n```\n\nIt's not very friendly to keep the concurrency level higher than the number of\nhosts - you may end up sending lots of requests to one host at once.\n\n### Delay\nBy default meg will wait 5000 milliseconds between requests to the same host.\nYou can override that with the `-d` or `--delay` option:\n\n```\n▶ meg --delay 10000\n```\n\n**Warning:** before reducing the delay, ensure that you have permission to make\nlarge volumes of requests to the hosts you're targeting.\n\n### Adding Headers\n\nYou can set additional headers on the requests with the `-H` or `--header`\noption:\n\n```\n▶ meg --header \"Origin: https://evil.com\"\n▶ grep -h '^\u003e' out/example.com/*\n\u003e GET /.well-known/security.txt HTTP/1.1\n\u003e Origin: https://evil.com\n\u003e Host: example.com\n...\n```\n\n### Raw HTTP (Experimental)\n\nIf you want to send requests that aren't valid - for example with invalid URL encoding -\nthe Go HTTP client will fail:\n\n```\n▶ meg /%%0a0afoo:bar\nrequest failed: parse https://example.org/%%0a0afoo:bar: invalid URL escape \"%%0\"\n```\n\nYou can use the `-r` or `--rawhttp` flag to enable use of the [rawhttp](https://github.com/tomnomnom/rawhttp)\nlibrary, which does little to no validation on the request:\n\n```\n▶ meg --verbose --rawhttp /%%0a0afoo:bar\nout/example.com/eac3a4978bfb95992e270c311582e6da4568d83d https://example.com/%%0a0afoo:bar (HTTP/1.1 404 Not Found)\n```\n\nThe `rawhttp` library and its use is experimental. Amongst other things it doesn't\nyet support chunked transfer encoding, so you may notice chunk lengths interspersed\nwith your output if you use it.\n\n### Saving Only Certain Status Codes\n\nIf you only want to save results that returned a certain status code, you can\nuse the `-s` or `--savestatus` option:\n\n```\n▶ meg --savestatus 200 /robots.txt\n```\n\n### Specifying The Method\n\nYou can specify which HTTP method to use with the `-X` or `--method` option:\n\n```\n▶ meg --method TRACE\n▶ grep -nri 'TRACE' out/\nout/example.com/61ac5fbb9d3dd054006ae82630b045ba730d8618:3:\u003e TRACE /robots.txt HTTP/1.1\nout/example.com/bd8d9f4c470ffa0e6ec8cfa8ba1c51d62289b6dd:3:\u003e TRACE /.well-known/security.txt HTTP/1.1\n...\n```\n","funding_links":[],"categories":["Content Discovery","Go","Weapons","Go (531)","其他_安全与渗透"],"sub_categories":["Crawlers","Tools","网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomnomnom%2Fmeg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomnomnom%2Fmeg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomnomnom%2Fmeg/lists"}