{"id":15374448,"url":"https://github.com/ivan-sincek/metagoofeel","last_synced_at":"2025-02-28T00:32:27.905Z","repository":{"id":107019204,"uuid":"195104981","full_name":"ivan-sincek/metagoofeel","owner":"ivan-sincek","description":"Web crawler and downloader based on GNU Wget.","archived":true,"fork":false,"pushed_at":"2023-04-25T15:09:39.000Z","size":4,"stargazers_count":12,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-16T14:56:51.536Z","etag":null,"topics":["bash","bug-bounty","downloader","ethical-hacking","gnu-wget","offensive-security","red-team-engagement","security","web","web-crawler","web-penetration-testing"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivan-sincek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-03T18:07:18.000Z","updated_at":"2024-11-17T22:23:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"9813217f-b268-4d81-b128-b82082e1e661","html_url":"https://github.com/ivan-sincek/metagoofeel","commit_stats":{"total_commits":1,"total_committers":1,"mean_commits":1.0,"dds":0.0,"last_synced_commit":"e7baad20fe9dea217b3d6ccca92226dd0feebd70"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-sincek%2Fmetagoofeel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-sincek%2Fmetagoofeel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-sincek%2Fmetagoofeel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-sincek%2Fmetagoofeel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivan-sincek","download_url":"https://codeload.github.com/ivan-sincek/metagoofeel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241078871,"owners_count":19905948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","bug-bounty","downloader","ethical-hacking","gnu-wget","offensive-security","red-team-engagement","security","web","web-crawler","web-penetration-testing"],"created_at":"2024-10-01T13:58:48.655Z","updated_at":"2025-02-28T00:32:27.899Z","avatar_url":"https://github.com/ivan-sincek.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Metagoofeel\n\nWeb crawler and downloader based on GNU Wget.\n\nThe goal is to be less intrusive than simply mirroring an entire website.\n\nYou can also import your own list with already crawled URLs (e.g. from Burp Suite).\n\nCurrent regular expression for extracting URLs from GNU Wget's output is `(?\u003c=URL\\:\\ )[^\\s]+(?=\\ 200\\ OK)` and for downloading is simply to check if the supplied keyword is contained in a URL.\n\nTweak this tool to your liking by modifying regular expressions.\n\nTested on Kali Linux v2021.2 (64-bit).\n\nMade for educational purposes. I hope it will help!\n\n## How to Run\n\nOpen your preferred console from [/src/](https://github.com/ivan-sincek/metagoofeel/tree/master/src) and run the commands shown below.\n\nInstall required packages:\n\n```fundamental\napt-get -y install bc\n```\n\nChange file permissions:\n\n```fundamental\nchmod +x metagoofeel.sh\n```\n\nRun the script:\n\n```fundamental\n./metagoofeel.sh\n```\n\nTail the crawling progress (optional):\n\n```fundamental\ntail -f metagoofeel_urls.txt\n```\n\n## Usage\n\n```fundamental\nMetagoofeel v2.2 ( github.com/ivan-sincek/metagoofeel )\n\n--- Crawl ---\nUsage:   ./metagoofeel.sh -d domain              [-r recursion]\nExample: ./metagoofeel.sh -d https://example.com [-r 20       ]\n\n--- Crawl and download ---\nUsage:   ./metagoofeel.sh -d domain              -k keyword [-r recursion]\nExample: ./metagoofeel.sh -d https://example.com -k all     [-r 20       ]\n\n--- Download from a file ---\nUsage:   ./metagoofeel.sh -f file                 -k keyword\nExample: ./metagoofeel.sh -f metagoofeel_urls.txt -k pdf\n\nDESCRIPTION\n    Crawl through an entire website and download specific or all files\nDOMAIN\n    Domain you want to crawl\n    -d \u003cdomain\u003e - https://example.com | https://192.168.1.10 | etc.\nKEYWORD\n    Keyword to download only specific files\n    Use 'all' to download all files\n    -k \u003ckeyword\u003e - pdf | js | png | all | etc.\nRECURSION\n    Maximum recursion depth\n    Use '0' for infinite\n    Default: 10\n    -r \u003crecursion\u003e - 0 | 5 | etc.\nFILE\n    File with [already crawled] URLs\n    -f \u003cfile\u003e - metagoofeel_urls.txt | etc.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivan-sincek%2Fmetagoofeel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivan-sincek%2Fmetagoofeel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivan-sincek%2Fmetagoofeel/lists"}