{"id":13537869,"url":"https://github.com/needmorecowbell/hamburglar","last_synced_at":"2025-04-05T09:04:57.633Z","repository":{"id":34780356,"uuid":"140905110","full_name":"needmorecowbell/Hamburglar","owner":"needmorecowbell","description":"Hamburglar -- collect useful information from urls, directories, and files","archived":false,"fork":false,"pushed_at":"2022-12-08T05:47:57.000Z","size":34280,"stargazers_count":318,"open_issues_count":11,"forks_count":24,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-29T08:06:13.653Z","etag":null,"topics":["information-gathering","python3"],"latest_commit_sha":null,"homepage":"","language":"YARA","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/needmorecowbell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-14T01:01:02.000Z","updated_at":"2025-03-24T02:36:10.000Z","dependencies_parsed_at":"2023-01-15T09:07:55.246Z","dependency_job_id":null,"html_url":"https://github.com/needmorecowbell/Hamburglar","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/needmorecowbell%2FHamburglar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/needmorecowbell%2FHamburglar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/needmorecowbell%2FHamburglar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/needmorecowbell%2FHamburglar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/needmorecowbell","download_url":"https://codeload.github.com/needmorecowbell/Hamburglar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247312077,"owners_count":20918344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["information-gathering","python3"],"created_at":"2024-08-01T09:01:04.441Z","updated_at":"2025-04-05T09:04:57.575Z","avatar_url":"https://github.com/needmorecowbell.png","language":"YARA","funding_links":[],"categories":["\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e新添加的","\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"31185b925d5152c7469b963809ceb22d\"\u003e\u003c/a\u003e新添加的"],"readme":"## The Hamburglar\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/7833164/51336290-29a79600-1a52-11e9-96a1-beac9207fdab.gif\"\u003e\u003c/img\u003e\n\u003c/p\u003e\n\n\n## Setup\n\nThere are 2 versions of hamburglar, full and lite. The main branch is the full version, and hamburglar lite is on a separate branch.\n\n**Hamburglar**\n\nFull fledged scraping tool for artifact retrieval from multiple sources. There are some dependencies, so install them first: \n\n`pip3 install -r requirements.txt`\n\nHamburglar also has the option of checking against file signatures during a hexdump. It will get skipped if not set up. To get it working, you will need to first create the database and a user:\n\n```sql\nCREATE DATABASE \nCREATE USER 'hamman'@'localhost' IDENTIFIED BY 'deadbeef';\nGRANT ALL PRIVILEGES ON fileSign.signatures TO 'hamman'@'localhost';\n```\n\nThen, run magic_sig_scraper. This can be run on a cronjob to regularly update it, or just run it once:\n\n`python3 magic_sig_scraper.py`\n\n**Hamburglar Lite**\n\nMultithreaded and recursive directory scraping script. Stores useful information with the filepath and finding. Hamburglar lite will never require external packages, and will always remain as a single script. Setup is as simple as requesting the file and using it:\n\n`wget https://raw.githubusercontent.com/needmorecowbell/Hamburglar/hamburglar-lite/hamburglar-lite.py`\n\nThis is designed to be quickly downloaded and executed on a machine.\n\n\n## Operation\n\n```\nusage: hamburglar.py [-h] [-g] [-x] [-v] [-w] [-i] [-o FILE] [-y YARA] path\n\npositional arguments:\n  path                  path to directory, url, or file, depending on flag\n                        used\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -g, --git             sets hamburglar into git mode\n  -x, --hexdump         give hexdump of file\n  -v, --verbose         increase output verbosity\n  -w, --web             sets Hamburgler to web request mode, enter url as path\n  -i, --ioc             uses iocextract to parse contents\n  -o FILE, --out FILE   write results to FILE\n  -y YARA, --yara YARA  use yara ruleset for checking\n```\n\n\n**Directory Traversal**\n\n- `python3 hamburglar.py ~/Directory/`\n    - This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filters\n\n**Single File Analysis**\n\n- `python3 hamburglar.py ~/Directory/file.txt`\n    - This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filters\n\n**YARA Rule Based Analysis**\n- `python3 hamburglar.py -y rules/ ~/Directory`\n    - This will compile the yara rule files in the rules directory and then check them against every item in Directory.\n\n**Git Scraping Mode**\n\n- `python3 hamburglar.py -g https://www.github.com/needmorecowbell/Hamburglar`\n    - Adding `-y \u003crulepath\u003e` will allow the repo to be scraped using yara rules\n\n**Web Request Mode**\n\n- `python3 hamburglar.py -w https://google.com`\n    - Adding a `-w` to hamburgler.py tells the script to handle the path as a url. \n    - Currently this does not spider the page, it just analyzes the requested html content\n\n**IOC Extraction**\n- `python3 hamburglar.py -w -i https://pastebin.com/SYisR95m`\n    - Adding a `-i` will use iocextract to extract any ioc's from the requested url\n    \n**Hex Dump Mode**\n\n- `python3 hamburglar.py -x ~/file-to-dump`\n    - This just does a hex dump and nothing more right now -- could be piped into a file\n    - This will eventually be used for binary analysis\n    \n**Tips**\n\n- Adding `-v` will set the script into verbose mode, and `-h` will show details of available arguments\n- Adding `-o FILENAME` will set the results filename, this is especially useful in scripting situations where you might want multiple results tables (ie github repo spidering)\n\n## Settings\n\n- `whitelistOn`: turns on or off whitelist checking\n- `maxWorkers`: number of worker threads to run concurrently when reading file stack \n- `whitelist`: list of files or directories to exclusively scan for (if whitelistOn=True)\n- `blacklist`: list of files, extensions, or directories to block in scan\n- `regexList`: dictionary of regex filters with filter type as the key\n\n## The Hamburglar can find\n\n- ipv4 addresses (public and local)\n- emails\n- private keys\n- urls\n- ioc's (using iocextract)\n- cryptocurrency addresses\n- anything you can imagine using regex filters and yara rules\n\n## Example output:\n\n```json\n{\n    \"/home/adam/Dev/test/email.txt\": {\n        \"emails\": \"{'testingtesting@gmail.com'}\"\n    },\n    \"/home/adam/Dev/test/email2.txt\": {\n        \"emails\": \"{'loall@gmail.com'}\"\n    },\n    \"/home/adam/Dev/test/ips.txt\": {\n        \"ipv4\": \"{'10.0.11.2', '192.168.1.1'}\"\n    },\n    \"/home/adam/Dev/test/test2/email.txt\": {\n        \"emails\": \"{'ntested4@gmail.com', 'ntested@gmail.com'}\"\n    },\n    \"/home/adam/Dev/test/test2/ips.txt\": {\n        \"ipv4\": \"{'10.0.11.2', '192.168.1.1'}\"\n    },\n    \"/home/adam/Dev/test/test2/links.txt\": {\n        \"site\": \"{'http://login.web.com'}\"\n    }\n}\n```\n## Contributions ##\n\n- Please contribute! If there's an error let me know -- even better if you can fix it :)\n\n- A big thank you to anyone who has helped:\n\n  - [adi928](https://github.com/adi928)\n  - [jaeger-2601](https://github.com/jaeger-2601)\n  - [tijko](https://github.com/tijko)\n  - [joanbono](https://github.com/joanbono) and [Xumeiquer](https://github.com/Xumeiquer) for the rules from [yara-forensics](https://github.com/Xumeiquer/yara-forensics)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneedmorecowbell%2Fhamburglar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneedmorecowbell%2Fhamburglar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneedmorecowbell%2Fhamburglar/lists"}