{"id":18155434,"url":"https://github.com/0x4f53/secretsnitch","last_synced_at":"2025-05-01T12:49:01.251Z","repository":{"id":248416610,"uuid":"812055003","full_name":"0x4f53/secretsnitch","owner":"0x4f53","description":"A lightning-fast, modular secret scanner","archived":false,"fork":false,"pushed_at":"2024-10-23T20:50:53.000Z","size":150,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-24T08:53:39.419Z","etag":null,"topics":["artificial-intelligence","blueteam","chatgpt","concurrency","golang","gpt","openai-api","pentesting","pentesting-tools","recon","reconnaissance","redteam","secrets","secrets-management","security","security-tools"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0x4f53.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-07T21:26:13.000Z","updated_at":"2024-10-21T14:06:00.000Z","dependencies_parsed_at":"2024-10-26T01:10:04.958Z","dependency_job_id":"16167a5b-3412-435d-8303-c3198be01c31","html_url":"https://github.com/0x4f53/secretsnitch","commit_stats":null,"previous_names":["0x4f53/secretsnitch"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x4f53%2Fsecretsnitch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x4f53%2Fsecretsnitch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x4f53%2Fsecretsnitch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0x4f53%2Fsecretsnitch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0x4f53","download_url":"https://codeload.github.com/0x4f53/secretsnitch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246379366,"owners_count":20767696,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","blueteam","chatgpt","concurrency","golang","gpt","openai-api","pentesting","pentesting-tools","recon","reconnaissance","redteam","secrets","secrets-management","security","security-tools"],"created_at":"2024-11-02T04:10:58.828Z","updated_at":"2025-03-31T18:30:20.591Z","avatar_url":"https://github.com/0x4f53.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Golang](https://img.shields.io/badge/Golang-eee.svg?style=flat-square\u0026logo=go)](https://go.dev)\n[![License](https://img.shields.io/badge/License-MIT-purple?style=flat-square\u0026logo=libreoffice)](LICENSE)\n[![Latest Version](https://img.shields.io/github/v/tag/0x4f53/secretsnitch?label=Version\u0026style=flat-square\u0026logo=semver)](https://github.com/0x4f53/secretsnitch/releases)\n\nPresented at \n\n[![BlackHat MEA 2024](https://img.shields.io/badge/BlackHat%20MEA%202024-222.svg?style=flat-square\u0026logo=redhat)](https://blackhatmea.com/blackhat-arsenal)\n\n\u003cimg src = \"media/logo.png\" alt = \"Secretsnitch logo\" width = \"60dp\"\u003e\n\n# Secretsnitch\n\nA fast continuous secret scanner and endpoint extractor in Golang!\n\n\u003cimg src = \"media/terminal.gif\" alt = \"Secretsnitch logo\" width = \"600dp\"\u003e\n\n*(note: all the secrets in this document are censored since they're real*)\n\n- **Continous scanning**: non-stop enumeration of scanning from several sources, helping organizations catch and secure accidental secret exposures. \n\n- **Concurrency**: lightning fast scanning with thousands of Goroutines - at once!\n\n- **Modular design**: Supports GitHub, GitLab, Phishtank and random web scraping via flags.\n\n- **Efficient networking**: reduced network usage with caching and instant output logging.\n\n- **Comprehensive checks**: huge signature list for variable names, secret strings etc combined with metadata and entropy scoring.\n\n- **User-friendly**: Designed for ease of use by pentesters, bounty hunters, or enterprise users via simple command-line execution.\n\n- **Community-driven**: sourced from Google searches, ChatGPT, and open-source lists like GitGuardian.\n\n- **Easy contribution**: Find a missing secret or blacklist regular expression? Simply make a pull request by following the [contributing.md] guide!\n\n## Usage\n\n```bash\n\n❯ go build\n❯ ./secretsnitch -h⠀⠀\n\nSecretsnitch - A lightning-fast secret scanner in Golang!\nhttps://github.com/0x4f53/secretsnitch\nCopyright © 2024 Owais Shaikh\n\nUsage:\n./secretsnitch [input options] [output options]\n\nInput (pick at least one):\n\n  --github                  Scan public GitHub commits from the past hour\n    --from                  (optional) Timestamp to start from (format: 2006-01-02-15)\n    --to                    (optional) Timestamp to stop at (format: 2006-01-02-15)\n\n  --github-gists            Scan the last 100 public GitHub Gists\n\n  --gitlab                  Scan the last 100 public GitLab commits\n\n  --phishtank               Scan reported phishtank.org URLs from the past day\n\n  --url=\u003chttp://url\u003e        A single URL to scan\n  --urlList=\u003cfile\u003e          A line-separated file containing a list of URLs to scan for secrets\n\n  --directory=\u003cdirectory/\u003e  Scan an entire directory\n  --file=\u003cfile.js\u003e          Scan a file\n\nOptional arguments:\n\n  --output                  Save scan output to a custom location\n\n  --workers                 Maximum number of workers to use (default: 5000)\n\n  --recursions=\u003cnumber\u003e     Crawl URLs and hyperlinks inside targets (default: 0)\n\n  --retries=\u003cnumber\u003e        Maximum retries before giving up (default: 3)\n\n  --secrets-optional        Display other data (such as endpoints, domains etc.) even if there are no secrets\n\n  --selenium                Scrape page using Selenium. This helps with pages that run client-side Javascript (note: this is slower)\n\npflag: help requested\n```\n\n### Examples\n\n#### All GitHub commits in a range (GitHub module)\n\nThe GitHub module lets you search for secrets based on a commit timestamp. It generates patch files from a dataset that Gharchive hosts, then tries scraping those links.\n\nGitHub uses something called \"patch\" files to keep track of changes that are made to repositories. These days, git has an in-built integration to prevent secret exposures. However, several secrets still slip by and some archived commits from the past may still contain secrets.\n\nLets say you would like to scan for commits from January 10, 2023 at 3 PM to January 15 2023 at 12 AM. You can run the below command\n\n```bash\n./secretsnitch --github --from=2024-01-10-15 --to=2024-01-15-0 --workers=100\n```\n\nSince the public GitHub API has rate limits, I recommend reducing the number of workers to strike the right balance between scraping commit files quickly and not exceed the official rate limit.\n\nThis results in the following output\n\n```bash\nSecretsnitch - A lightning-fast secret scanner in Golang!\nhttps://github.com/0x4f53/secretsnitch\nCopyright © 2024 Owais Shaikh\nDownloading and extracting non-concurrently...\nJSON file .githubCommits//2024-10-31-13.json already exists. Continuing...\n2024/10/31 20:31:40 Skipping https://github.com/cellatlas/human/commit/55f43f34ca277991e3a9b96a490999ea6e524d8b.patch as it is already cached at .urlCache/b65fb311.cache\n2024/10/31 20:31:40 Skipping https://github.com/omarzydan610/Calculator_React-SpringBoot/commit/6b26bcd9f9e926dc5cc03cbd7f5cab01b966bced.patch as it is already cached at .urlCache/156e201d.cache\n...\n\n2024/10/31 20:32:30 \n---\n\nSECRET DETECTED:\n    - Type:            Generic Password Variable\n    - Variable Name:   password\n    - Value:           Tes█████████\n    - Position:        26:27\n    - Source:          https://github.com/kal█████████████████████████████████████████████a8f37b12b061db16f5a13e.patch\n    - Cached Location: .urlCache/674d3824.cache\n    - Tags:            [\"regexMatched\"]\n    - Tsallis Entropy: 0.843750\n2024/10/31 20:32:30 Searching for secrets in: https://github.com/aaron-hwang/SRL/commit/d1e34fad5b7bdda6498ec5b6654323cbb0754c1a.patch (cached at: .urlCache/def4ec10.cache)\n...\n```\n\nIf you want to run the scan on just the last hour, simply run the command without the `from` and `to` flags.\n\n```bash\n./secretsnitch --github --workers=100\n```\n\n#### Single URLs\n\nSay you have the following page:\n\nhttps://0x4f.in\n\nThis page has a hardcoded OpenAI API Key in a Javascript file that it calls, named `security.js`. Simply running secretsnitch with the following command:\n\n```bash\n❯ go build\n❯ ./secretsnitch --url=https://0x4f.in --recursions=1 --retries=0\n```\n\nHere, the `recursions` flag is set to `1`. This means that only the first layer of URLs within this page will be scanned, and not the URLs within those pages. `retries` being set to `0` means if a page can't be scraped, it will be left alone.\n\nThis gives you the following output:\n\n```bash\n2024/10/31 06:25:48 Content from https://0x4f.in saved to .urlCache/\n2024/10/31 06:25:48 Searching for secrets in: https://0x4f.in (cached at: .urlCache/495c7d01.cache)\n2024/10/31 06:25:50 Content from https://0x4f.in//assets/js/security.js saved to .urlCache/\n2024/10/31 06:25:50 Content from https://play.google.com/store/apps/details?id=zeroxfourf.wristkey saved to .urlCache/\n...\n```\n\nOnce the pages are cached, the secret scanning begins.\n\n```bash\n...\n2024/10/31 06:28:01 Searching for secrets in: https://0x4f.in//assets/js/security.js (cached at: .urlCache/bf212e67.cache)\n2024/10/31 06:28:02 \n---\n\nSECRET DETECTED:\n    - Type: OpenAI Service API Key\n    - Variable Name: openAiApiKey\n    - Value: sk-proj-█████████████████████████████████████████████\n    - Position: 24:29\n    - Source: https://0x4f.in/assets/js/security.js\n    - Cached Location: .urlCache/0918b0a2.cache\n    - Tags: [\"regexMatched\",\"providerDetected\",\"longString\"]\n    - Tsallis Entropy: 0.979253\n\nDOMAINS FOUND:\n    - 1. 0x4f.in\n    - 2. github.com\n\nURLs FOUND:\n    - 1. https://github.com/payloadbox/xss-payload-list\n    - 2. https://0x4f.in/assets/images/ryanbeckford.gif\n...\n```\n\nAnd thus, the secret is caught.\n\n#### Stolen API Keys from phishing websites (Phishtank module)\n\nRunning secretsnitch with the following command:\n\n```bash\n./secretsnitch --phishtank\n```\n\nGrabs the latest URL archive from the Phishtank API. It then begins downloading and scraping all the pages from Phishtank first, post which the secret analysis begins.\n\n### Components\n\n#### Basic flow\n\nThe basic operation of the tool is as follows\n\n\u003cimg src = \"media/secretsnitch_basic.drawio.png\" alt = \"Basic workflow\"\u003e\n\n#### Modules\n\n##### Services\n\n- **github**: Scan public GitHub commits from the past hour\n  - (optional) **from**: Timestamp to start from (format: 2006-01-02-15)\n  - (optional) **to**:   Timestamp to stop at (format: 2006-01-02-15)\n- **github-gists**:      Scan the last 100 public GitHub Gists\n- **gitlab**:            Scan the last 100 public GitLab commits\n- **phishtank**:         Scan reported phishtank.org URLs from the past day\n\n##### Miscelllaneous\n\n- **url**:  A single URL to scan. *Tip: some URLs use client-side Javascript rendering (CSR). Use the `--selenium` flag to beat this and scraped rendered pages.*\n- **urlList**:           A line-separated file containing a list of URLs to scan for secrets\n- **directory**:         Scan an entire directory. *Tip: if you're bored waiting for pages to finish scraping, you can simply terminate the program and run it with `--directory=.urlCache/`* ;)\n- **file**:              Scan a single file\n\nYou can also import the secretsnitch scanner as a golang package within your own security tools.\n\nTo do this, simply open your Go project and type\n\n```bash\ngo get -u github.com/0x4f53/secretsnitch\n```\n\n#### Caching\n\nThis tool supports caching in order to save you some time. When a single URL or a list of URLs is supplied to the tool, each URL is sent to a worker. Each worker then scrapes and logs it to a patch file stored in `.urlCache/`. The named of each file is an MD5 hash of the URL in a primary-key like fashion, to make it easy to store, retrieve and detect the presence of pre-cached files without using a database.\n\n\u003cimg src = \"media/secretsnitch_caching.drawio.png\" alt = \"Caching workflow\"\u003e\n\nThis is quite useful in several scenarios, such as when starts are restarted with reduced workers if the tool crashes, or if the tool is restarted due to rate-limits being indefinite.\n\nAlthough sometimes, this caching feature can interfere with your use of this tool, for example, while bounty hunting. To fix this, simply delete the `.urlCache/` directory and start the tool again.\n\n#### Tunables\n\nSecretsnitch is extremely tunable via for different use cases, whether its the worker count to prevent slowdowns on older devices, or the output destination for logshipping via tools like Filebeat.\n\nTunables available:\n\n- **output**: Save scan output to a custom location. Directories and subdirectories will be created if they don't exist.\n\n- **workers**: The amount of workers that can run each operation concurrently. This should be set according to hardware factors like CPU threads, rate-limits etc. When you're bruteforcing a large list of URLs or a directory on a powerful server, set this number to a high number.\n\n- **recursions**: Crawl URLs inside files, then crawl the URLs inside those URLs, then the URLs in the URLs in the URLs, then the URLs in the URLs in the URLs in the URLs, then... you get the point.\n\n- **retries**: Give up after trying so many times. Useful if the destination is misbehaving or crashing.\n\n- **secrets-optional**: Display other data such as URLs and domains even if there are no secrets. Useful for asset extraction.\n\n- **selenium**: If a site uses client-side rendering (CSR), you can use the Selenium plugin to have the Javascript be rendered first, then extract secrets from it. Please note that Docker needs to be installed for this to work.\n\n#### Scanning\n\n#### Tokenization\n\n\u003cimg src = \"media/secretsnitch_tokenizer.drawio.png\" alt = \"Tokenizer workflow\"\u003e\n\nWhen a file containing code is passed to the tool, it uses tokenization techniques via in-built regular expressions, string splitting and so on.\n\nThese techniques are tested and optimized for codebases of languages commonly used with backend development, such as\n\n- Javascript\n- Golang\n- Bash\n- Python\n- Java\n- etc.\n\nThere is also support for common structured file formats such as\n\n- JSON\n- env\n- XML\n- HTML\n- etc.\n\n#### Parsing\n\n\u003cimg src = \"media/secretsnitch_variable_scanner.drawio.png\" alt = \"Scanner workflow\"\u003e\n\nSecretsnitch looks for two classifications of secrets:\n\n1. Single secrets: These include\n    - API Keys\n    - Password strings\n    - URLs with leaked authentication (such as `mysql://` connection strings)\n\n2. Secret files: These include\n    - SSH Private Keys\n    - `.pem` files\n  \nIn addition to the above, the tool also looks for other assets that could work in conjunction with the secrets. This includes\n\n- URLs: These may be URLs for things like S3 storage, documentation, or may contain PII.\n\n- Domains These are domains that the tool captures and validates with the system's DNS resolvers. This is especially useful for correlating secrets with a particular organization.\n\n#### Detection\n\n\u003cimg src = \"media/sigs_screenshot.png\" alt = \"Signatures\"\u003e\n\nDetection uses two signature files supplied with the tool.\n\n- `signatures.yaml`\nThis file contains a list of regular expressions that the tool uses to catch secrets. They are sorted by the service provider (Amazon, Microsoft etc.) and the service they provide (e.g.: SQS, SharePoint etc.)\n\n    It contains two classifications of regular expressions:\n\n  - Variable patterns: These are patterns that are used to check for variable names where the secret may not have a recognizable pattern (for example, passwords). These are marked by the usage of the word `Variable` at the end of the signature's key.\n\n  - Secret patterns: these are patterns that are commonly used by secrets, for example, `AIza...` for GCP keys, `AKIA` for AWS keys and so on.\n\n- `blacklist.yaml`\nThis file contains a list of blacklist patterns that are skipped. These include things like blob data patterns for images and audio, certain placeholderstrings etc.\n\nThese files are parsed, then the compiled patterns are matched against variable names and values using Golang's `regexp` library in a loop. The result from the `FindSecrets()` function is returned as a slice of `ToolData` and is logged immediately.\n\n## Troubleshooting and tips\n\n### False positives and false negatives\n\nSome false positive secrets simply cannot be mitigated. Others, however, can be improved through continuous improvement of the signatures list and the algorithms. False negatives may also occur if the secret patterns are too strict.\n\nIf you have any ideas to prevent false positives and false negatives, please [raise an issue](https://github.com/0x4f53/secretsnitch/issues) or [contribute](contributing.md) to the project.\n\n### URL Cache contains data that's too old\n\nSometimes, caching can interfere with your use of this tool with outdated or inaccurate data. To fix this, simply delete the `.urlCache/` directory and start the tool again.\n\n### GitHub rate limits\n\nIf your worker count is above GitHub's permitted public API limits, blasting multiple queries will result in an error `429` and a rate-limit. To prevent this, simply set the `workers` flag to 100 or a lower number. This trick also works with the URL list option if the source URLs have rate-limiting enabled.\n\n### Tool stops instantly\n\nSometimes, the tool just stops as soon as it is started. This is due to a bug with the concurrency. Simply re-run the tool a few times if this happens.\n\n### Selenium mode\n\nIf you receive a message like the one below on Linux\n\n```bash\nError creating container: Post \"http://unix.sock/containers/create?\": dial unix /var/run/docker.sock: connect: permission denied\n```\n\nSimply run\n\n```bash\nsudo usermod -aG docker $USER\nnewgrp docker\n```\n\nAnd try again.\n\nFor Windows, try the following\n\n1. Open Docker Desktop Settings\n2. Right-click the Docker icon in the system tray and select Settings.\n3. Enable the TCP Endpoint\n4. Go to the General or Resources \u003e Advanced settings (the exact menu depends on the Docker Desktop version).\n5. Look for an option to Expose daemon on tcp://localhost:2375 without TLS.\n6. Enable this option if you’re okay with using an insecure endpoint, or set up certificates for localhost:2376 if you prefer TLS.\n7. Go to the `dockerSelenium.go` file and replace `unix:///var/run/docker.sock` with `http://localhost:2375`.\n\nIf you can successfully build the docker image manually but can't trigger it via secretsnitch, try running the tool as superuser via `sudo`.\n\n## Contribution\n\nThis tool needs contributors to grow and become extremely versatile and powerful. Please read the [contributing documentation](contributing.md) if you'd like to contribute.\n\n## Acknowledgements\n\n- [GitGuardian](https://docs.gitguardian.com/secrets-detection/secrets-detection-engine/detectors/supported_credentials), for their extensive regular expressions list. Can't thank them enough for kickstarting this.\n- [GitSecrets](https://github.com/awslabs/git-secrets)\n- [Secretsearch](https://gitlab.com/redhuntlabs/secretsearch), the predecessor of this project. I do not maintain it anymore.\n\n## License\n\nMultimedia licensed under [![License: CC BY-NC-SA 4.0](https://licensebuttons.net/l/by-nc-sa/4.0/80x15.png)](https://creativecommons.org/licenses/by-nc-sa/4.0/)\n\n[Copyright © 2024 Owais Shaikh](LICENSE)\n\n## Donate\n\nIf you'd like to donate to me, [visit my GitHub page](https://github.com/0x4f53). It incentivizes me to develop more.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0x4f53%2Fsecretsnitch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0x4f53%2Fsecretsnitch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0x4f53%2Fsecretsnitch/lists"}