{"id":13546071,"url":"https://github.com/praetorian-inc/noseyparker","last_synced_at":"2025-05-14T03:11:27.015Z","repository":{"id":64484681,"uuid":"563584480","full_name":"praetorian-inc/noseyparker","owner":"praetorian-inc","description":"Nosey Parker is a command-line tool that finds secrets and sensitive information in textual data and Git history.","archived":false,"fork":false,"pushed_at":"2025-05-08T15:20:51.000Z","size":29652,"stargazers_count":1918,"open_issues_count":22,"forks_count":100,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-05-08T15:33:13.402Z","etag":null,"topics":["credentials","devsecops","noseyparker","penetration-testing","rust","scanner","secrets","secrets-detection","security","security-tools"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/praetorian-inc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-08T23:09:17.000Z","updated_at":"2025-05-08T15:10:40.000Z","dependencies_parsed_at":"2023-10-13T12:01:49.848Z","dependency_job_id":"ed823fae-b9c5-45b4-b434-52b6edef01e0","html_url":"https://github.com/praetorian-inc/noseyparker","commit_stats":{"total_commits":481,"total_committers":15,"mean_commits":32.06666666666667,"dds":0.07692307692307687,"last_synced_commit":"b142a76a6aeac5d1ab980867652838e844554715"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praetorian-inc%2Fnoseyparker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praetorian-inc%2Fnoseyparker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praetorian-inc%2Fnoseyparker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/praetorian-inc%2Fnoseyparker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/praetorian-inc","download_url":"https://codeload.github.com/praetorian-inc/noseyparker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254059519,"owners_count":22007771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["credentials","devsecops","noseyparker","penetration-testing","rust","scanner","secrets","secrets-detection","security","security-tools"],"created_at":"2024-08-01T12:00:31.051Z","updated_at":"2025-05-14T03:11:21.996Z","avatar_url":"https://github.com/praetorian-inc.png","language":"Rust","funding_links":[],"categories":["Vulnerability Assessment","Miscellaneous","security-tools","Rust","Projects"],"sub_categories":["Static Code Auditing","Secrets"],"readme":"# Nosey Parker: Find secrets in textual data\n\n## Overview\n\nNosey Parker is a CLI tool that finds secrets and sensitive information in textual data.\nIt is essentially a special-purpose `grep`-like tool for detection of secrets.\n\nIt has been designed for offensive security (e.g., enabling lateral movement on red teams), but it can also be useful for defensive security testing.\nIt has found secrets in hundreds of offensive security engagements at [Praetorian](https://praetorian.com).\n\n**Key features:**\n- **Flexiblity:** It natively scans files, directories, GitHub, and Git history, and has an extensible input enumeration mechanism\n- **Field-tested rules:** It uses regular expressions with [183 patterns](crates/noseyparker/data/default/builtin/rules) chosen for high precision based on feedback from security engineers\n- **Signal-to-noise:** It deduplicates matches that share the same secret, reducing review burden by 10-1000x or more\n- **Speed \u0026 scalability:** it can scan at GB/s on a multicore system, and has scanned inputs as large as 20TB during security engagements\n\nThe typical workflow is three phases:\n\n1. Scan inputs of interest using the `scan` command\n2. Report details of scan results using the `report` command\n3. Review and triage findings\n\n## Installation\n\n### [Homebrew](https://brew.sh) formula\n\n```shell\nbrew install noseyparker\n```\n\n\n### Prebuilt binaries\n\nThe [latest release page](https://github.com/praetorian-inc/noseyparker/releases/latest) contains prebuilt binaries for x86_64/aarch64 Linux and macOS.\n\n\n### Docker: x86_64/aarch64\n\n```shell\ndocker pull ghcr.io/praetorian-inc/noseyparker:latest\n```\n\nThe **most recent commit** is also available via the `main` tag.\n\n### Docker: x86_64/aarch64, Alpine base:\n\n```shell\ndocker pull ghcr.io/praetorian-inc/noseyparker-alpine:latest\n```\n\nThe **most recent commit** is also available via the `main` tag.\n\n\n### Arch Linux package\n\n\u003chttps://aur.archlinux.org/packages/noseyparker\u003e\n\n\n### Windows\n\nNosey Parker does not build natively on Windows ([#121](https://github.com/praetorian-inc/noseyparker/issues/121)).\nIt _is_ possible to run on Windows using [WSL1](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux) and the native Linux release.\n\n\n### Building from source\n\n\u003cdetails\u003e\n\n#### 1. Install prerequisites\nThis has been tested with several versions of Ubuntu Linux and macOS on both x86_64 and aarch64.\n\nRequired dependencies:\n- `cargo`: recommended approach: install from \u003chttps://rustup.rs\u003e\n- `cmake`: needed for building the `vectorscan-sys` crate and some other dependencies\n- `boost`: needed for building the `vectorscan-sys` crate (supported version `\u003e=1.57`)\n- `git`: needed for embedding version information into the `noseyparker` CLI\n- `patch`: needed for building the `vectorscan-sys` crate\n- `pkg-config`: needed for building the `vectorscan-sys` crate\n- `sha256sum`: needed for computing digests (often provided by the `coreutils` package)\n- `zsh`: needed for build scripts\n\n#### 2. Build using the [`create-release.zsh`](scripts/create-release.zsh) script\n```shell\n$ rm -rf release \u0026\u0026 ./scripts/create-release.zsh\n```\n\nIf successful, this will produce a directory structure at `release` populated with release artifacts.\nThe command-line program will be at `release/bin/noseyparker`.\n\u003c/details\u003e\n\n\n## Getting help\n\nRunning the `noseyparker` binary without arguments prints top-level help and exits.\nYou can get abbreviated help for a particular command by running `noseyparker COMMAND -h`.\nMore detailed help is available with the `help` command or long-form `--help` option.\n\nThe prebuilt releases also include manpages that collect the command-line help in one place.\nThese manpages converted into Markdown format are also included in the repository [here](docs/v0.17.0/man/man1).\n\nIf you have a question that's not answered by this documentation, please [start a discussion](https://github.com/praetorian-inc/noseyparker/discussions/new/choose).\n\n\n## Terminology and data model\n\n### The datastore\nThe _datastore_ is a special directory that Nosey Parker uses to record its findings and maintain its internal state.\nA datastore will be implicitly created by the `scan` command if needed.\n\n### Blobs\nEach scanned input is called a _blob_. Each blob has a unique blob ID, which is a SHA-1 digest computed the same way `git` does.\n\n### Provenance\nEach blob has one or more _provenance_ entries associated with it.\nA provenance entry is metadata that describes how the input was discovered, such as a file on the filesystem or a file in Git repository history.\n\n### Rules\nNosey Parker is a rule-based system that uses regular expressions.\nEach _rule_ has a single pattern with at least one capture group that isolates the match content from the surrounding context.\nYou can list available rules with `noseyparker rules list`.\n\n### Rulesets\nA collection of rules is organized into a _ruleset_.\nNosey Parker's default ruleset includes rules that detect things that appear to be secrets.\nOther rulesets are available; you can list them with `noseyparker rules list.`\n\n### Matches\nWhen a rule's pattern matches an input, it produces a _match_.\nA match is uniquely defined by a rule, blob ID, start byte offset, and end byte offset; these fields are used to compute a unique match identifier.\n\n### Findings\nMatches that share a rule and capture groups are combined into a _finding_.\nIn other words, a _finding_ is a group of matches.\nThis is Nosey Parker's top-level unit of reporting.\n\n\n## Usage examples\n\n### NOTE: When using Docker...\n\nWhen using the Docker image, replace `noseyparker` in the following commands with a Docker invocation that uses a mounted volume:\n\n```shell\ndocker run -v \"$PWD\":/scan ghcr.io/praetorian-inc/noseyparker:latest \u003cARGS\u003e\n```\n\nThe Docker container runs with `/scan` as its working directory, so mounting `$PWD` at `/scan` in the container will make tab completion and relative paths in your command-line invocation work.\n\n\n### Scan filesystem content, including local Git repos\n![Screenshot showing Nosey Parker's workflow for scanning the filesystem for secrets](docs/usage-examples/gifs/02-scan-git-history.gif)\n\nNosey Parker has native support for scanning files, directories, and the entire history of Git repositories.\n\nFor example, if you have a Git clone of [CPython](https://github.com/python/cpython) locally at `cpython.git`, you can scan it with the `scan` command.\nNosey Parker will create a new datastore at `cpython.np` and saves its findings there.\n(The name `cpython.np` is innessential, and can be whatever you want.)\n```\n$ noseyparker scan -d cpython.np cpython.git\nScanned 19.19 GiB from 335,849 blobs in 17 seconds (1.11 GiB/s); 2,178/2,178 new matches\n\n Rule                            Findings   Matches   Accepted   Rejected   Mixed   Unlabeled\n──────────────────────────────────────────────────────────────────────────────────────────────\n Generic API Key                        1         8          0          0       0           1\n Generic Password                       8     1,283          0          0       0           8\n Generic Username and Password          2        40          0          0       0           2\n HTTP Bearer Token                      1       108          0          0       0           1\n PEM-Encoded Private Key               61       151          0          0       0          61\n netrc Credentials                     27       588          0          0       0          27\n\nRun the `report` command next to show finding details.\n```\n\nSee `noseyparker help scan` for more details.\n\n### Scan a Git repo from an HTTPS URL\n\nFor example, to scan the Nosey Parker repo itself:\n```\nnoseyparker scan --datastore np.noseyparker --git-url https://github.com/praetorian-inc/noseyparker\n```\n\nSee `noseyparker help scan` for more details.\n\n### Scan Git repos of a GitHub user or organization\n\nUse `--github-user=USER` or `--github-org=ORG`. For example, to scan accessible repositories belonging to the [`octocat`](https://github.com/octocat) user:\n```\nnoseyparker scan --datastore np.noseyparker --github-user octocat\n```\n\nThese input specifiers will use an optional GitHub token if available in the `NP_GITHUB_TOKEN` environment variable.\nProviding an access token gives a higher API rate limit and may make additional repositories accessible to you.\n\nSee `noseyparker help scan` for more details.\n\n\n### Interactively review and annotate findings\nSee the companion project, [Nosey Parker Explorer](https://github.com/praetorian-inc/noseyparkerexplorer):\n![Screenshot showing the main interface of Nosey Parker Explorer](https://github.com/praetorian-inc/noseyparkerexplorer/blob/32e9133600c79eee53cd9000e37b71792e555fdd/docs/img/main-screen.png?raw=true)\n\n### Report findings in human-readable text format\n![Screenshot showing Nosey Parker's workflow for rendering its findings in human-readable format](docs/usage-examples/gifs/03-report-human.gif)\n\n\n### Report findings in JSON format\n![Screenshot showing Nosey Parker's workflow for rendering its findings in JSON format](docs/usage-examples/gifs/04-report-json.gif)\n\n\n### Summarize findings\n\nNosey Parker prints out a summary of its findings when it finishes scanning.\nYou can also run this step separately after scanning:\n```\n$ noseyparker summarize --datastore np.cpython\n\n Rule                      Distinct Groups   Total Matches\n───────────────────────────────────────────────────────────\n PEM-Encoded Private Key             1,076           1,192\n Generic Secret                        331             478\n netrc Credentials                      42           3,201\n Generic API Key                         2              31\n md5crypt Hash                           1               2\n```\n\nAdditional output formats are supported, including JSON and JSON lines, via the `--format=FORMAT` option.\n\nSee `noseyparker help summarize` for more details.\n\n\n### Enumerate repositories from GitHub\n\nUse `github repos list` command to list URLs for repositories belonging to GitHub users or organizations.\nThis command uses the GitHub REST API to enumerate repositories belonging to users or organizations.\nFor example:\n```\n$ noseyparker github repos list --user octocat\nhttps://github.com/octocat/Hello-World.git\nhttps://github.com/octocat/Spoon-Knife.git\nhttps://github.com/octocat/boysenberry-repo-1.git\nhttps://github.com/octocat/git-consortium.git\nhttps://github.com/octocat/hello-worId.git\nhttps://github.com/octocat/linguist.git\nhttps://github.com/octocat/octocat.github.io.git\nhttps://github.com/octocat/test-repo1.git\n```\n\nThis command will use an optional GitHub token if available in the `NP_GITHUB_TOKEN` environment variable.\nProviding an access token gives a higher API rate limit and may make additional repositories accessible to you.\n\nAdditional output formats are supported, including JSON and JSON lines, via the `--format=FORMAT` option.\n\nSee `noseyparker help github` for more details.\n\n\n## Integrations\n\nNosey Parker has a few third-party integrations:\n\n- Nosey Parker is packaged in [Homebrew](https://formulae.brew.sh/formula/noseyparker)\n- Nosey Parker is packaged in [Arch Linux](https://aur.archlinux.org/packages/noseyparker)\n- A [GitHub Action](https://github.com/bpsizemore/noseyparker-action) that runs Nosey Parker is available\n- [DefectDojo](https://defectdojo.org) includes a [parser for Nosey Parker v0.16 JSON](https://github.com/DefectDojo/django-DefectDojo/blob/c182e9ca9d8f981c15de2018f948fe69c4d1a800/docs/content/en/integrations/parsers/file/noseyparker.md)\n- [Nemesis](https://github.com/SpecterOps/Nemesis) includes support for Nosey Parker\n\nIf you have an integration you'd like to share that's not listed here, please create a PR.\n\n\n## Contributing\n\nAsk questions or share ideas in the [Discussions](https://github.com/praetorian-inc/noseyparker/discussions) area.\n\nContributions are welcome, especially new regex rules.\nDeveloping new regex rules is detailed in a [separate document](docs/RULES.md).\n\nIf you are considering making significant code changes, please [open an issue](https://github.com/praetorian-inc/noseyparker/issues/new) or [start a discussion](https://github.com/praetorian-inc/noseyparker/discussions/new/choose) first.\n\nThis project has a number of [pre-commit](https://pre-commit.com/) hooks enabled that you are encouraged to use.\nTo install them in your local repo, make sure you have `pre-commit` installed and run:\n```\n$ pre-commit install\n```\nThese checks will help to quickly detect simple errors.\n\n\n## License\n\nNosey Parker is licensed under the [Apache License, Version 2.0](LICENSE).\n\nAny contribution intentionally submitted for inclusion in Nosey Parker by you, as defined in the Apache 2.0 license, shall be licensed as above, without any additional terms or conditions.\n\nNosey Parker also includes vendored copies of several other packages released under the Apache License and other permissive licenses; see [`LICENSE`](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpraetorian-inc%2Fnoseyparker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpraetorian-inc%2Fnoseyparker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpraetorian-inc%2Fnoseyparker/lists"}