https://github.com/grab/secret-scanner
https://github.com/grab/secret-scanner
Last synced: 9 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/grab/secret-scanner
- Owner: grab
- License: mit
- Created: 2019-11-13T08:47:37.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2024-01-02T04:26:58.000Z (about 2 years ago)
- Last Synced: 2025-03-25T08:03:15.915Z (10 months ago)
- Language: Go
- Size: 15.5 MB
- Stars: 49
- Watchers: 6
- Forks: 12
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Secret Scanner
Secret scanner is a command-line tool to scan Git repositories for any sensitive information such as private keys, API secrets and tokens, etc.
It does so by looking at file names, extensions, and content, attempting to match them against a list of signatures.
The tool is based on Gitrob, with added support for Gitlab and Bitbucket on top of Github.
## Setup
## Auth Tokens
The use of this tool requires you to set various Git provider (Github / Gitlab / Bitbucket) authentication token in your environment.
You can do so by:
```
export GITHUB_TOKEN=my-token; secret-scanner -repos jquery/jquery
```
To persist the various Git provider tokens, you can add them into your `.bash_profile` or create a `.env` file. See `.env.example`.
### Skip Files
You can define paths to be excluded from scanning by defining them in a comma separated format in `.env` file.
`SKIP_EXT` defines the file extensions to be excluded
`SKIP_PATHS` defines the paths/files to be excluded if the path matches one of the patterns defined in the list
`SKIP_TEST_PATHS` defines any test directories/files that you would like to skip. It is being kept separately from `SKIP_PATHS` because sometimes it may be useful to scan the test files as well. You can toggle to scan test files by giving `-skip-tests=false` in the CLI.
## Usage
For `bool` CLI flags, use `=` between key-val pair. Eg `-ui=false`
### Basic
The most basic usage requires a list of Github repository identifiers in the form of `org/repo`.
```
./secret-scanner -repos jquery/jquery
```
To scan repositories in other Git providers, simply specify the Git provider name.
For Gitlab, provide the project ID instead of `org/repo`.
```
./secret-scanner -git bitbucket -repos litmis/mama
./secret-scanner -git gitlab -repos 3836952
```
You can scan multiple repositories from the same Git provider by providing multiple identifiers separated by commas.
```
./secret-scanner -repos jquery/jquery,lodash/lodash
```
### Local Scan
By default, the tool will attempt to make a clone before scanning the files.
If you already have a copy of the repository on local disk, you can do a local scan by specifying the `dir` parameter.
```
./secret-scanner -dir /dir/path/to/local/repository
```
### Sub-directory Scan
In instances where a repository contains multiple projects (i.e monorepo), or you simply want to scan specific sub-directory, you can do so by providing `sub-dir`.
Example:
https://github.com/user/awesome-projects contains
- build/
- dist/
- src/
- test/
- ...
Caveat: Only works with single `repos`
To scan `src` only:
```
./secret-scanner -repos jquery/jquery -sub-dir src
```
## Scan Results as Output
By default, findings found during the scan will be printed as console output. You can save it as JSON to path by specifying the `output` param
```
./secret-scanner -repos jquery/jquery -output ~/report.json
```
The output file will contain the lines containing the potential secrets. In circumstances where you do not want to expose them, you can specify `-log-secret=false`
## Scan State
By default, no scan state is being kept, meaning every scan on the same repository will start afresh.
If scan state is enabled, the scanner will save the latest scan session and commit hash in JSON format. From the next scan onwards for the same repository,the scanner will only scan changes since the last saved commit hash.
The default location of scan state JSON file is in `~/.secret-scanner/`.
```
./secret-scanner -repos jquery/jquery -use-state=true
```
## CLI Args
```
-baseurl string
Specify Git provider base URL
-commit-depth int
Number of repository commits to process (default 500)
-debug
Print debugging information
-env string
.env file path containing Git provider base URLs and tokens
-git string
Name of git provider (Eg. github, gitlab, bitbucket) (default "github")
-load string
Load session file
-dir string
Specify the local git repo path to scan
-log-secret
If true, the matched secret will be included in output file (default true)
-output string
Save session to file
-repos string
Comma-separated list of repos to scan
-sub-dir string
Sub-directory within the repository to scan
-quiet
Suppress all output except for errors
-skip-tests
Skips possible test contexts (default true)
-use-state
If use-state is off, every scan will be treated as a brand new scan.
-threads int
Number of concurrent threads (default number of logical CPUs)
-token string
Specify Git provider token
```
## Credits
Project is built upon the ground work laid in Gitrob by Michael Henriksen.
And many secret signatures was taken from shhgit.