{"id":13820903,"url":"https://github.com/PaperMtn/github-watchman","last_synced_at":"2025-05-16T10:33:28.677Z","repository":{"id":56320720,"uuid":"297713046","full_name":"PaperMtn/github-watchman","owner":"PaperMtn","description":"Monitoring GitHub for sensitive data shared publicly","archived":false,"fork":false,"pushed_at":"2021-12-20T21:59:57.000Z","size":38,"stargazers_count":66,"open_issues_count":1,"forks_count":7,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-15T22:40:23.005Z","etag":null,"topics":["blue-team","blueteam","cybersecurity","data-loss-prevention","dlp","github","github-api","infosec","monitoring","purple-team","purpleteam","red-team","redteam","tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PaperMtn.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-22T16:48:50.000Z","updated_at":"2024-08-26T22:50:55.000Z","dependencies_parsed_at":"2022-08-15T16:40:18.471Z","dependency_job_id":null,"html_url":"https://github.com/PaperMtn/github-watchman","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgithub-watchman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgithub-watchman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgithub-watchman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgithub-watchman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PaperMtn","download_url":"https://codeload.github.com/PaperMtn/github-watchman/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254512897,"owners_count":22083478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blue-team","blueteam","cybersecurity","data-loss-prevention","dlp","github","github-api","infosec","monitoring","purple-team","purpleteam","red-team","redteam","tools"],"created_at":"2024-08-04T08:01:11.318Z","updated_at":"2025-05-16T10:33:28.654Z","avatar_url":"https://github.com/PaperMtn.png","language":"Python","readme":"\u003cimg src=\"https://i.imgur.com/4lNLwdV.png\" width=\"550\"\u003e\n\n# GitHub Watchman\n![Python 2.7 and 3 compatible](https://img.shields.io/pypi/pyversions/github-watchman)\n![PyPI version](https://img.shields.io/pypi/v/github-watchman.svg)\n![License: MIT](https://img.shields.io/pypi/l/github-watchman.svg)\n\n## About GitHub Watchman\n\nGitHub Watchman is an application that uses the GitHub API to audit GitHub for sensitive data and credentials exposed internally.\n\n### Features\nIt searches GitHub for internally shared projects and looks at:\n- Code\n- Commits\n- Issues\n- Repositories\n\nFor the following data:\n- GCP keys and service account files\n- AWS keys\n- Azure keys and service account files\n- Google API keys\n- Slack API tokens \u0026 webhooks\n- Private keys (SSH, PGP, any other misc private key)\n- Exposed tokens (Bearer tokens, access tokens, client_secret etc.)\n- S3 config files\n- Passwords in plaintext\n- and more\n\n#### Time based searching\nYou can run GitHub Watchman to look for results going back as far as:\n- 24 hours\n- 7 days\n- 30 days\n- All time\n\nThis means after one deep scan, you can schedule GitHub Watchman to run regularly and only return results from your chosen timeframe.\n\n### Rules\nGitHub Watchman uses custom YAML rules to detect matches in GitHub.\n\nThey follow this format:\n\n```yaml\n---\nfilename:\nenabled: #[true|false]\nmeta:\n  name:\n  author:\n  date:\n  description: #what the search should find#\n  severity: #rating out of 100#\nscope: #what to search, any combination of the below#\n- code\n- commits\n- issues\n- repositories\ntest_cases:\n  match_cases:\n  - #test case that should match the regex#\n  fail_cases:\n  - #test case that should not match the regex#\nstrings:\n- #search query to use in GitHub#\npattern: #Regex pattern to filter out false positives#\n```\nThere are Python tests to ensure rules are formatted properly and that the Regex patterns work in the `tests` dir\n\nMore information about rules, and how you can add your own, is in the file `docs/rules.md`.\n\n\n### Logging\n\nGitHub Watchman gives the following logging options:\n- CSV\n- Log file\n- Stdout\n- TCP stream\n\nWhen using CSV logging, searches for rules are returned in separate CSV files, for all other methods of logging, results are output in JSON format, perfect for ingesting into a SIEM or other log analysis platform.\n\nFor file and TCP stream logging, configuration options need to be passed via `.conf` file or environment variable. See the file `docs/logging.md` for instructions on how to set it up.\n\nIf no logging option is given, GitHub Watchman defaults to CSV logging.\n\n## Requirements\n\n### GitHub versions\nGitHub Watchman uses the v3 API, and works with GitHub Enterprise Server versions that support the v3 API.\n\nGitHub Watchman also works with GitHub.com (Free, Pro and Team) using the API.\n\n### GitHub personal access token\nTo run GitHub Watchman, you will need a GitHub personal access token.\n\nYou can create a personal access token in the GitHub GUI via Settings -\u003e Developer settings -\u003e Personal access tokens\n\nThe token needs no specific scopes assigned, as it searches public repositories in the GitHub instance.\n\n**Note**: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will have access to your private repositories.\n\n### GitHub URL\n\nYou also need to provide the URL of your GitHub instance.\n\n#### Providing token \u0026 URL\nGitHub Watchman will first try to get the the GitHub token and URL from the environment variables `GITHUB_WATCHMAN_TOKEN` and `GITHUB_WATCHMAN_URL`, if this fails they will be taken from .conf file (see below).\n\n### .conf file\nConfiguration options can be passed in a file named `watchman.conf` which must be stored in your home directory. The file should follow the YAML format, and should look like below:\n```yaml\ngithub_watchman:\n  token: abc123\n  url: https://github.example.com\n  logging:\n    file_logging:\n      path:\n    json_tcp:\n      host:\n      port:\n```\nGitHub Watchman will look for this file at runtime, and use the configuration options from here. If you are not using the advanced logging features, leave them blank.\n\nIf you are having issues with your .conf file, run it through a YAML linter.\n\nAn example file is in `docs/example.conf`\n\n**Note** If you use any other Watchman applications and already have a `watchman.conf` file, just append the conf data for GitHub Watchman to the existing file.\n\n## Installation\nInstall via pip\n\n`pip install github-watchman`\n\nOr via source\n\n## Usage\nGitHub Watchman will be installed as a global command, use as follows:\n```\nusage: github-watchman [-h] --timeframe {d,w,m,a} --output\n                   {csv,file,stdout,stream} [--version] [--all] [--code]\n                   [--commits] [--issues] [--repositories]\n\nMonitoring GitHub for sensitive data shared publicly\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             show program's version number and exit\n  --all                 Find everything\n  --code                Search code\n  --commits             Search commits\n  --issues              Search issues\n  --repositories        Search merge requests\n\nrequired arguments:\n  --timeframe {d,w,m,a}\n                        How far back to search: d = 24 hours w = 7 days, m =\n                        30 days, a = all time\n  --output {csv,file,stdout,stream}\n                        Where to send results\n\n\n  ```\n\nYou can run GitHub Watchman to look for everything, and output to default CSV:\n\n`github-watchman --timeframe a --all`\n\nOr arguments can be grouped together to search more granularly. This will look for commits and milestones for the last 30 days, and output the results to a TCP stream:\n\n`github-watchman --timeframe m --commits --milestones --output stream`\n\n## Other Watchman apps\nYou may be interested in some of the other apps in the Watchman family:\n- [Slack Watchman](https://github.com/PaperMtn/slack-watchman)\n- [GitLab Watchman](https://github.com/PaperMtn/gitlab-watchman)\n\n## License\nThe source code for this project is released under the [GNU General Public Licence](https://www.gnu.org/licenses/licenses.html#GPL). This project is not associated with GitHub.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaperMtn%2Fgithub-watchman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPaperMtn%2Fgithub-watchman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaperMtn%2Fgithub-watchman/lists"}