{"id":13619001,"url":"https://github.com/PaperMtn/gitlab-watchman","last_synced_at":"2025-04-14T15:33:54.498Z","repository":{"id":42522280,"uuid":"295512149","full_name":"PaperMtn/gitlab-watchman","owner":"PaperMtn","description":"Finding exposed secrets and personal data in GitLab","archived":false,"fork":false,"pushed_at":"2024-11-04T19:11:27.000Z","size":176,"stargazers_count":195,"open_issues_count":1,"forks_count":27,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-04T20:23:50.243Z","etag":null,"topics":["blue-team","blueteam","cybersecurity","data-loss-prevention","dlp","gitlab","gitlab-api","gitlab-watchman","infosec","monitoring","purple-team","purpleteam","red-team","redteam","tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PaperMtn.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-09-14T19:03:02.000Z","updated_at":"2024-08-26T22:50:33.000Z","dependencies_parsed_at":"2024-01-16T00:22:43.131Z","dependency_job_id":"986d4c58-b4d6-41c1-8331-aefcac306e93","html_url":"https://github.com/PaperMtn/gitlab-watchman","commit_stats":{"total_commits":53,"total_committers":3,"mean_commits":"17.666666666666668","dds":0.05660377358490565,"last_synced_commit":"d06cd2db2e2bb7699b2242c30cc73db162640d29"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgitlab-watchman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgitlab-watchman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgitlab-watchman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaperMtn%2Fgitlab-watchman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PaperMtn","download_url":"https://codeload.github.com/PaperMtn/gitlab-watchman/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223066150,"owners_count":17082015,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blue-team","blueteam","cybersecurity","data-loss-prevention","dlp","gitlab","gitlab-api","gitlab-watchman","infosec","monitoring","purple-team","purpleteam","red-team","redteam","tools"],"created_at":"2024-08-01T21:00:33.580Z","updated_at":"2025-04-14T15:33:54.491Z","avatar_url":"https://github.com/PaperMtn.png","language":"Python","readme":"\u003cimg src=\"https://i.imgur.com/6uh3Gh4.png\" width=\"550\"\u003e\n\n# GitLab Watchman\n![Python 2.7 and 3 compatible](https://img.shields.io/pypi/pyversions/gitlab-watchman)\n![PyPI version](https://img.shields.io/pypi/v/gitlab-watchman.svg)\n![License: MIT](https://img.shields.io/pypi/l/gitlab-watchman.svg)\n\n## About GitLab Watchman\n\nGitLab Watchman is an application that uses the GitLab API to detect exposed secrets and personal data. It also enumerates the GitLab instance for any useful information.\n\n### Features\n\n#### Secrets Detection\nIt searches GitLab for internally shared projects and looks at:\n- Code\n- Commits\n- Wiki pages\n- Issues\n- Merge requests\n- Milestones\n- Notes\n- Snippets\n\nFor the following data:\n- GCP keys and service account files\n- AWS keys\n- Azure keys and service account files\n- Google API keys\n- Slack API tokens \u0026 webhooks\n- Private keys (SSH, PGP, any other misc private key)\n- Exposed tokens (Bearer tokens, access tokens, client_secret etc.)\n- S3 config files\n- Tokens for services such as Heroku, PayPal and more\n- Passwords in plaintext\n- and more\n\n##### Time based searching\nYou can run GitLab Watchman to look for results going back as far as:\n- 24 hours\n- 7 days\n- 30 days\n- All time\n\nThis means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.\n\n#### Enumeration\nGitLab Watchman can enumerate potentially useful information from a GitLab instance:\n- Instance metadata\n- Information on the calling user/token being used\n- Output all users to CSV file\n- Output all projects to CSV file\n- Output all groups to CSV file\n\n### Signatures\nGitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures). Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.\n\n#### Suppressing Signatures\nYou can define signatures that you want to disable when running GitLab Watchman by adding their IDs to the `disabled_signatures` section of the `watchman.conf` file. For example:\n\n```yaml\ngitlab_watchman:\n  disabled_signatures:\n    - tokens_generic_bearer_tokens\n    - tokens_generic_access_tokens\n```\n\nYou can find the ID of a signature in the individual YAML files in [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures).\n\n### Logging\n\nGitLab Watchman gives the following logging options:\n- Terminal-friendly Stdout\n- JSON to Stdout\n\nGitLab Watchman defaults to terminal-friendly stdout logging if no option is given. This is designed to be easier for humans to read.\n\nJSON logging is also available, which is perfect for ingesting into a SIEM or other log analysis platforms.\n\nJSON formatted logging can be easily redirected to a file as below:\n```commandline\ngitlab-watchman --timeframe a --all --output json \u003e\u003e gitlab_watchman_log.json \n```\n\n## Requirements\n\n### GitLab versions\nGitLab Watchman uses the v4 API, and works with GitLab Enterprise Edition versions:\n- 13.0 and above - Yes\n\n- GitLab.com - Yes\n- 12.0 - 12.10 - Maybe, untested but if using v4 of the API then it could work\n\n### GitLab Licence \u0026 Elasticsearch\nTo search the scopes:\n- blobs\n- wiki_blobs\n- commits\n\nThe GitLab instance must have [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) configured, and be running Enterprise Edition with a minimum GitLab Starter or Bronze Licence.\n\n### GitLab personal access token\nTo run GitLab Watchman, you will need a GitLab personal access token.\n\nYou can create a personal access token in the GitLab GUI via Settings -\u003e Access Tokens -\u003e Add a personal access token\n\nThe token needs permission for the following scopes:\n```\napi\n```\n\n**Note**: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will have access to your private repositories.\n\n### GitLab URL\n\nYou also need to provide the URL of your GitLab instance.\n\n#### Providing token \u0026 URL\nGitLab Watchman will get the GitLab token and URL from the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`.\n\n### watchman.conf file\nConfiguration options can be passed in a file named `watchman.conf` which must be stored in your home directory. The file should follow the YAML format, and should look like below:\n```yaml\ngitlab_watchman:\n  disabled_signatures:\n    - tokens_generic_bearer_tokens\n    - tokens_generic_access_tokens\n```\nGitLab Watchman will look for this file at runtime, and use the configuration options from here.\n\n## Installation\nYou can install the latest stable version via pip:\n\n`python3 -m pip install gitlab-watchman`\n\nOr build from source yourself. Download the release source files, then from the top level repository run:\n```shell\npython3 -m build\npython3 -m pip install --force-reinstall dist/*.whl\n```\n\n## Docker Image\n\nGitLab Watchman is also available from the Docker hub as a Docker image:\n\n`docker pull papermountain/gitlab-watchman:latest`\n\nYou can then run GitLab Watchman in a container, making sure you pass the required environment variables:\n\n```\n// help\ndocker run --rm papermountain/gitlab-watchman -h\n\n// scan all\ndocker run --rm -e GITLAB_WATCHMAN_TOKEN=abc123 -e GITLAB_WATCHMAN_URL=https://example.gitlab.com papermountain/gitlab-watchman --timeframe a --all\ndocker run --rm --env-file .env papermountain/gitlab-watchman --timeframe a --all\n```\n\n## Usage\nGitLab Watchman will be installed as a global command, use as follows:\n```\nusage: gitlab-watchman [-h] --timeframe {d,w,m,a} [--output {json,stdout}] [--version] [--all] [--blobs] [--commits] [--wiki-blobs] [--issues]\n                   [--merge-requests] [--milestones] [--notes] [--snippets] [--enumerate] [--debug] [--verbose]\n\nFinding exposed secrets and personal data in GitLab\n\noptions:\n  -h, --help            show this help message and exit\n  --output {json,stdout}, -o {json,stdout}\n                        Where to send results\n  --version, -v         show program's version number and exit\n  --all, -a             Find everything\n  --blobs, -b           Search code blobs\n  --commits, -c         Search commits\n  --wiki-blobs, -w      Search wiki blobs\n  --issues, -i          Search issues\n  --merge-requests, -mr\n                        Search merge requests\n  --milestones, -m      Search milestones\n  --notes, -n           Search notes\n  --snippets, -s        Search snippets\n  --enumerate, -e       Enumerate this GitLab instance for users, groups, projects.Output will be saved to CSV files\n  --debug, -d           Turn on debug level logging\n  --verbose, -V         Turn on more verbose output for JSON logging. This includes more fields, but is larger\n\nrequired arguments:\n  --timeframe {d,w,m,a}\n                        How far back to search: d = 24 hours w = 7 days, m = 30 days, a = all time\n\n  ```\n\n## Other Watchman apps\nYou may be interested in the other apps in the Watchman family:\n- [Slack Watchman](https://github.com/PaperMtn/slack-watchman)\n- [Slack Watchman for Enterprise Grid](https://github.com/PaperMtn/slack-watchman-enterprise-grid)\n- [GitHub Watchman](https://github.com/PaperMtn/github-watchman)\n\n## License\nThe source code for this project is released under the [GNU General Public Licence](https://www.gnu.org/licenses/licenses.html#GPL). This project is not associated with GitLab.\n","funding_links":[],"categories":["Communication and Collaboration Tools","Python","Python (1887)","相关工具","Инструменты"],"sub_categories":["Version Control, Wiki, Knowledge base","隐私相关领域法规/条例","Поиск секретов"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaperMtn%2Fgitlab-watchman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPaperMtn%2Fgitlab-watchman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaperMtn%2Fgitlab-watchman/lists"}