{"id":13537808,"url":"https://github.com/srcclr/commit-watcher","last_synced_at":"2026-01-16T16:26:11.214Z","repository":{"id":70008038,"uuid":"52403819","full_name":"srcclr/commit-watcher","owner":"srcclr","description":"Find interesting and potentially hazardous commits in git projects","archived":false,"fork":false,"pushed_at":"2018-09-06T01:06:34.000Z","size":262,"stargazers_count":351,"open_issues_count":0,"forks_count":41,"subscribers_count":47,"default_branch":"master","last_synced_at":"2024-11-03T02:33:11.062Z","etag":null,"topics":["auditing","database","docker","email-notification","favor","rails","rds","regular-expression","security-vulnerability","watch"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/srcclr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-02-24T01:09:48.000Z","updated_at":"2024-04-15T10:44:05.000Z","dependencies_parsed_at":"2023-02-22T19:00:18.438Z","dependency_job_id":null,"html_url":"https://github.com/srcclr/commit-watcher","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srcclr%2Fcommit-watcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srcclr%2Fcommit-watcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srcclr%2Fcommit-watcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srcclr%2Fcommit-watcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/srcclr","download_url":"https://codeload.github.com/srcclr/commit-watcher/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246757344,"owners_count":20828884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auditing","database","docker","email-notification","favor","rails","rds","regular-expression","security-vulnerability","watch"],"created_at":"2024-08-01T09:01:03.891Z","updated_at":"2026-01-16T16:26:11.155Z","avatar_url":"https://github.com/srcclr.png","language":"Ruby","readme":"# Commit Watcher\n\nCommit Watcher finds interesting and potentially hazardous commits in git projects. Watch your own projects to make sure you didn't accidentally leak your AWS keys or other credentials, and watch open-source projects you use to find undisclosed security vulnerabilities and patches.\n\nAt [SourceClear](https://www.sourceclear.com/), we want to help you use open-source software safely. Oftentimes when a security vulnerability is discovered and fixed in an open-source project, there isn't a public disclosure about it. In part, this is because the [CVE](https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures) process is onerous and labor intensive, and notifying all the users of a project isn't possible.\n\nOh, and about that UI. Commit Watcher is intended to be an API accessible backend service. The UI is only there for testing, and the scope of functionality is limited to collecting commits and auditing them against a set of rules.\n\n## Contributing\n\nCheck out the dozens of rules and patterns in the [srcclr/commit-watcher-rules](https://github.com/srcclr/commit-watcher-rules) repository that help find leaked credentials and potential security issues. Just open an issue or PR in that repo if there's a rule you'd like to see added.\n\nAdditionally, if you find a security issue on an open-source project using Commit Watcher, our security research team would love to help verify it. You can open an issue against this repo from the UI, or just drop a link to the offending commit in a [new issue](https://github.com/srcclr/commit-watcher/issues/new).\n\n## Setup\n\nInstall and configure Ruby using RVM or Rbenv. Avoid using the system's bundled Ruby to avoid permission issues during installation/setup.\n\n```bash\nRVM: https://rvm.io\nRbenv: https://github.com/rbenv/rbenv\n```\n\nInstall MySQL and Redis. On Mac, with Brew, you can do that with this command:\n\n```bash\nbrew install mysql redis\n```\n\nFollow the instructions Brew gives you so the services are started properly.\n\nInstall gem dependencies:\n\n```bash\ngem install bundler\nbundle install\n```\n\nThen setup some Rails secrets and passwords:\n\n```bash\nfigaro install\necho \"COMMIT_WATCHER_DATABASE_PASSWORD: 'changeme123'\" \u003e\u003e config/application.yml\necho \"SECRET_KEY_BASE: `rake secret`\" \u003e\u003e config/application.yml\n```\n\nThe rest of the setup depends on how you want to run Commit Watcher. You can either run it locally, which is good for quick development, or you can run it with Docker.\n\n#### *Optional:* Configuring Email Notifications\n\nTo use email notifications, set your Gmail username and password with these commands:\n\n```bash\necho \"GMAIL_USERNAME: 'sah.dude@gmail.com'\" \u003e\u003e config/application.yml\necho \"GMAIL_PASSWORD: 'urpassbro'\" \u003e\u003e config/application.yml\n```\n\nIf you'd like to use another email provider other than Gmail, you'll have to change these two files: [`config/environments/development.rb`](config/environments/development.rb) and [`config/environments/production.rb`](config/environments/production.rb).\n\n### Running Locally\n\nCreate the database, load the schema, and seed it with some sample rules:\n\n```bash\nrails db:setup\n```\n\nNow you're ready to start Rails with:\n\n```bash\nrails s\n```\n\nTo start processing jobs, in another terminal:\n\n```bash\nbundle exec sidekiq\n```\n\n### Running with Docker / RDS\n\nFirst, change the root and user passwords in [`.env.db`](.env.db).\n\n```\n# Not used but should set one for security.\nMYSQL_ROOT_PASSWORD=changeme123\n\n# This is for the commit_watcher user.\nMYSQL_PASSWORD=changeme123\n```\n\nSecond, modify [`config/database.yml`](config/database.yml) by commenting out `socket` in favor of `host`, like this:\n\n```yaml\n  # Use this for local mysql instances\n  #socket: /tmp/mysql.sock\n\n  # Use this for Docker\n  host: db\n```\n\nAlternatively, for RDS, setup the external RDS URL:\n\n```bash\necho \"COMMIT_WATCHER_EXTERNAL_DATABASE_URL: 'somedb.rds.amazonaws.com'\" \u003e\u003e config/application.yml\n```\n\nThen, modify [`config/database.yml`](config/database.yml) by commenting out `socket` in favor of `host`, like this:\n\n```yaml\n  # Use this for local mysql instances\n  #socket: /tmp/mysql.sock\n\n  # Use this for Docker\n  #host: db\n\n  # Use this for External RDS\n  host: \u003c%= ENV['COMMIT_WATCHER_EXTERNAL_DATABASE_URL'] %\u003e\n```\n\nAnd modify [`docker-compose.yml`](docker-compose.yml) by commenting out `- db` in the `web:` and `sidekiq:` sections, like this:\n\n```yaml\n  web:\n    build: .\n    volumes:\n      - .:/myapp\n    ports:\n      - '3000:3000'\n    links:\n      #- db\n      - redis\n    ...\n    sidekiq:\n      build: .\n      volumes:\n        - .:/myapp\n      links:\n        #- db\n        - redis\n```\n\nNow start everything going with:\n\n```bash\ndocker-compose up\n```\n\nThis downloads the images and builds the database and rails app containers. When it's finished building, and both containers are running, you should see rails messages like this:\n\n```\n77bcf6cd5a_commitwatcher_web_1 | [2016-03-09 18:29:36] INFO  WEBrick 1.3.1\n77bcf6cd5a_commitwatcher_web_1 | [2016-03-09 18:29:36] INFO  ruby 2.2.2 (2015-04-13) [x86_64-linux]\n77bcf6cd5a_commitwatcher_web_1 | [2016-03-09 18:29:36] INFO  WEBrick::HTTPServer#start: pid=1 port=3000\n```\n\nStop Docker with `Ctrl+C` so the database can be setup with:\n\n```bash\ndocker-compose run web bundle exec rake db:schema:load db:seed\n```\n\nNow start everything up again with:\n\n```bash\ndocker-compose up\n```\n\n## Use\n\nIf using Docker, the server will be accessible from the IP address given by:\n\n```bash\ndocker-machine ip default\n```\n\nTo crawl any projects, you must set a [GitHub API token](https://github.com/settings/tokens) in the default configuration. This can be reached here: [http://localhost:3000/configurations/1/edit](http://localhost:3000/configurations/1/edit).\n\nThe web UI contains a dashboard which links to all available pages. It's located here: [http://localhost:3000/](http://localhost:3000/).\n\nSidekiq dashboard is here: [http://localhost:3000/sidekiq/cron](http://localhost:3000/sidekiq/cron).\n\n### Overview\n\nThe process starts by every few minutes any project which hasn't been checked in a while is polled for new commits. These commits are then checked against whatever rules are setup for the project. Any commits which match are recorded and available at the `/commits` endpoint.\n\nEverything is broken up into different Sidekiq jobs. There are three:\n\n1. Selecting projects which need to be polled\n2. Collecting new commits\n3. Auditing a single commit\n\n### API Access\n\nThe API endpoints are similar to the web UI and are documented by code.\n\nThe app must have a hostname to access the API endpoints. This can be done in development by adding a record to the host file:\n\n```bash\necho \"127.0.0.1 api.my_app.dev\" \u003e\u003e /etc/hosts\n```\n\nThen the API can be accessed by:\n```bash\ncurl http://api.my_app.dev:3000/v1/commits\n```\n\n## Rules\n\nRule types are defined and described in [config/rule_types.yml](config/rule_types.yml). They are:\n\n* `filename_pattern` - Regular expression for a filename\n* `changed_code_pattern` - Regular expression for a changed line\n* `code_pattern` - Regular expression for any code in a changed file\n* `message_pattern` - Regular expression for a commit message\n* `author_pattern` - Regular expression for a commit author name, normalized to \"name \u003cemail\u003e\"\n* `commit_pattern` - Combination of code_pattern and message_pattern\n* `expression` - Boolean expression referencing one or more rules\n\n### Expression Rules\n\nThis is a special rule type that allows for combining multiple rules in a boolean expression. The boolean expression has three operators: `\u0026\u0026` (and), `||` (or), `!` (not), and also allows for parenthetical expressions.\n\nFor example, if there are three rules:\n\n1. `is_txt` - `/\\.txt\\z/` (filename_pattern)\n2. `has_lulz_msg` - `/\\blulz\\b/` (message_pattern)\n3. `has_42` - `/\\b42\\b/` (code_pattern)\n\nTo create an expression rule which would match commits that include \"lulz\" in the commit message and contains at least a single text file _or_ has a file with the word \"42\":\n\n```\n(is_txt \u0026\u0026 has_lulz_msg) || has_42\n```\n\nTo match a commit where any file is not a text file and includes \"42\":\n\n```\n!is_txt \u0026\u0026 has_42\n```\n\n## Publications \n\n[Automated identification of security issues from commit messages and bug reports](https://dl.acm.org/citation.cfm?id=3117771), FSE 2017\n","funding_links":[],"categories":["\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e新添加的","\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e工具","Ruby"],"sub_categories":["\u003ca id=\"31185b925d5152c7469b963809ceb22d\"\u003e\u003c/a\u003e新添加的"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrcclr%2Fcommit-watcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrcclr%2Fcommit-watcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrcclr%2Fcommit-watcher/lists"}