{"id":27948473,"url":"https://github.com/twilio-labs/deadshot","last_synced_at":"2025-05-07T14:58:54.107Z","repository":{"id":39585064,"uuid":"325103512","full_name":"twilio-labs/deadshot","owner":"twilio-labs","description":"Deadshot is a Github pull request scanner to identify sensitive data being committed to a repository","archived":false,"fork":false,"pushed_at":"2024-10-27T21:22:29.000Z","size":81,"stargazers_count":191,"open_issues_count":36,"forks_count":13,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-07T14:58:48.109Z","etag":null,"topics":["automation","credentials","git","github","pull-requests","scanning","secrets","security","sensitive-data","sensitive-data-security","vulnerabilities"],"latest_commit_sha":null,"homepage":"https://www.twilio.com/blog/introducing-twilio-deadshot","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twilio-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-28T19:51:52.000Z","updated_at":"2025-02-01T01:15:03.000Z","dependencies_parsed_at":"2023-01-30T19:30:53.376Z","dependency_job_id":null,"html_url":"https://github.com/twilio-labs/deadshot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twilio-labs%2Fdeadshot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twilio-labs%2Fdeadshot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twilio-labs%2Fdeadshot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twilio-labs%2Fdeadshot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twilio-labs","download_url":"https://codeload.github.com/twilio-labs/deadshot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252902615,"owners_count":21822258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","credentials","git","github","pull-requests","scanning","secrets","security","sensitive-data","sensitive-data-security","vulnerabilities"],"created_at":"2025-05-07T14:58:53.444Z","updated_at":"2025-05-07T14:58:54.088Z","avatar_url":"https://github.com/twilio-labs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deadshot\nDeadshot is a Pull Request scanner that looks for the introduction of secrets via PRs by matching each diff line against a set of known secret expressions.\n\n## Application capabilities:\nService is responsible for:\n- Real-time Pull Request diff processor to check for secrets being committed to Github via new code\n- Notify the user on the PR conversation if it flags something\n- Slack notify the security team channel when certain secrets are identified in code for which you've enabled slack notifications via a flag in regex.json\n\nService does NOT:\n- Do any static or dynamic code analysis\n\n## How does it work?\nDeadshot is a Flask-Celery-Redis multi-container application that is installed as a Github app to run on every Pull Request created against the main branch of a repo on which the Github app is installed.\n\nThe Flask container is the entry point for the service by exposing API routes defined in blueprints.py. Once a Pull request payload is received on the API route the service forwards the payload to a Redis queue for the Celery container to pick up and\nscan through the diff of the Pull Request. After the celery container scans for specified secrets regular expressions, it comments on PRs, slack notifies the security team channel, or creates a JIRA ticket for the team to follow up on.\nThe Github app is configured with the Flask API URL and a shared secret used for generating the payload SHA checksum.\n\nOne way the API URL can be setup is by deploying this code on an host and assigning a application load balancer to this host.\n\n### Creating a Github App\nNote: When creating the app please make sure you have a DNS ready for host on which you'll be deploying Deadshot containers and a secure secret string for the webhook secret.\n\nGithub Admins would need to create and install a Github app on Github before running or deploying the Deadshot application.\nTo know more about creating a Github app please read this [guide](https://docs.github.com/en/free-pro-team@latest/developers/apps/creating-a-github-app)\n\nApp Name: deadshot (All lower case. This is important as the service uses this name to fetch previous comments it has made on a PR)\n\nWebhook URL: http(s)://your-hosted-deadshot-dns/api/v1/deadshot-webhook\n\nTo test this locally you can create a ngrok endpoint to feed into your Github app webhook section\n\n### Github App Permissions\nFor this application to work your Github app will have to enable the following permissions and subscriptions on the permissions page of the Github app:\nRepository Permissions:\n- Metadata: Read-only\n- PullRequests: Read \u0026 write\n- Webhooks: Read \u0026 write\n\nAll other permissions are left unchanged to the default value of No access\n\nSubscribe to events:\n- Pull request\n- Pull request review\n\nFinally click “Create GitHub App”. After successful app creation follow the “generate a private key” link in the top section of the app web page\n\n\nOnce the private key is generated store it in a secure location.\nThis generated private key is one of the pieces of data used to generate a session token for app interaction.\n\nAfter generating the private key, install the app on all the orgs you want it to monitor.\n\n## Running Deadshot\nThis is a multi-container application designed to bring up all three containers (Flask, Celery, Redis) via the /bin/run.sh, so running the Dockerfile image should bring up the entirety of the application\n\n### Environment variables:\n#### Note: For deployment using docker-compose.yaml populate the these environment variables in [localdev.env](https://github.com/twilio-labs/deadshot/blob/main/configuration/environment/localdev.env). If you're deploying this by building and running each container image individually via Dockerfile.api, Dockerfile.celery then the these environment variables are in the respective Dockerfiles\nThe three variables below are single string values provided by the user\n- GITHUB_URL: This is the URL behind which your Github instance is accessed. Please provide the DNS without scheme or port. Eg. if your Github web URL is https://github.mockcompany.com then provide the value github.mockcompany.com\n- GITHUB_API: This is the API URL for Github. Eg. if you have your Github DNS as https://github.mockcompany.com then your API would be something like https://github.mockcompany.com/api/v3\n- JIRA_SERVER= Your company's JIRA server web URL\n\nThe below environment variables load path to files with credentials in them. Load the json file key values in the files available [here](https://github.com/twilio-labs/deadshot/tree/main/local_dev_secrets) before running the application. \n- SECRET_GITHUB_SECRET: This variable loads github_secrets.json and has the Github app's shared webhook secret, integration ID, and the pem key. All these three secrets are obtained from the Github app settings page\n  webhook secret - This is the secret configured during the app creation process\n  integration ID - This is the app ID shown on the github app settings page\n  pem key - this is the private key generated during the app installation process\n- SECRET_SLACK_WEBHOOKS: This slack_webhook.json and has the webhook URL to which the deadshot app will send slack notifications when it finds secrets in a PR for which you set slack_alert=True in regex.json\n- SECRET_JIRA_AUTH: This loads jira_user.json and has the username and password for the user ID to access the org's JIRA board\n  Note: If you do not provide valid values in SECRET_SLACK_WEBHOOKS and SECRET_JIRA_AUTH the service will soft fail and print error messages about failure to initiate slack and jira methods in the docker container logs \n\nNote: If you do not move the JSON secrets files location then you do not need to update the above three environment variables values already present in the Dockerfiles or docker-compose.yaml\n\n### Running/Serving the Docker Image\nThis command will use docker-compose.yaml to bring up all the containers. Please update configuration/environment/localdev.env with values relevant to your organisation before running the below command\n```bash\nmake serve\n```\nOnce you’ve done this and do not intend to use Dockerfile for serving the application then jump to “Server Healthcheck” section\n\n### Building and running the service using Dockerfiles\nThere are two ways to build and run the Dockerfiles. There are four Dockerfiles present in the repository, three of which are used to generate an individual image for each container needed for this service to work, and the fourth one is a Dockerfile setup to create a image that can be used to either bring up the Flask application or the celery worker depending on the DEADSHOT_RUN_MODE environment variable value (api or worker) provided\nTo run any of the steps below you need to be present in the root folder of the repository\n\nNote: Ensure you’ve updated the environment variables in Dockerfile.api and Dockerfile.celery files\n\n\n#### Building images from individual Dockerfiles\nThere are three Dockerfiles relevant to this step. Dockerfile.api, Dockerfile.celery, and Dockerfile.redis\n\n###### To build the Flask API image\n```\ndocker build -f Dockerfile.api -t deadshot-api:\u003cversion\u003e .\n```\n\n###### To build the celery image\n```\ndocker build -f Dockerfile.celery -t deadshot-worker:\u003cversion\u003e .\n```\n\n###### To build the redis image\n```\ndocker build -f Dockerfile.redis -t deadshot-redis:\u003cversion\u003e .\n```\n\n#### Running built images\nThe three images built in the previous steps all run in separate networks due to which they won't be able to talk to each other. To enable inter-container communications we need to add them to a container network\n\n##### Create a docker network\n```\ndocker network create deadshot-network\n```\nRun the images using the created network in the following order:\nStart redis container:\n```\ndocker run --net deadshot-network --name redis deadshot-redis:\u003cversion\u003e\n```\n\nStart celery container:\n```\ndocker run --net deadshot-network deadshot-worker:\u003cversion\u003e\n```\n\nStart Flask API container:\n```\ndocker run --net deadshot-network -p 9001:9001 deadshot-api:\u003cversion\u003e\n```\n\n### Building and running a single image for Flask API container and celery worker container\n#### This step is useful only if you have a orchestration that allows you to feed in environment variables, secrets and other configurations at deployment time. Please use the above method of running the containers if you don't have a configurable CI/CD setup.\nTo build a single docker image for bringing up the api and celery worker based on DEADSHOT_RUN_MODE environment variable\n```bash\nmake build\n```\nThis command will also create the redis image that is needed for service\n\nIf the built image is run with the environment variable DEADSHOT_RUN_MODE=api, it will bring up the Flask application\nIf the image is run with environment variable DEADSHOT_RUN_MODE=worker then the celery worker will be initiated\n\n### Server Healthcheck\nNow that the API is ready to receive requests navigating to `http://localhost:9001/api/v1/heartbeat` in a browser should return a valid response or you could do a curl\n```bash\ncurl localhost:9001/api/v1/healthcheck\n```\nBoth should show the following message:\n`{\"healthcheck\": \"ready\"}`\n\n### Running a Pull Request scan\nIf you have a webhook payload of the Github app for your Pull Request then you can run the following curl command locally to test your application:\n```bash\ncurl -X POST -H \"content-type: application/json\" -H \"X-GitHub-Enterprise-Host: github.mockcompany.com\" -H \"X-Hub-Signature: sha1=85df4936c6396c149be94144befab41168149840\" -H \"X-GitHub-Event: pull_request\" -d @tests/fixtures/good_pr.json http://localhost:9001/api/v1/deadshot-webhook\n```\n## Adding new regular expressions\nIf you want the tool to monitor other types of secrets then add your regular expressions in the [regex.json](https://github.com/twilio-labs/deadshot/blob/main/deadshot/services/scanner/secrets_json/regex.json) file\n\nNote: Entropy check flag allows you to look for high entropy findings in addition to the regular expression match\n\n## Limitations\nAt this time, Deadshot has only tested with Github Enterprise, but should work with Github cloud as well.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwilio-labs%2Fdeadshot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwilio-labs%2Fdeadshot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwilio-labs%2Fdeadshot/lists"}