{"id":27871176,"url":"https://github.com/ventz/pii-scan","last_synced_at":"2025-05-04T23:33:59.323Z","repository":{"id":290128741,"uuid":"973456560","full_name":"ventz/pii-scan","owner":"ventz","description":"A lightweight, highly configurable tool for scanning files for personally identifiable information (PII) and other sensitive data. It uses AWS Comprehend and custom regex patterns, supports fast multi-threaded execution, and can run as a standalone tool or as a GitHub Action.","archived":false,"fork":false,"pushed_at":"2025-04-27T03:12:58.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-27T04:20:33.683Z","etag":null,"topics":["aws-comprehend","dlp","nightfall","pii","pii-scan","pii-scanner"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ventz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-27T03:03:56.000Z","updated_at":"2025-04-27T03:13:01.000Z","dependencies_parsed_at":"2025-04-27T04:20:51.336Z","dependency_job_id":"5eb45aa0-f4b3-43a9-86f2-13c660254566","html_url":"https://github.com/ventz/pii-scan","commit_stats":null,"previous_names":["ventz/pii-scan"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ventz%2Fpii-scan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ventz%2Fpii-scan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ventz%2Fpii-scan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ventz%2Fpii-scan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ventz","download_url":"https://codeload.github.com/ventz/pii-scan/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252415007,"owners_count":21744215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-comprehend","dlp","nightfall","pii","pii-scan","pii-scanner"],"created_at":"2025-05-04T23:33:58.739Z","updated_at":"2025-05-04T23:33:59.317Z","avatar_url":"https://github.com/ventz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pii-scan - a PII Scanner using AWS Comprehend\n\nA lightweight, highly configurable tool for scanning files for personally identifiable information (PII) and other sensitive data. It uses AWS Comprehend and custom regex patterns, supports fast multi-threaded execution, and can run as a standalone tool or as a GitHub Action.\n\n## Features\n\n- **AWS Comprehend Integration**: Leverages AWS's powerful NLP capabilities to detect PII entities\n- **Custom Regex Patterns**: Add your own regex patterns to detect organization-specific sensitive data\n- **Git Integration**: Scan only files that have changed in a git repository\n- **Multiple Output Formats**: Output findings in text, JSON, or CSV format\n- **Configurable**: Adjust confidence thresholds, excluded directories, and more\n- **Multithreaded**: Process files in parallel for faster scanning\n- **GitHub Action Support**: Use as a GitHub Action to scan code changes in PRs\n\n## Installation\n\n```bash\npip install boto3\n```\n\n## Usage\n\n### Basic Usage\n\n```bash\n# Scan only git changes (default)\npii_scan\n\n# Scan all files recursively\npii_scan --all\n\n# Show help\npii_scan --help\n```\n\n### Command Line Options\n\n```\nOptions:\n    --all                   Scan all files recursively (default: only scan git changes)\n    --help, -h              Show this help message and exit\n    --config FILE           Path to configuration file\n    --min-confidence FLOAT  Minimum confidence score (0.0-1.0) for PII detection\n    --output FORMAT         Output format: text, json, or csv (default: text)\n    --custom-regex FILE     Path to file containing custom regex patterns\n    --exclude-dirs DIRS     Comma-separated list of directories to exclude\n    --exclude-exts EXTS     Comma-separated list of file extensions to exclude\n    --workers INT           Number of worker threads (default: 8)\n    --verbose, -v           Enable verbose output\n    --quiet, -q             Suppress all output except findings and errors\n    --region REGION         AWS region to use for Comprehend API\n```\n\n### Examples\n\n```bash\n# Scan all files with custom regex patterns\npii_scan --all --custom-regex custom_regex_patterns.json\n\n# Scan git changes with higher confidence threshold\npii_scan --min-confidence 0.9\n\n# Output findings in JSON format\npii_scan --output json\n\n# Exclude additional directories\npii_scan --exclude-dirs \"build,dist,node_modules\"\n\n# Use a specific AWS region\npii_scan --region us-west-2\n```\n\n## Custom Regex Patterns\n\nYou can define custom regex patterns in a JSON file and provide it using the `--custom-regex` option. The file should contain a JSON object where keys are pattern names and values are regex patterns.\n\nExample `custom_regex_patterns.json`:\n\n```json\n{\n  \"Social Security Number\": \"\\\\b(?!000|666|9\\\\d{2})([0-8]\\\\d{2}|7([0-6]\\\\d|7[012]))([-]?)(?!00)\\\\d\\\\d\\\\3(?!0000)\\\\d{4}\\\\b\",\n  \"Credit Card Number\": \"\\\\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|6(?:011|5[0-9]{2})[0-9]{12}|(?:2131|1800|35\\\\d{3})\\\\d{11})\\\\b\",\n  \"Harvard ID\": \"\\\\b\\\\d{8}\\\\b\",\n  \"API Key Pattern\": \"(?i)(api[_-]?key|apikey)\\\\s*[:=]\\\\s*['\\\"]([^'\\\"]{10,})['\\\"]\",\n  \"Database Connection String\": \"(?i)(jdbc|mongodb|postgresql|mysql|sqlserver):[^\\\\s]+\"\n}\n```\n\n## Configuration File\n\nYou can provide a configuration file using the `--config` option. The file should contain a JSON object with configuration parameters.\n\nExample `config.json`:\n\n```json\n{\n  \"max_workers\": 8,\n  \"min_confidence_score\": 0.9,\n  \"excluded_dirs\": [\".git\", \".venv\", \"node_modules\", \"__pycache__\", \"dist\", \"build\"],\n  \"excluded_extensions\": [\".min.js\", \".map\", \".svg\", \".woff\", \".ttf\", \".png\", \".jpg\"],\n  \"critical_entity_types\": [\"AWS_ACCESS_KEY\", \"AWS_SECRET_KEY\", \"PASSWORD\", \"CREDIT_CARD\"],\n  \"security_relevant_entity_types\": [\n    \"AWS_ACCESS_KEY\",\n    \"AWS_SECRET_KEY\",\n    \"PASSWORD\",\n    \"USERNAME\",\n    \"IP_ADDRESS\",\n    \"EMAIL\",\n    \"CREDIT_CARD\",\n    \"PHONE_NUMBER\"\n  ]\n}\n```\n\n## GitHub Action\n\nThis tool can be used as a GitHub Action to scan code changes in pull requests. See the `action.yml` file for details.\n\nExample workflow:\n\n```yaml\nname: Scan for PII\n\non:\n  push:\n    branches: [main]\n  pull_request:\n    branches: [main]\n\njobs:\n  pii-scan:\n    name: Scan for PII\n    runs-on: ubuntu-latest\n\n    steps:\n    - name: Checkout code\n      uses: actions/checkout@v3\n      with:\n        fetch-depth: 0\n\n    - name: Run PII Scanner\n      uses: harvard-ea/action-pii-scanning-using-aws@main\n      with:\n        custom-regex-file: pii-engine/custom_regex_patterns.json\n        min-confidence: '0.85'\n      env:\n        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n```\n\n## AWS Permissions\n\nThe tool requires AWS credentials with permissions to use the Comprehend service. You can provide these credentials using environment variables or AWS configuration files.\n\nRequired permissions:\n- `comprehend:DetectPiiEntities`\n\n## Motivation\n\nThe aim was to build a free, open-source, drop-in alternative to Nightfall, which is both overpriced and limited in functionality.\n\n## Bugs, Improvements, or Questions?\n\nPlease open an [Issue](https://github.com/ventz/pii-scan/issues) — contributions via PRs are also welcome!\n\n## License\n\nThis project is licensed under the MIT License.\n\n© Ventz Petkov\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fventz%2Fpii-scan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fventz%2Fpii-scan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fventz%2Fpii-scan/lists"}