{"id":50420809,"url":"https://github.com/nao1215/sensitive","last_synced_at":"2026-05-31T08:02:40.929Z","repository":{"id":337668872,"uuid":"1152552300","full_name":"nao1215/sensitive","owner":"nao1215","description":"Detect and optionally mask sensitive data in text — credit card numbers, emails, and more","archived":false,"fork":false,"pushed_at":"2026-02-10T14:22:33.000Z","size":1790,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-10T21:07:50.468Z","etag":null,"topics":["go","golang","sensitive","sensitive-data","sensitive-word-filter","zero-dependency"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nao1215.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"nao1215"}},"created_at":"2026-02-08T03:32:46.000Z","updated_at":"2026-02-10T14:22:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nao1215/sensitive","commit_stats":null,"previous_names":["nao1215/sensitive"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/nao1215/sensitive","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nao1215%2Fsensitive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nao1215%2Fsensitive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nao1215%2Fsensitive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nao1215%2Fsensitive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nao1215","download_url":"https://codeload.github.com/nao1215/sensitive/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nao1215%2Fsensitive/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33723550,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","sensitive","sensitive-data","sensitive-word-filter","zero-dependency"],"created_at":"2026-05-31T08:02:40.862Z","updated_at":"2026-05-31T08:02:40.920Z","avatar_url":"https://github.com/nao1215.png","language":"Go","funding_links":["https://github.com/sponsors/nao1215"],"categories":[],"sub_categories":[],"readme":"# sensitive\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/nao1215/sensitive.svg)](https://pkg.go.dev/github.com/nao1215/sensitive)\n[![Go Report Card](https://goreportcard.com/badge/github.com/nao1215/sensitive)](https://goreportcard.com/report/github.com/nao1215/sensitive)\n[![Coverage](https://github.com/nao1215/sensitive/actions/workflows/coverage.yml/badge.svg)](https://github.com/nao1215/sensitive/actions/workflows/coverage.yml)\n[![MultiPlatformUnitTest](https://github.com/nao1215/sensitive/actions/workflows/unit_test.yml/badge.svg)](https://github.com/nao1215/sensitive/actions/workflows/unit_test.yml)\n\n![logo](./doc/images/logo-small.png)\n\n\n**sensitive** is a Go library that detects sensitive data in text. It scans for credit card numbers, email addresses, Japanese phone numbers, Japanese My Number, JWTs, AWS access keys, IBANs, IP addresses, Bitcoin addresses, and Ethereum addresses, returning the position, type, and confidence level of each match. It also includes international and fintech-focused detectors such as SWIFT/BIC, US ABA routing numbers, UK sort codes, payment tokens, card CVV/expiry, and ACH trace numbers. Masking is available as an optional helper, but detection is the core focus.\n\nThe library has zero external dependencies and relies only on the Go standard library.\n\n## Requirements\n\n- Go Version: 1.24 or later\n- Operating Systems (tested on):\n  - Linux\n  - macOS\n  - Windows\n\n\n## Installation\n\n```bash\ngo get github.com/nao1215/sensitive\n```\n\n## Quick Start\n\nCreate a Scanner, choose which detectors to enable, call `ScanString`, and optionally mask findings:\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n\n    \"github.com/nao1215/sensitive\"\n    \"github.com/nao1215/sensitive/detector\"\n    \"github.com/nao1215/sensitive/mask\"\n)\n\nfunc main() {\n    scanner := sensitive.NewScanner(sensitive.WithAll())\n    text := \"user tanaka@example.com paid with 4532015112830366\"\n    findings := scanner.ScanString(text)\n\n    for _, f := range findings {\n        fmt.Printf(\"type=%s raw=%s confidence=%.2f\\n\",\n            f.DetectorName, f.RawValue, f.Confidence)\n    }\n\n    masked := mask.Mask(text, findings, map[sensitive.DetectorName]mask.Strategy{\n        detector.NamePAN:   mask.Last4,\n        detector.NameEmail: mask.Partial,\n    })\n    fmt.Println(masked)\n}\n```\n\nOutput (order may vary):\n\n```\ntype=pan raw=4532015112830366 confidence=1.00\ntype=email raw=tanaka@example.com confidence=1.00\nuser t*****@example.com paid with ************0366\n```\n\n`WithAll()` turns on every built-in detector. If you only care about specific types, pick them individually:\n\n```go\nscanner := sensitive.NewScanner(sensitive.WithPAN(), sensitive.WithEmail())\n```\n\n\u003e **Caution on `WithAll()`:** `WithAll()` enables *all* built-in detectors, including context-based weak detectors (`WithBankAccount`, `WithACHTrace`, `WithMerchantID`, `WithCVV`, `WithCardExpiry`). These detectors rely on nearby keywords rather than checksums and may produce false positives. In strict/financial-audit scenarios where false positive cost is high, avoid `WithAll()` and enable only the specific detectors you need.\n\n\u003e **Note:** `NewScanner()` with no options creates a scanner with zero detectors, so `Scan` will always return an empty result. You must pass at least one `With*()` option to enable detection.\n\n**Common mistakes:**\n\n```go\n// Mistake 1: No detectors — always returns empty results.\nscanner := sensitive.NewScanner()\nfindings := scanner.ScanString(\"4532015112830366\") // findings is empty!\n\n// Mistake 2: WithAll() in strict mode produces noise from weak detectors.\n// Use specific options instead.\nscanner = sensitive.NewScanner(sensitive.WithPAN(), sensitive.WithEmail())\n```\n\n## Supported Detectors\n\n| Option | Detects | Validation |\n|--------|---------|------------|\n| `WithPAN()` | Credit card numbers (Visa, Mastercard, Amex, JCB, Discover, Diners, UnionPay) | BIN prefix + Luhn algorithm |\n| `WithEmail()` | Email addresses | Structure + known TLD check |\n| `WithJPPhone()` | Japanese phone numbers (mobile, landline, IP phone, toll-free, M2M/IoT, service) | Prefix classification + digit count |\n| `WithMyNumber()` | Japanese My Number (12-digit individual number) | MOD 11 check digit |\n| `WithJWT()` | JSON Web Tokens | Header decode + `alg` key check |\n| `WithAWSKey()` | AWS Access Key IDs (`AKIA...` / `ASIA...`) | Prefix + 20-char alphanumeric |\n| `WithIBAN()` | International Bank Account Numbers | Country code + MOD 97 check digit |\n| `WithIPAddr()` | IPv4 and IPv6 addresses | `net.ParseIP` + octet range |\n| `WithSWIFTBIC()` | SWIFT/BIC codes | Format + country code validation |\n| `WithABARouting()` | US ABA routing numbers | Prefix range + checksum |\n| `WithUKSortCode()` | UK sort codes (XX-XX-XX) | Pattern + boundary checks |\n| `WithCVV()` | Card verification values (CVV/CVC/CID) | Context keyword + digit length (context-based, weaker) |\n| `WithCardExpiry()` | Card expiration dates | Context keyword + MM/YY validation (context-based, weaker) |\n| `WithPaymentToken()` | Payment processor tokens (Stripe/PayPal/Square) | Prefix + minimum body length |\n| `WithBankAccount()` | Bank account numbers (context-based) | Context keyword + digit range (context-based, weaker) |\n| `WithACHTrace()` | ACH trace numbers | Context keyword + prefix range (context-based, weaker) |\n| `WithMerchantID()` | Merchant/terminal IDs | Context keyword + format (context-based, weaker) |\n| `WithBTC()` | Bitcoin addresses (P2PKH, P2SH, Bech32, Bech32m/Taproot) | Base58Check (double SHA-256) / Bech32 polynomial checksum |\n| `WithETH()` | Ethereum addresses (0x + 40 hex) | EIP-55 mixed-case checksum (Keccak-256) |\n| `WithAll()` | All of the above | |\n\n## Benchmarks\n\n**Measurement conditions:**\n\n- **Command:** `go test -bench BenchmarkScanner -benchmem -benchtime 3s -count 5 -run '^$'`\n- **Go version:** 1.24 (linux/amd64)\n- **GOMAXPROCS:** 16\n- **CPU:** AMD RYZEN AI MAX+ 395 w/ Radeon 8060S\n- **Commit:** [b7e0cdc](https://github.com/nao1215/sensitive/commit/b7e0cdc)\n\nTo reproduce, run the command above. Use `-count 5` and take the median for stable results.\nBenchmark numbers are environment-sensitive. Expect variation across Go versions, CPUs, and background load, and refresh results periodically if you publish them for compliance or audit purposes.\n\n### Per-detector benchmarks (single detector enabled)\n\n| Benchmark | ns/op | B/op | allocs/op |\n|-----------|-------|------|-----------|\n| PAN | 286.7 | 944 | 16 |\n| Email | 188.2 | 288 | 9 |\n| JPPhone | 171.3 | 464 | 8 |\n| MyNumber | 142.0 | 392 | 6 |\n| JWT | 1001 | 1208 | 25 |\n| AWSKey | 147.1 | 280 | 8 |\n| IBAN | 205.7 | 226 | 6 |\n| IPAddr | 209.8 | 312 | 10 |\n| SWIFTBIC | 176.1 | 288 | 9 |\n| ABARouting | 132.7 | 376 | 6 |\n| UKSortCode | 128.4 | 248 | 8 |\n| CVV | 289.6 | 568 | 18 |\n| CardExpiry | 261.4 | 456 | 16 |\n| PaymentToken | 276.7 | 688 | 20 |\n| BankAccount | 435.1 | 760 | 22 |\n| ACHTrace | 325.9 | 480 | 17 |\n| MerchantID | 343.4 | 568 | 18 |\n| BTC | 514.5 | 328 | 7 |\n| ETH | 2118 | 329 | 7 |\n\n### Multi-detector and edge-case benchmarks\n\n| Benchmark | Description |\n|-----------|-------------|\n| `BenchmarkScannerNoMatch` | All detectors enabled, input with no sensitive data. Note: detectors with nil hints (IBAN, SWIFT/BIC, ABA, MyNumber) always run regardless of input content. |\n| `BenchmarkScannerAllDetectors` | All detectors enabled, input containing email + PAN + IP |\n| `BenchmarkScannerEmptyInput` | All detectors enabled, nil input |\n| `BenchmarkScannerLargeInput` | All detectors enabled, ~4KB log block with no sensitive data |\n| `BenchmarkScannerHintMatchNoDetection` | All detectors enabled, hints match but no valid sensitive data found |\n| `BenchmarkScannerFullWidthInput` | All detectors enabled, full-width digit input requiring normalization |\n\n## Scanning Streams\n\nFor log files and other line-oriented input, use `ScanLines` to process data incrementally without loading the entire content into memory. The callback is invoked only for lines that contain findings:\n\n```go\nf, _ := os.Open(\"access.log\")\ndefer f.Close()\n\nscanner := sensitive.NewScanner(sensitive.WithAll())\nerr := scanner.ScanLines(f, func(lineNum int, line []byte, findings []sensitive.Finding) {\n    for _, finding := range findings {\n        fmt.Printf(\"line %d: %s (%s)\\n\", lineNum, finding.DetectorName, finding.RawValue)\n    }\n})\nif err != nil {\n    log.Fatal(err)\n}\n```\n\nIf the entire content fits in memory, `ScanReader` is a simpler alternative:\n\n```go\nf, _ := os.Open(\"data.txt\")\ndefer f.Close()\n\nfindings, err := scanner.ScanReader(f)\n```\n\n## Confidence Filtering\n\nUse `WithMinConfidence` to control the strictness of detection. Findings below the threshold are filtered out:\n\n```go\n// Strict mode: only high-confidence findings (\u003e= 0.8).\nscanner := sensitive.NewScanner(sensitive.WithAll(), sensitive.WithMinConfidence(0.8))\n\n// Loose mode: include medium-confidence and above (\u003e= 0.4).\nscanner = sensitive.NewScanner(sensitive.WithAll(), sensitive.WithMinConfidence(0.4))\n```\n\nThis is useful for suppressing noise from context-based weak detectors (BankAccount, CVV, CardExpiry, etc.) while keeping strong checksum-validated results.\n\n## Classifying Findings by Kind\n\nEach finding has a `Kind()` method that returns a broad semantic category (`financial`, `pii`, or `credential`), enabling downstream classification without switching on all detector names:\n\n```go\nfor _, f := range findings {\n    switch f.Kind() {\n    case detector.KindFinancial:\n        // PAN, IBAN, ABA routing, sort code, CVV, card expiry, etc.\n    case detector.KindPII:\n        // email, phone, My Number, IP address\n    case detector.KindCredential:\n        // JWT, AWS key, payment token\n    }\n}\n```\n\n## Working with Findings\n\nEach `Finding` contains the detector name, byte offsets, confidence score (0.0--1.0), the raw matched string, and a Detail struct with detector-specific information.\n\n\u003e **Note:** `Start` and `End` are **byte offsets**, not rune (character) offsets. For multi-byte UTF-8 text (e.g., Japanese), use the byte positions directly when slicing `[]byte` data.\n\u003e\n\u003e Context-based detectors (`WithBankAccount`, `WithACHTrace`, `WithMerchantID`, `WithCVV`, `WithCardExpiry`) rely on nearby keywords rather than checksums, so they are more prone to false positives than checksum-validated detectors. Confidence scores vary by detector: `WithBankAccount` returns 0.50--0.65, `WithMerchantID` and `WithACHTrace` return 0.70--0.75, and `WithCVV` and `WithCardExpiry` return 0.85.\n\n### Checking the detector type\n\n```go\nfor _, f := range findings {\n    if f.IsPAN() {\n        // handle credit card\n    }\n    if f.IsEmail() {\n        // handle email\n    }\n}\n```\n\nThere is also a generic `Is` method that takes a detector name constant:\n\n```go\nif f.Is(detector.NamePAN) { ... }\n```\n\n### Confidence levels\n\nConfidence is a float between 0.0 and 1.0. When you do not need the exact score, use `Level()` to get a categorical assessment:\n\n```go\nswitch f.Level() {\ncase detector.ConfidenceHigh:   // \u003e= 0.8\ncase detector.ConfidenceMedium: // \u003e= 0.4\ncase detector.ConfidenceLow:    // \u003c 0.4\n}\n```\n\n### Getting detector-specific details\n\nEvery finding carries a `Detail` field. Instead of type-asserting it yourself, use the typed accessor methods. Each returns a pointer and a boolean indicating success:\n\n```go\nscanner := sensitive.NewScanner(sensitive.WithPAN())\nfindings := scanner.ScanString(\"4532015112830366\")\n\nif detail, ok := findings[0].PANDetail(); ok {\n    fmt.Println(detail.Brand)  // \"Visa\"\n    fmt.Println(detail.Last4)  // \"0366\"\n    fmt.Println(detail.Luhn)   // true\n}\n```\n\nThe available accessors and their fields:\n\n| Method | Fields |\n|--------|--------|\n| `PANDetail()` | Brand, BIN, Last4, Luhn, Length |\n| `EmailDetail()` | Local, Domain |\n| `JPPhoneDetail()` | PhoneType (`JPPhoneTypeMobile`, `JPPhoneTypeLandline`, `JPPhoneTypeIPPhone`, `JPPhoneTypeTollFree`, `JPPhoneTypeM2M`, `JPPhoneTypeService`) |\n| `JWTDetail()` | Algorithm (e.g. `HS256`, `RS256`) |\n| `AWSKeyDetail()` | KeyType (`AWSKeyTypeLongTerm` or `AWSKeyTypeTemporary`) |\n| `IBANDetail()` | CountryCode (ISO 3166-1 alpha-2) |\n| `IPAddrDetail()` | Version (4 or 6) |\n| `MyNumberDetail()` | CheckDigitValid |\n| `BTCDetail()` | AddressType (`BTCAddressP2PKH`, `BTCAddressP2SH`, `BTCAddressBech32`, `BTCAddressBech32m`) |\n| `ETHDetail()` | EIP55 (bool, whether EIP-55 checksum validated) |\n\n## Masking\n\nThe `mask` sub-package provides five masking strategies:\n\n| Strategy | Example |\n|----------|---------|\n| `Redact` | `4532015112830366` -\u003e `****************` |\n| `Last4` | `4532015112830366` -\u003e `************0366` |\n| `First1Last4` | `4532015112830366` -\u003e `4***********0366` |\n| `Partial` | `tanaka@example.com` -\u003e `t*****@example.com` |\n| `Hash` | `4532015112830366` -\u003e `a8f5f167` (SHA-256 prefix) |\n\nUse `mask.Mask` to apply different strategies per detector:\n\n```go\nimport (\n    \"github.com/nao1215/sensitive\"\n    \"github.com/nao1215/sensitive/detector\"\n    \"github.com/nao1215/sensitive/mask\"\n)\n\nscanner := sensitive.NewScanner(sensitive.WithPAN(), sensitive.WithEmail())\ntext := \"user tanaka@example.com paid with 4532015112830366\"\nfindings := scanner.ScanString(text)\n\nmasked := mask.Mask(text, findings, map[sensitive.DetectorName]mask.Strategy{\n    detector.NamePAN:   mask.Last4,\n    detector.NameEmail: mask.Partial,\n})\n\nfmt.Println(masked)\n// user t*****@example.com paid with ************0366\n```\n\nIf you want the same strategy for everything, use `mask.MaskAll`:\n\n```go\nmasked := mask.MaskAll(text, findings, mask.Redact)\n// user ****************** paid with ****************\n```\n\n## Custom Detectors\n\nYou can add your own detectors. The simplest way is `detector.NewRegex`, which wraps a compiled regular expression:\n\n```go\nimport (\n    \"regexp\"\n\n    \"github.com/nao1215/sensitive\"\n    \"github.com/nao1215/sensitive/detector\"\n)\n\nticketDetector := detector.NewRegex(\n    detector.DetectorName(\"ticket_id\"),\n    regexp.MustCompile(`TICKET-\\d{4}`),\n    [][]byte{[]byte(\"TICKET-\")},   // hint for pre-filtering\n    0.9,                            // fixed confidence\n)\n\nscanner := sensitive.NewScanner(\n    sensitive.WithPAN(),\n    sensitive.WithDetector(ticketDetector),\n)\n```\n\nThe hints parameter is important for performance. The scanner uses `bytes.Contains` to check hints before calling `Scan`, so a good hint lets the scanner skip the regex entirely for inputs that cannot match.\n\nFor more complex logic, implement the `Detector` interface directly:\n\n```go\ntype Detector interface {\n    Name() detector.DetectorName\n    Hints() [][]byte\n    Scan(data []byte) []detector.Finding\n}\n```\n\n## Full-Width Digit Support\n\nJapanese text often uses full-width digits (０-９). Detectors that parse digit sequences directly (PAN, JPPhone, MyNumber, ABA routing, BankAccount) normalize full-width digits to half-width before detection, so a phone number written as `０９０－１２３４－５６７８` or a bank account number written as `口座番号 １２３４５６７８` is correctly recognized. IBAN and UK sort code do **not** normalize full-width digits because their formats are primarily used in Western contexts where full-width encoding is uncommon. Context-based detectors (CVV, CardExpiry, ACHTrace, MerchantID) also do **not** normalize full-width digits. The utility function is also available for direct use:\n\n```go\nnormalized, posMap := detector.NormalizeFullWidthDigits([]byte(\"０９０－１２３４－５６７８\"))\nfmt.Println(string(normalized)) // 090-1234-5678\n```\n\n## How It Works\n\nThe scanner runs a multi-stage filtering pipeline to keep scan cost low.\n\n```mermaid\nsequenceDiagram\n    participant Caller\n    participant Scanner\n    participant HintFilter as Hint Filter\n    participant Detector\n    participant Dedup as Dedup \u0026 Sort\n\n    Caller-\u003e\u003eScanner: Scan(data)\n    alt input is empty\n        Scanner--\u003e\u003eCaller: nil\n    end\n\n    loop for each registered Detector\n        Scanner-\u003e\u003eHintFilter: bytes.Contains(data, hint) (~15 ns, SIMD)\n        alt no hint matched\n            HintFilter--\u003e\u003eScanner: skip\n        else hint matched\n            HintFilter--\u003e\u003eScanner: pass\n            Scanner-\u003e\u003eDetector: Scan(data)\n            Note right of Detector: domain-specific validation\u003cbr/\u003e(BIN, Luhn, MOD 97, etc.)\n            Detector--\u003e\u003eScanner: []Finding\n        end\n    end\n\n    Scanner-\u003e\u003eDedup: merge all findings\n    Note right of Dedup: dedup overlapping (keep highest confidence)\u003cbr/\u003esort by confidence desc\n    Dedup--\u003e\u003eScanner: []Finding\n    Scanner--\u003e\u003eCaller: []Finding\n```\n\n## Contributing\n\nContributions are welcome!\n\nIf you would like to send comments such as \"find a bug\" or \"request for additional features\" to the developer, please use one of the following contacts.\n\n- [GitHub Issue](https://github.com/nao1215/sensitive/issues)\n\n\n## License\n\n[MIT LICENSE](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnao1215%2Fsensitive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnao1215%2Fsensitive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnao1215%2Fsensitive/lists"}