{"id":51497023,"url":"https://github.com/avifenesh/scrump","last_synced_at":"2026-07-07T16:30:31.431Z","repository":{"id":358906293,"uuid":"1243646484","full_name":"avifenesh/scrump","owner":"avifenesh","description":"Fast, format-aware secret scrubber for binary capture artifacts (perf.data, nsys-rep, ELF core, hprof, JFR, pcap, SQLite, tar/zip).","archived":false,"fork":false,"pushed_at":"2026-06-29T18:42:31.000Z","size":357,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-29T20:20:48.323Z","etag":null,"topics":["core-dump","data-loss-prevention","hprof","jfr","nsys","observability","pcap","perf","pii","profiling","redaction","rust","secret-scanning","secrets","security","sqlite"],"latest_commit_sha":null,"homepage":"https://github.com/avifenesh/scrump","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/avifenesh.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-19T14:30:00.000Z","updated_at":"2026-06-29T18:42:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/avifenesh/scrump","commit_stats":null,"previous_names":["avifenesh/scrump"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/avifenesh/scrump","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avifenesh%2Fscrump","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avifenesh%2Fscrump/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avifenesh%2Fscrump/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avifenesh%2Fscrump/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/avifenesh","download_url":"https://codeload.github.com/avifenesh/scrump/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avifenesh%2Fscrump/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35235814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["core-dump","data-loss-prevention","hprof","jfr","nsys","observability","pcap","perf","pii","profiling","redaction","rust","secret-scanning","secrets","security","sqlite"],"created_at":"2026-07-07T16:30:30.518Z","updated_at":"2026-07-07T16:30:31.406Z","avatar_url":"https://github.com/avifenesh.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scrump\n\n\u003e Fast, format-aware secret scrubber for binary capture artifacts.\n\nBorn from a real incident: a `perf` profile tarball uploaded to a public\nGitHub issue leaked a `GH_TOKEN`, because the environment block of the\ncaptured process — including every API key in scope at runtime — sat\ninside the binary blob, and GitHub's native secret scanning couldn't see\nthrough the format.\n\n`scrump` opens captures using the same on-disk specs the originating tools\nuse, finds dangerous content with a 1,100+ rule pattern engine, zero-fills\nit in place, and returns the file in its original shape. The redacted\n`perf.data` still loads in `perf report`; the redacted `nsys-rep` still\nopens in Nsight Systems; the redacted SQLite still passes `sqlite3\n\".schema\"`; the redacted pcap still opens in Wireshark.\n\n[![CI](https://github.com/avifenesh/scrump/actions/workflows/ci.yml/badge.svg)](https://github.com/avifenesh/scrump/actions/workflows/ci.yml)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n[![Rust: 1.75+](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org)\n[![GitHub release](https://img.shields.io/github/v/release/avifenesh/scrump?include_prereleases\u0026sort=semver)](https://github.com/avifenesh/scrump/releases)\n\n## Formats\n\n| Format | Crate | What it understands |\n|---|---|---|\n| `passthrough` | `scrump-format-passthrough` | Any file — single raw chunk fallback |\n| `perf` | `scrump-format-perf` | `PERFILE2`. Header + feature sections (`HEADER_CMDLINE`, etc.) + data section. |\n| `tar` | `scrump-format-tar` | `tar` / `tar.gz` / `tar.zst` / `zip`. Each member is recursively format-dispatched. |\n| `sqlite` | `scrump-format-sqlite` | `SQLite format 3`. Walks every user table, redacts TEXT/BLOB cells via `UPDATE` + `VACUUM`. |\n| `nsys` | `scrump-format-nsys` | NVIDIA `.nsys-rep` / `.ncu-rep`. tar envelope + format-aware SQLite handling for the inner DB. |\n| `elf-core` | `scrump-format-core` | 64-bit LE ELF `ET_CORE`. Walks `PT_NOTE` for `NT_PRPSINFO` cmdline + `PT_LOAD` env pages. |\n| `hprof` | `scrump-format-hprof` | Java HPROF. `JAVA PROFILE` header + record stream; tight UTF8 STRING chunks. |\n| `jfr` | `scrump-format-jfr` | Java Flight Recorder. Walks chunks via `FLR\\0` magic + `chunk_size`; refuses to touch chunk headers. |\n| `pcap` | `scrump-format-pcap` | tcpdump pcap + pcapng. Per-packet payload chunks (Authorization headers, query strings); framing untouched. |\n\n## Install\n\n```sh\n# From source (Rust 1.75+)\ncargo install --path crates/scrump-cli\n\n# Or grab a pre-built binary from the latest release (see the Releases tab\n# for the current version; tarballs are signed and shasum'd):\ngh release download --repo avifenesh/scrump --pattern '*-x86_64-unknown-linux-gnu.tar.gz'\n```\n\nSupported targets out of the box:\n\n| Target | Tier |\n|---|---|\n| `x86_64-unknown-linux-gnu` | tier-1 (cross-compiled in CI release) |\n| `aarch64-unknown-linux-gnu` | tier-1 (cross-compiled in CI release) |\n| `aarch64-apple-darwin`     | tier-1 (cross-compiled in CI release) |\n\n## Use\n\n```sh\nscrump scan some-file              # dry-run: report findings, never mutate\nscrump scrub some-file             # redact in place (atomic tmp+rename)\nscrump scrub some-file -o clean    # write clean copy elsewhere\nscrump scrub some-file --backup    # also keep the original at *.orig\nscrump scrub some-file --format perf    # force a specific format handler\nscrump scrub some-file --rules-path my.yaml    # add custom rules\n```\n\n## How detection works\n\nTwo-layer ruleset:\n\n1. **Curated default rules** (`crates/scrump-rules/rules/default.yaml`) —\n   tightly-scoped patterns for the ML/inference ecosystem: GitHub PATs,\n   HuggingFace, OpenAI, Anthropic, AWS, Slack, NVIDIA NGC, W\u0026B, Stripe.\n2. **Auto-extracted TruffleHog mirror** (`rules/trufflehog.yaml`,\n   regenerated by `cargo run -p scrump-trufflehog-compat --bin th-extract`) —\n   1,100+ rules covering every detector under `pkg/detectors/`.\n3. **Hand-coded** detectors for things regex can't express alone —\n   currently `JwtHsAware` which base64-decodes the JWT header and\n   rejects HMAC-signed tokens, mirroring TruffleHog's filtering.\n\nThe engine supports `capture_index` for keyword-proximity patterns (e.g.\nW\u0026B's bare 40-hex token near a `wandb` keyword) and `post_filter` for\nsemantic constraints beyond regex.\n\n## Format support — how it stays structure-preserving\n\nEach format crate implements:\n\n```rust\npub trait Format: Send {\n    fn name(\u0026self) -\u003e \u0026'static str;\n    fn chunks\u003c'a\u003e(\u0026'a self) -\u003e Box\u003cdyn Iterator\u003cItem = Chunk\u003c'a\u003e\u003e + 'a\u003e;\n    fn apply(\u0026mut self, hits: \u0026[Hit]) -\u003e Result\u003c()\u003e;\n    fn to_bytes(\u0026self) -\u003e Result\u003cVec\u003cu8\u003e\u003e;\n}\n```\n\nThe format decides which byte ranges are *scannable* (cmdline strings,\nTEXT cells, packet payloads, chunk bodies) and which are *structural*\n(magic words, length prefixes, varints, checksums). `apply` refuses to\nredact structural bytes. The result: scrump can never produce a file\nits own format parser couldn't parse.\n\n## Compatibility evidence\n\nscrump is validated against two third-party test corpora. The harnesses\nlive under `crates/scrump-{trufflehog,presidio}-compat/`.\n\n### TruffleHog detectors\n\n`cargo run -p scrump-trufflehog-compat --bin trufflehog-compat` walks\nevery `*_test.go` under TruffleHog's `pkg/detectors/`, parses each\nparametrized test, and runs scrump against the test input.\n\n**Last full run: 2,069 of 2,192 cases pass across 743 providers\n(94.4%).** The remaining 123 are negative-case false-positives where\nprovider A's no-hit-expected input still trips provider B's\nauto-extracted `PrefixRegex` (e.g. a `sugester` test input fires the\n`tableau` rule). They are over-detection in a scrubbing context —\nnothing TruffleHog catches is missed by scrump. CI gates on\n`SCRUMP_TH_MAX_FAILURES=123`; lowering this number must accompany rule\nfixes, and any increase fails the build. After the #9 rule curation,\n260+ structurally-broken patterns are dropped at load time (see\n`TH_QUARANTINE` in `scrump-rules`) — this is what drove the harness\nfrom 201 failures down to 123 and the per-MB hit rate on real SQLite\nlog artifacts from ~85,000 to under 0.5.\n\n### Microsoft Presidio (PII) cross-format\n\n`cargo run -p scrump-presidio-compat --bin presidio-compat` takes every\nPresidio recognizer test (52 recognizers, 671 cases), then for every\ncase embeds the test text into every binary format scrump supports\n(7 + passthrough = 8) and runs the detector against the embedded blob.\n\n**Last full run: 617 of 671 (92.0%) pass — identical pass count across\nall 8 formats**, proving the format wrapper is transparent to detection.\nThe remaining 54 failures are entirely Presidio patterns that use\nlookbehind / backreferences that Rust's `regex` crate doesn't support\n(IP recognizer, MAC, Canadian SIN with backref-bound separator).\n\n```\nFORMAT             PASS     FAIL     SKIP    PASS%\n--------------------------------------------------------\npassthrough         617       54        0    92.0%\ntar                 617       54        0    92.0%\nperf                617       54        0    92.0%\nsqlite              617       54        0    92.0%\nelf-core            617       54        0    92.0%\nhprof               617       54        0    92.0%\njfr                 617       54        0    92.0%\npcap                617       54        0    92.0%\n```\n\n## Local dev loop\n\nIf you have [`just`](https://github.com/casey/just) installed:\n\n```sh\njust check                  # fmt + clippy + tests + docs\njust e2e                    # all 8 phase-gate scripts\njust compat-trufflehog      # upstream TruffleHog parity (clones vendor/trufflehog on first run)\njust compat-presidio        # 52 recognizers × 8 formats\njust deny                   # cargo-deny supply-chain audit\njust ci                     # everything CI runs, in order\n```\n\nWithout `just`, see the recipes in the [`Justfile`](Justfile) for the\nunderlying `cargo` invocations.\n\n## Phase gates\n\nEnd-to-end gates live under `tests/`:\n\n- `tests/e2e.sh` — phase 0 (passthrough on a planted-token text file)\n- `tests/e2e_phase1.sh` — perf.data\n- `tests/e2e_phase2.sh` — tar / tar.gz / tar.zst / zip\n- `tests/e2e_phase3.sh` — sqlite + nsys-rep\n- `tests/e2e_phase4.sh` — ELF core\n- `tests/e2e_phase5.sh` — Java HPROF\n- `tests/e2e_phase6.sh` — JFR\n- `tests/e2e_phase7.sh` — pcap\n- `tests/e2e_all.sh` — master gate; runs all 8\n\nEach gate plants known token shapes, runs `scrump scrub`, then asserts\nthe file size is preserved, the format's magic / structural fields are\nuntouched, the format's native tooling still parses it, and no token\nprefix remains in the raw bytes.\n\n## Project layout\n\n```\nscrump/\n├── crates/\n│   ├── scrump-core/                # Format trait, Hit, Dispatcher\n│   ├── scrump-detect/              # regex + entropy engine\n│   ├── scrump-rules/               # curated + auto-extracted rule sets\n│   ├── scrump-cli/                 # the `scrump` binary\n│   ├── scrump-format-passthrough/  # text-and-anything fallback\n│   ├── scrump-format-perf/         # PERFILE2\n│   ├── scrump-format-tar/          # tar / zip / gz / zst (recursive)\n│   ├── scrump-format-sqlite/       # SQLite3\n│   ├── scrump-format-nsys/         # NVIDIA nsys-rep / ncu-rep\n│   ├── scrump-format-core/         # ELF core dumps\n│   ├── scrump-format-hprof/        # Java HPROF\n│   ├── scrump-format-jfr/          # Java Flight Recorder\n│   ├── scrump-format-pcap/         # pcap / pcapng\n│   ├── scrump-test-fixtures/       # spec-compliant generators\n│   ├── scrump-trufflehog-compat/   # TruffleHog parity harness\n│   └── scrump-presidio-compat/     # 8-format × 52-recognizer harness\n├── tests/                          # phase 0..7 e2e gates\n└── docs/                           # architecture, threat model\n```\n\nSee [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the internal design,\nand [`CONTRIBUTING.md`](CONTRIBUTING.md) for the format/detector\nadd-a-new-X checklists.\n\n## Security\n\nscrump is a security tool — please report vulnerabilities privately via\nthe process in [`SECURITY.md`](SECURITY.md).\n\n## License\n\n[Apache-2.0](LICENSE). Inspired by — but does not wrap — TruffleHog and\nnoseyparker (both Apache-2.0).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favifenesh%2Fscrump","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Favifenesh%2Fscrump","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favifenesh%2Fscrump/lists"}