{"id":17059564,"url":"https://github.com/trapexit/scorch","last_synced_at":"2026-03-02T10:02:29.437Z","repository":{"id":40633338,"uuid":"70624628","full_name":"trapexit/scorch","owner":"trapexit","description":"Silent CORruption CHecker and filesystem audit tool","archived":false,"fork":false,"pushed_at":"2023-02-04T23:59:24.000Z","size":72,"stargazers_count":203,"open_issues_count":7,"forks_count":11,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-02T06:11:14.626Z","etag":null,"topics":["bitrot","corruption","data-integrity","filesystem","md5","md5sum","sha1sum"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trapexit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-11T18:43:23.000Z","updated_at":"2025-03-25T11:07:04.000Z","dependencies_parsed_at":"2024-10-27T12:50:45.811Z","dependency_job_id":"9fb425a2-434b-4fb3-8ed4-d471e90ba5be","html_url":"https://github.com/trapexit/scorch","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/trapexit/scorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fscorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fscorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fscorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fscorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trapexit","download_url":"https://codeload.github.com/trapexit/scorch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fscorch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29998079,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-02T09:59:02.300Z","status":"ssl_error","status_checked_at":"2026-03-02T09:59:02.001Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitrot","corruption","data-integrity","filesystem","md5","md5sum","sha1sum"],"created_at":"2024-10-14T10:34:46.853Z","updated_at":"2026-03-02T10:02:29.342Z","avatar_url":"https://github.com/trapexit.png","language":"Python","funding_links":["https://www.patreon.com/trapexit"],"categories":[],"sub_categories":[],"readme":"# scorch (Silent CORruption CHecker)\n\nscorch is a tool to catalog files and their hashes to help in discovering file corruption, missing files, duplicate files, etc.\n\n### Usage\n\n```\nusage: scorch [\u003coptions\u003e] \u003cinstruction\u003e [\u003cdirectory\u003e]\n\nscorch (Silent CORruption CHecker) is a tool to catalog files, hash\ndigests, and other metadata to help in discovering file corruption,\nmissing files, duplicates, etc.\n\npositional arguments:\n  instruction:           * add: compute \u0026 store digests for found files\n                         * append: compute \u0026 store digests for unhashed files\n                         * backup: backs up selected database\n                         * restore: restore backed up database\n                         * list-backups: list database backups\n                         * diff-backup: show diff between current \u0026 backup DB\n                         * hashes: print available hash functions\n                         * check: check stored info against files\n                         * update: update metadata of changed files\n                         * check+update: check and update if new\n                         * cleanup: remove info of missing files\n                         * delete: remove info for found files\n                         * list: md5sum'ish compatible listing\n                         * list-unhashed: list files not yet hashed\n                         * list-missing: list files no longer on filesystem\n                         * list-dups: list files w/ dup digests\n                         * list-solo: list files w/ no dup digests\n                         * list-failed: list files marked failed\n                         * list-changed: list files marked changed\n                         * in-db: show if files exist in DB\n                         * found-in-db: print files found in DB\n                         * notfound-in-db: print files not found in DB\n  directory:             Directory or file to scan.\n\noptional arguments:\n  -d, --db=:             File to store digests and other metadata in. See\n                         docs for info. (default: /var/tmp/scorch/scorch.db)\n  -v, --verbose:         Make `instruction` more verbose. Actual behavior\n                         depends on the instruction. Can be used multiple\n                         times.\n  -q, --quote:           Shell quote/escape filenames when printed.\n  -r, --restrict=:       * sticky: restrict scan to files with sticky bit\n                         * readonly: restrict scan to readonly files\n  -f, --fnfilter=:       Restrict actions to files which match regex.\n  -F, --negate-fnfilter  Negate the fnfilter regex match.\n  -s, --sort=:           Sorting routine on input \u0026 output. (default: natural)\n                         * random: shuffled / random\n                         * natural: human-friendly sort, ascending\n                         * natural-desc: human-friendly sort, descending\n                         * radix: RADIX sort, ascending\n                         * radix-desc: RADIX sort, descending\n                         * mtime: sort by file mtime, ascending\n                         * mtime-desc: sort by file mtime, descending\n                         * checked: sort by last time checked, ascending\n                         * checked-desc: sort by last time checked, descending\n  -m, --maxactions=:     Max actions before exiting. (default: maxint)\n  -M, --maxdata=:        Max bytes to process before exiting. (default: maxint)\n                         Can use 'K', 'M', 'G', 'T' suffix.\n  -T, --maxtime=:        Max time to process before exiting. (default: maxint)\n                         Can use 's', 'm', 'h', 'd' suffix.\n  -b, --break-on-error:  Any error or digest mismatch will cause an exit.\n  -D, --diff-fields=:    Fields to use to indicate a file has 'changed' (vs.\n                         bitrot / modified) and should be rehashed.\n                         Combine with ','. (default: size)\n                         * size\n                         * inode\n                         * mtime\n                         * mode\n  -H, --hash=:           Hash algo. Use 'scorch hashes' get available algos.\n                         (default: md5)\n  -h, --help:            Print this message.\n\nexit codes:\n  *  0 : success, behavior executed, something found\n  *  1 : processing error\n  *  2 : error with command line arguments\n  *  4 : hash mismatch\n  *  8 : found\n  * 16 : not found, nothing processed\n  * 32 : interrupted\n```\n\n### Database\n\n#### Format\n\nThe file is simply CSV compressed with gzip.\n\n```\n$ # file, hash:digest, size, mode, mtime, inode, state, checked\n$ zcat /var/tmp/scorch/scorch.db\n/tmp/files/a,md5:d41d8cd98f00b204e9800998ecf8427e,0,33188,1546377833.3844686,123456,0,1588895022.6193066\n```\n\nThe 'state' value can be 'U' for unknown, 'C' for changed, 'F' for failed, or 'O' for OK.\n\nThe 'mtime' and 'checked' values are floating point seconds since epoch.\n\n\n#### --db argument\n\nThe `--db` argument can take more than a path.\n\n* /tmp/test/myfiles.db : Full path. Used as is.\n* /tmp/test : If /tmp/test is a directory -\u003e /tmp/test/scorch.db\n* /tmp/test/ : Force interpretation as directory -\u003e /tmp/test/scorch.db\n* /tmp/test : /tmp/test is not a directory -\u003e /tmp/test.db\n* ./test : Prepend current working directory and same as above. Any relative path with a '/'.\n* test : No forward slashes -\u003e /var/tmp/scorch/test.db\n\nIf there is no extension then `.db` will be added.\n\n\n#### Backup / Restore\n\nTo simplify backing up the scorch database there is a backup command. Without a directory defined it will store the database to the same location as the database. If directories are added to the arguments then the database backup will be stored there.\n\n```\n$ scorch -v backup\n/var/tmp/scorch/scorch.db.backup_2019-07-29T02:35:46Z\n$ scorch -v backup /tmp\n/tmp/scorch.db.backup_2019-07-29T02:36:12Z\n$ scorch list-backups\n/var/tmp/scorch/scorch.db.backup_2019-07-29T02:35:46Z\n$ scorch list-backups /tmp\n/tmp/scorch.db.backup_2019-07-29T02:36:12Z\n/tmp/scorch.db.backup_2019-07-29T02:13:34Z\n$ scorch restore /tmp/scorch.db.backup_2019-07-29T02:36:12Z\n```\n\n\n### Example\n\n```\n$ ls -lh /tmp/files\ntotal 0\n-rw-rw-r-- 1 nobody nogroup 0 May  3 16:30 a\n-rw-rw-r-- 1 nobody nogroup 0 May  3 16:30 b\n-rw-rw-r-- 1 nobody nogroup 0 May  3 16:30 c\n\n$ scorch -v -d /tmp/hash.db add /tmp/files\n1/3 /tmp/files/c: d41d8cd98f00b204e9800998ecf8427e\n2/3 /tmp/files/a: d41d8cd98f00b204e9800998ecf8427e\n3/3 /tmp/files/b: d41d8cd98f00b204e9800998ecf8427e\n\n$ scorch -v -d /tmp/hash.db check /tmp/files\n1/3 /tmp/files/a: OK\n2/3 /tmp/files/b: OK\n3/3 /tmp/files/c: OK\n\n$ echo asdf \u003e /tmp/files/d\n\n$ scorch -v -d /tmp/hash.db list-unhashed /tmp/files\n/tmp/files/d\n\n$ scorch -v -d /tmp/hash.db append /tmp/files\n1/1 /tmp/files/d: md5:2b00042f7481c7b056c4b410d28f33cf\n\n$ scorch -d /tmp/hash.db list-dups /tmp/files\nmd5:d41d8cd98f00b204e9800998ecf8427e /tmp/files/a /tmp/files/b /tmp/files/c\n\n$ scorch -v -d /tmp/hash.db list-dups /tmp/files\nmd5:d41d8cd98f00b204e9800998ecf8427e\n - /tmp/files/a\n - /tmp/files/b\n - /tmp/files/c\n\n$ echo foo \u003e /tmp/files/a\n$ scorch -v -d /tmp/hash.db check+update /tmp/files\n1/4 /tmp/files/b: OK\n2/4 /tmp/files/c: OK\n3/3 /tmp/files/c: FILE CHANGED\n - size: 0B -\u003e 4B\n - mtime: Tue Jan  1 16:23:57 2019 -\u003e Tue Jan  1 16:24:09 2019\n - hash: d41d8cd98f00b204e9800998ecf8427e -\u003e d3b07384d113edec49eaa6238ad5ff00\n4/4 /tmp/files/d: OK\n\n$ scorch -v -d /tmp/hash.db list /tmp/files | cut -d: -f2- | md5sum -c\n/tmp/files/c: OK\n/tmp/files/d: OK\n/tmp/files/a: OK\n/tmp/files/b: OK\n```\n\n\n### Automation\n\nA typical setup would probably be initialized manually by using **add** or **append**. After it's finished creating the database a cron job can be created to check, update, append, and cleanup the database. By not placing **scorch** into verbose mode only differences or failures will be printed and the output from the job running will be emailed to the user (if setup to do so).\n\n```\n#!/bin/sh\n\nscorch -M 128G -T 2h check+update /tmp/files\nscorch append /tmp/files\nscorch cleanup /tmp/files\n```\n\n\n# Support\n\n#### Contact / Issue submission\n\n* github.com: https://github.com/trapexit/scorch/issues\n* email: trapexit@spawn.link\n* twitter: https://twitter.com/_trapexit\n* reddit: https://www.reddit.com/user/trapexit\n* discord: https://discord.gg/MpAr69V\n\n\n#### Support development\n\nThis software is free to use and released under a very liberal license. That said if you like this software and would like to support its development donations are welcome.\n\n* PayPal: trapexit@spawn.link\n* Patreon: https://www.patreon.com/trapexit\n* Bitcoin (BTC): 1DfoUd2m5WCxJAMvcFuvDpT4DR2gWX2PWb\n* Bitcoin Cash (BCH): qrf257j0l09yxty4kur8dk2uma8p5vntdcpks72l8z\n* Ethereum (ETH): 0xb486C0270fF75872Fc51d85879b9c15C380E66CA\n* Litecoin (LTC): LW1rvHRPWtm2NUEMhJpP4DjHZY1FaJ1WYs\n* Basic Attention Token (BAT): 0xE651d4900B4C305284Da43E2e182e9abE149A87A\n* Zcash (ZEC): t1ZwTgmbQF23DJrzqbAmw8kXWvU2xUkkhTt\n* Zcoin (XZC): a8L5Vz35KdCQe7Y7urK2pcCGau7JsqZ5Gw\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrapexit%2Fscorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrapexit%2Fscorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrapexit%2Fscorch/lists"}