{"id":19361583,"url":"https://github.com/mxk/fsx","last_synced_at":"2026-07-07T07:31:46.048Z","repository":{"id":213440027,"uuid":"729249479","full_name":"mxk/fsx","owner":"mxk","description":"File system toolbox","archived":false,"fork":false,"pushed_at":"2024-02-03T23:52:21.000Z","size":335,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-22T00:51:47.289Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mxk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-08T18:05:23.000Z","updated_at":"2024-05-09T10:30:23.000Z","dependencies_parsed_at":"2024-06-21T17:29:31.950Z","dependency_job_id":"f1c4ca6f-96cf-4be7-9643-c81b78fc8b33","html_url":"https://github.com/mxk/fsx","commit_stats":null,"previous_names":["mxk/fsx"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mxk/fsx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxk%2Ffsx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxk%2Ffsx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxk%2Ffsx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxk%2Ffsx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mxk","download_url":"https://codeload.github.com/mxk/fsx/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxk%2Ffsx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35219594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T07:23:59.413Z","updated_at":"2026-07-07T07:31:46.023Z","avatar_url":"https://github.com/mxk.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# File System Toolbox\n\nWORK IN PROGRESS\n\nfsx is a command-line tool designed primarily to help with file deduplication. It also includes subcommands for managing Volume Shadow Copies on Windows.\n\n## File Deduplication Overview\n\nfsx provides commands for identifying and removing duplicate files and directories. [BLAKE3] hash function is used to compare file contents. A description of the file system state is stored in a compressed text-based index file (format described below). The general workflow is:\n\n1. Build or update an index of the file system.\n2. Identify possible duplicate directories.\n3. Categorize directories as either keep or delete.\n4. Execute any pending delete operations.\n5. Repeat.\n\n[BLAKE3]: https://en.wikipedia.org/wiki/BLAKE_%28hash_function%29\n\n## Index File Format\n\nThe index file is a zstd-compressed line-based text file designed to provide a simple overview of duplicate files within a directory tree. fsx does not care about the file name, but the convention is to use a `.fsidx` file extension.\n\nThe first two lines are the header consisting of the format version and the root directory that was scanned to generate the index. The root is treated as a raw string and may be empty if the index is of something other than the local file system.\n\nThe remaining lines consist of groups of files that share identical content (same digest and size). Files in each group begin with flags that describe the per-file state, followed by a path relative to the root. The first path in each group is followed by the file modification time. Subsequent files in the group may omit the modification time if it matches the predecessor. File paths may contain any valid UTF-8 byte sequence except LF, may not start with a tab, and must be slash-separated, relative, and [clean](https://pkg.go.dev/path#Clean).\n\nEach group ends with a singe line, identified by the double tab prefix, consisting of the 256-bit BLAKE3 digest and size shared by all files in that group. If the size is 0 (empty file), then the digest is calculated from the path.\n\n### ABNF\n\nIndex file syntax in [RFC 5234](https://datatracker.ietf.org/doc/html/rfc5234) ABNF format:\n\n```ABNF\nindex      =  header *group\n\nheader     =  version LF root-path LF\nversion    =  \"fsx index v1\"           ; File format signature and version\nroot-path  =  *( path-step / \"/\" )     ; Index root path\n\ngroup      =  file LF *( file-cont LF ) attr LF\nattr       =  2HTAB digest HTAB size\n\nfile       =  file-path path-term *HTAB mtime\nfile-cont  =  file-path [ path-term [ *HTAB mtime ] ]\nfile-path  =  [ file-flag ] HTAB rel-path\n\nfile-flag  =  \"D\" [ \"X\" ]  ; Duplicate (this copy may be removed)\nfile-flag  =/ \"J\" [ \"X\" ]  ; Junk (all copies may be removed)\nfile-flag  =/ \"K\" [ \"X\" ]  ; Keep (this copy should be preserved)\n                           ; \"X\" indicates that file no longer exists\n\nrel-path   =  path-step *( \"/\" path-step )       ; Relative UTF-8 slash-separated file path\npath-step  =  1*( %x00-09 / %x0B-2E / %x30-FF )  ; Any byte except LF and \"/\"\n\npath-term  =  HTAB \"//\"  ; Terminator added to unambiguously separate the path\n                         ; from the modification time. It is also added if the\n                         ; path ends with whitespace to prevent trimming it.\n\nmtime      =  date-time  ; RFC 3339 file modification time\ndigest     =  64HEXDIG   ; 256-bit BLAKE3 digest\nsize       =  1*DIGIT    ; File size in bytes\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmxk%2Ffsx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmxk%2Ffsx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmxk%2Ffsx/lists"}