{"id":17263476,"url":"https://github.com/colinh/filez","last_synced_at":"2026-02-14T03:05:02.291Z","repository":{"id":205481310,"uuid":"584502775","full_name":"ColinH/Filez","owner":"ColinH","description":"A couple of file tools for the Unix shell","archived":false,"fork":false,"pushed_at":"2024-08-11T08:08:19.000Z","size":130,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-27T21:38:52.672Z","etag":null,"topics":["backup","cli"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ColinH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-02T18:49:07.000Z","updated_at":"2024-08-11T08:08:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"489e3792-a21a-42d3-af0e-936dc6e7d6cd","html_url":"https://github.com/ColinH/Filez","commit_stats":null,"previous_names":["colinh/filez"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColinH%2FFilez","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColinH%2FFilez/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColinH%2FFilez/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColinH%2FFilez/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ColinH","download_url":"https://codeload.github.com/ColinH/Filez/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248844081,"owners_count":21170486,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","cli"],"created_at":"2024-10-15T07:56:39.999Z","updated_at":"2026-02-14T03:05:02.284Z","avatar_url":"https://github.com/ColinH.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Filez\n\nA couple of file tools for the Unix shell.\n\n * [Duplicates](#duplicates) -- Find duplicate files.\n * [Variations](#variations) -- Find (meta-)data variations.\n * [Deduplicate](#deduplicate) -- Copy tree with hard links instead of duplicate files.\n * [Incremental](#incremental) -- Incremental backup with multiple search strategies.\n * [Link First Node](#link-first-node) -- Copies only the first hard link to a new tree.\n * [Tree Struct Diff](#tree-struct-diff) -- Compares structure of two file-and-directory trees.\n\nThe [*smart hash*](#the-smart-hash) used by these tools is explained at [the end of this file](#the-smart-hash).\n\nImplemented in C++20 using the C and C++ standard libraries and Posix standard functions.\n\n## Building\n\nRequires a Unix system with GNU make and a C++20 capable Clang or GCC.\n\nEnter `make` in the main directory, a successful run generates the binaries in `build/bin`.\n\nTested with Clang 14 from Xcode 14 under macOS 13 on ARM64, and with GCC 12 under Linux on x86-64.\n\n### Linux\n\nPlease consult your distribution's manual on how to install GCC.\nIf the version included with your distribution does not have enough C++20 support you might need to search the web for other solutions.\n\n### macOS\n\nIf the `clang++` included with your Xcode does not support C++20 you will need something more recent, for example using [Homebrew](https://brew.sh) to install the latest LLVM.\n\n### Others\n\nIf you are trying to build this on any other Unix or Unix-like system, for example one of the BSDs, you probably know how to install or build yourself a suitable compiler...\n\n## The Tools\n\nNone of these tools ever delete or overwrite any existing filesystem object!\n\n### Duplicates\n\n```\nUsage: build/bin/duplicates [OPTION]... DIRECTORY [DIRECTORY]...\n  Finds duplicate files in one or more directories.\n  Files are duplicates when they have the same...\n    -n   file name.\n    -N   file name and size.\n    -i   device and inode mode 1.\n    -I   device and inode mode 2.\n    -h   smart hash and size (default).\n    -H   total hash and size.\n    -x   file name, smart hash and size.\n    -X   file name, total hash and size.\n  Additional options are...\n    -R   to change to non-recursive scanning.\n    -C   to disable normalising the given paths.\n  Special files like devices and pipes are ignored.\n  The smart hash only hashes two or three small chunks\n    when the file is large and the extension is one for\n    which a partial hash is usually sufficient.\n  The details are in hash_file.hpp and hash_size.hpp.\n```\n\n### Variations\n\n```\nUsage: build/bin/variations [OPTION]... DIRECTORY [DIRECTORY]...\n  Finds file meta data variations in one or more directories.\n    -s   Finds variations of file size for the same file name.\n    -i   Finds variations of file name for the same device and inode.\n    -n   Finds variations of smart hash for the same file name (default).\n    -N   Finds variations of total hash for the same file name.\n    -h   Finds variations of file name for the same smart hash.\n    -H   Finds variations of file name for the same total hash.\n    -x   Finds variations of device and inode for the same smart hash.\n    -X   Finds variations of device and inode for the same total hash.\n  Additional options are...\n    -R   to change to non-recursive scanning.\n    -C   to disable normalising the given paths.\n```\n\n### Deduplicate\n\n```\nUsage: build/bin/deduplicate [options]... \u003csource_dir\u003e \u003cmerged_dir\u003e\n  Creates a new directory hierarchy under merged_dir that mirrors source_dir.\n  Directories are newly created. Files are hard-linked, not copied, such that\n  when source_dir contains multiple identical copies of a file then all of the\n  hard-linked versions in merged_dir will link to the same randomly chosen\n  version of the file in source_dir.\n  Files in the source dir are considered identical when...\n    -h   the file size and smart hash match.\n    -H   the file size and total hash match.\n    -c N Copy instead of hard link all files smaller than N bytes, default 0.\n  Source and merged dir must be on the same filesystem. Merged dir must not exist.\n  Exactly one of -h and -H must be given.\n```\n\n### Incremental\n\n```\nUsage: build/bin/incremental [options]... \u003csource_dir\u003e [old_backup]... \u003cnew_backup\u003e\n  Creates a new directory hierarchy under new_backup that mirrors source_dir.\n  Hard links files from the old_backups into new_backup when possible, copies\n  the files from source_dir when the file does not exist in a previous backup.\n  Files in the old backup(s) are hard linked instead of copied when...\n    -h   the file size and smart hash match.\n    -H   the file size and total hash match.\n    -n   the file size and file name and smart hash match.\n    -N   the file size and file name and total hash match.\n    -p   the file size and file name match.\n    -P   the file size and relative path within source_dir and the old_backup dir match, including file name.\n    -x   Consider freshly copied files as candidates for hard linking.\n    -c N Copy instead of hard link all files smaller than N, default 0.\n  Options can be combined, e.g. -hP (or -h -P) will search according to both -h and -P.\n  At least one option different from -c must be given -- though everything except -x is a nop without an old_backup.\n```\n\n### Link First Node\n\n```\nUsage: link_first_node \u003csource_dir\u003e \u003csparse_dir\u003e\n  Creates a new directory hierarchy under sparse_dir that partially mirrors\n  source_dir. Files under source_dir are hard-linked correspondingly into\n  sparse_dir only if no other hard-link to the same inode was already created\n  during the current operation, and directories are created under sparse_dir\n  as required by the hard-linked files. Which of multiple paths from source_dir\n  sharing the same inode will be created under sparse_dir is unspecified.\n  Source and sparse dir must be on the same filesystem. Sparse dir must not exist.\n```\n\n### Tree Struct Diff\n\n```\nUsage: build/bin/tree_struct_diff [OPTION]... DIRECTORY DIRECTORY\n  Compares the structure of two directory trees comparing\n  the file names present or absent in each (sub-)directory.\n  Options are...\n    -C   to disable normalising the given paths.\n    -s   to also check file sizes for differences.\n    -t   to also check file types for differences.\n  File types are 'directory', 'file', etc.\n```\n\n## The Smart Hash\n\nThe \"smart hash\" used throughout these tools is a way to speed up hashing when it can be assumed that two files with the same size are either identical *or* sufficiently different to make this difference apparent in a partial hash.\n\nFor file extensions corresponding to (usually) compressed file formats like `.mov`, `.bz2` and `.jpeg` the header [`hash_size.hpp`](https://github.com/ColinH/Filez/blob/main/src/hash_size.hpp) contains a mapping from file extension to *chunk size*.\n\nThe smart hash for a file that does *not* have a defined *chunk size* is the hash of the entire file, i.e. the same as the \"total\" hash.\nThis can be considered the default case.\n\nThe smart hash for a file that *does* have a defined *chunk size* in the aforementioned header also depends on the size of the file.\n\n* If the file is smaller than 3 times the *chunk size* then the entire file is hashed.\n\n* If the file is larger than 3 times the *chunk size* then only the *chunk size* first and last bytes of the file are hashed.\n\n* If the file is larger than 1024 times the *chunk size* then an additional *chunk size* bytes near the centre of the file are also included in the smart hash.\n\nNote that due to page alignment and/or rounding sizes up to the system page size, slightly more data than indicated might be included in the smart hash.\n\n## Limitations\n\nCurrently soft links (symbolic links) are always ignored and never followed.\n\nNot everything that should be configurable via command line arguments is configurable -- yet.\n\nThe smart hash is a heuristic that might not always be appropriate.\n\nCopyright (c) 2022-2025 Dr. Colin Hirsch\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolinh%2Ffilez","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcolinh%2Ffilez","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolinh%2Ffilez/lists"}