{"id":16904360,"url":"https://github.com/naiquevin/dupenukem","last_synced_at":"2025-07-16T21:15:17.198Z","repository":{"id":217813617,"uuid":"655260682","full_name":"naiquevin/dupenukem","owner":"naiquevin","description":"A command line file deduplication tool","archived":false,"fork":false,"pushed_at":"2024-04-27T10:19:59.000Z","size":307,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T10:42:38.236Z","etag":null,"topics":["cli","deduplication","filesystem"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/naiquevin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-18T11:28:33.000Z","updated_at":"2025-02-16T09:52:23.000Z","dependencies_parsed_at":"2024-03-17T19:31:29.693Z","dependency_job_id":"24fc7f82-8157-41ec-9861-32b4d63adf60","html_url":"https://github.com/naiquevin/dupenukem","commit_stats":null,"previous_names":["naiquevin/dupenukem"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naiquevin%2Fdupenukem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naiquevin%2Fdupenukem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naiquevin%2Fdupenukem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naiquevin%2Fdupenukem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/naiquevin","download_url":"https://codeload.github.com/naiquevin/dupenukem/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248419760,"owners_count":21100246,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","deduplication","filesystem"],"created_at":"2024-10-13T18:33:00.113Z","updated_at":"2025-04-11T14:32:57.789Z","avatar_url":"https://github.com/naiquevin.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"Dupenukem\n=========\n\nDupenukem is a simple command line utility for file deduplication.\n\n:warning: Warning\n-----------------\n\nThis is a personal project for learning and experimenting with the\n[Rust programming language](https://www.rust-lang.org/). It doesn't\nclaim to be fast or efficient by any means. It doesn't support Windows\n(at present). Moreover, it's designed to perform destructive\noperations such as deleting files from your computer. Please use with\ncaution.\n\nIf you're looking for a serious file deduplication software there is\n[fclones](https://github.com/pkolaczk/fclones) which is highly\nperformant and popular. There must be other alternatives too.\n\nHaving said that, I've used `dupenukem` to clean my Dropbox folder and\na couple of external hard drives. I plan to ship features and\nimprovements based on my use case, or just use as an opportunity to\ncode in rust.\n\nIt has been tested only on MacOS, although it should theoretically\nwork on Linux too. As I don't have access to a Windows machine, there\nis no plan to support Windows at least in the near future.\n\nInstallation\n------------\n\nI am still figuring out how to use github workflows for building and\ndistributing binaries. In the meanwhile, you can install it using\n`cargo`, directly from github,\n\n``` shell\n    cargo install --git https://github.com/naiquevin/dupenukem.git\n```\n\nOr build from source, again using `cargo`.\n\n``` shell\n    git clone git@github.com:naiquevin/dupenukem.git\n    cd dupenukem\n    cargo build --release\n\n    # Copy the binary to some dir in your PATH\n    cp target/release/dupenukem ~/bin\n\n    # You can now run it\n    dupenukem --help\n```\n\nUsage\n-----\n\n`dupenukem` provides three commands for a three step deduplication\nworkflow:\n\n### Step 1: Finding duplicates and generating a snapshot\n\nThe `find` command accepts a `rootdir` and finds all duplicate files\nunder it. The output is what is called a \"snapshot\". This is nothing\nbut text representation of the state of duplicate files inside the\ndirectory captured at that moment. This output is printed to stdout\nand users must store it inside a file.\n\nThe snapshot format is explained in detail later in the example\nsection.\n\n### Step 2: Editing the snapshot and validating changes\n\nOnce the snapshot file is generated, the user is supposed to edit it\nin order to tell this tool what should be done with the duplicate\nfiles. Only 2 options are currently supported:\n\n1. duplicate files can be marked for deletion\n2. duplicate files can be marked for symlinking i.e. a duplicate file\n   will be replaced with a symlink to an original one (can be decided\n   by the user)\n\nAn updated snapshot can be validated using the `validate` command\nwhich basically checks for compatibility of the snapshot and the\nchanges w.r.t the current state of the files. This is to protect\nagainst data loss in case any changes get made to a previously\nidentified duplicate file.\n\n### Step 3: Applying the changes\n\nOnce a user-edited snapshot has been validated it can be given as\ninput to the `apply` command, which will actually execute the\nactions. The apply command also implicitly runs the `validate` step\nagain considering the time-of-check to time-of-use (TOCTOU) nature of\nthe workflow.\n\nAs it performs destructive operations, two safeguards are implemented:\n\n1. The `apply` command can be run with a `--dry-run` flag which will\n   cause all actions to be only logged and not actually executed. When\n   run without the `--dry-run` flag, the user is also asked for\n   `yes/no` confirmation to proceed.\n\n2. Before deleting a file or replacing it with symlink, a backup is\n   taken at another location (preserving the original directory\n   structure). The user may delete the backup directory after\n   verifying the actual changes performed on disk.\n\nThe apply command is also idempotent i.e. if run multiple times, the\nalready applied changes will be skipped. More accurately, the `apply`\ncommand tries to get the files into the intended state indicated by\nthe action marker. If a file is already in that state, it will no-op\nand move on. This way, the user may incrementally fix and verify one\ngroup of duplicates or even one file at a time.\n\nExample\n-------\n\nIt's easier to explain the usage in detail with the help of an\nexample. For that let's first create a dummy directory with a few\nduplicate files.\n\n``` shell\n    mkdir ~/dpnktest\n    cd ~/dpnktest\n    mkdir foo bar cat\n\n    echo ONE \u003e foo/1.txt\n    cp foo/1.txt bar/\n\n    echo TWO \u003e foo/2.txt\n    cp foo/2.txt cat\n\n    echo THREE \u003e foo/3.txt\n    echo FOUR \u003e bar/4.txt\n```\n\nThis resulting dir structure will be:\n\n``` shell\n    $ tree --charset=ascii\n    .\n    |-- bar\n    |   |-- 1.txt\n    |   `-- 4.txt\n    |-- cat\n    |   `-- 2.txt\n    `-- foo\n        |-- 1.txt\n        |-- 2.txt\n        `-- 3.txt\n\n    4 directories, 6 files\n```\n\nNow let's use `dupenukem` to find and fix duplicates inside this root\ndirectory. It's assumed that the user running `dupenukem` has the\npermissions to read and write files inside the root directory.\n\nWe'll begin by running the `find` command:\n\n``` text\n    $ dupenukem find -v ~/dpnktest | tee ~/dpnktest_snapshot.txt\n    [2024-03-26T12:58:06Z INFO  dupenukem] Generating snapshot for dir: /Users/vineet/dpnktest\n    [2024-03-26T12:58:06Z INFO  dupenukem] A max of 8 bytes space can be freed by deduplication\n    #! Root Directory: /Users/vineet/dpnktest\n    #! Generated at: Tue, 26 Mar 2024 18:28:06 +0530\n\n    [13062064944137093030]\n    keep cat/2.txt\n    keep foo/2.txt\n\n    [10098984572146910405]\n    keep foo/1.txt\n    keep bar/1.txt\n\n    # Reference:\n    # keep \u003ctarget\u003e = keep the target path as it is\n    # delete \u003ctarget\u003e = delete the target path\n    # symlink \u003ctarget\u003e [-\u003e \u003csrc\u003e] = Replace target with a symlink\n    # .       If 'src' is specified, it can either be an absolute or\n    # .       relative (to 'target'). Else one of the duplicates marked\n    # .       as 'keep' will be considered. If 'src' is not specified,\n    # .       a relative symlink will be created.\n    #\n    # This section is a comment and will be ignored by the tool\n```\n\nThings to note:\n\n- Two groups of duplicate files have been found. Each group has a\n  unique identifier - `13062064944137093030` and\n  `10098984572146910405`. These are nothing but 64-bit\n  [xxhash3](https://xxhash.com/) hashes of the contents of the files.\n\n- Under every group (indicated by the hash within square brackets),\n  duplicate files in that group are listed along with an \"action\n  marker\" which currently says `keep` for the files. Note that the\n  file paths are relative to the root directory.\n\n- The snapshot only contains duplicate files. For e.g. the files\n  `foo/3.txt` and `bar/4.txt` have no duplicates so they are not\n  included in the snapshot. Also, only the duplicate files located\n  under the root dir are considered. Eg. If `bar/4.txt` happens to be\n  a copy of `~/some/other/root/dir/4.txt`, it will still be excluded\n  from the snapshot.\n\n- At the beginning of the output, there are a couple of lines prefixed\n  with `#!`, which are for storing/defining metadata. Users must not\n  modify these lines.\n\n- Near the end of the output there is a block of text with all lines\n  prefixed with `#`. These are comments. The snapshot includes a\n  simple reference for the action markers that the user may use when\n  editing the file.\n\n- Notice the log lines before the snapshot output. Logs are printed to\n  `stderr` and the level can be controlled using the `-v`\n  option. Starting version `0.2.0` (unreleased), the find command logs\n  the max space that can be freed up by deduplication.\n\n- Finally, we've redirected the (std) output to the file\n  `~/dpnktest_snapshot.txt` in order to store the snapshot.\n\nNow let's ask `dupenukem` to fix the duplicates as follows,\n\n1. delete `cat/2.txt`\n2. replace `bar/1.txt` with a symlink that points to `foo/1.txt`\n\nTo do that we'll edit the file as follows (excluding metadata and\ncomments for brevity):\n\n``` text\n    [..snip..]\n\n    [13062064944137093030]\n    delete cat/2.txt\n    keep foo/2.txt\n\n    [10098984572146910405]\n    keep foo/1.txt\n    symlink bar/1.txt\n\n    [..snip..]\n```\n\nAfter making the above changes, we should validate the snapshot file.\n\n``` shell\n    $ dupenukem validate ~/dpnktest_snapshot.txt\n    Snapshot is valid!\n    No. of pending action(s): 2\n```\n\nBefore proceeding with the `apply` command, let's consider the case\nwhere some other process modifies the `bar/1.txt` file in the\nmeanwhile. Then the `validate` command would fail as `bar/1.txt` would\nno longer be a duplicate of `foo/1.txt`.\n\nHowever in this example, the snapshot is valid and there are 2 pending\nactions to be performed. Before actually executing these actions we\ncan run the `apply` command with `--dry-run` flag to see what exactly\nwill happen:\n\n``` shell\n    $ dupenukem apply --dry-run ~/dpnktest_snapshot.txt\n    [DRY RUN] File to be replaced with symlink: bar/1.txt -\u003e ../foo/1.txt\n    [DRY RUN] File to be deleted: cat/2.txt\n    [DRY RUN] Backup will be stored under /Users/vineet/.dupenukem/backups\n    [DRY RUN] 8 bytes of space will be freed up\n```\n\nNotice the second last line that mentions the backup location inside\n`~/.dupenukem/backups`. It's assumed that the current user has\npermissions to write to this location. Backups will be taken inside a\nnew directory under this location, with the directory name derived\nfrom the current timestamp. This will ensure that multiple backups can\ncoexist. This also implies that it's up to the user to cleanup older\nbackups that are no longer required. The user can also choose to\noverride the backup directory by specifying the `--backup-dir` option.\n\nThe last line mentions the amount of space that will be freed.\n\nLet's now proceed with running the `apply` command without the\n`--dry-run` flag.\n\n``` shell\n    $ dupenukem apply ~/dpnktest_snapshot.txt\n    \u003e All changes will be executed. Do you want to proceed? Yes\n    8 bytes of space has been freed up\n```\n\nWithout the `--dry-run` flag, it asks for confirmation before\nexecuting the actions. Let's inspect the directory structure now using\nthe same `tree` command:\n\n``` shell\n    $ cd ~/dpnktest\n    $ tree --charset=ascii\n    .\n    |-- bar\n    |   |-- 1.txt -\u003e ../foo/1.txt\n    |   `-- 4.txt\n    |-- cat\n    `-- foo\n        |-- 1.txt\n        |-- 2.txt\n        `-- 3.txt\n\n    4 directories, 5 files\n```\n\nAnd the desired changes can be seen.\n\nThe backup can be found under the default backup directory\n`~/.dupenukem/backups`.\n\n``` shell\n    $ tree --charset=ascii ~/.dupenukem/backups\n    /Users/vineet/.dupenukem/backups\n    `-- 20240116160509\n        |-- bar\n        |   `-- 1.txt\n        `-- cat\n            `-- 2.txt\n\n    4 directories, 2 files\n```\n\nNotice the dir name derived from timestamp and that the directory\nstructure is preserved. After verifying the changes, if the user\nwishes to restore any files, it can be done easily. If everything\nlooks good, they may easily delete the backup dir\n`~/.dupenukem/backups/20240116160509`.\n\nThe `apply` command is idempotent i.e. if we try running the `apply`\ncommand once again, it will no-op.\n\nNow let's see what happens if we run the `find` command once again on\nthe current state of the `~/dpnktest` directory.\n\n``` text\n    $ dupenukem find -v ~/dpnktest\n    [2024-03-26T13:10:08Z INFO  dupenukem] Generating snapshot for dir: /Users/vineet/dpnktest\n    [2024-03-26T13:10:08Z INFO  dupenukem] A max of 0 bytes space can be freed by deduplication\n    #! Root Directory: /Users/vineet/dpnktest\n    #! Generated at: Tue, 26 Mar 2024 18:40:08 +0530\n\n    [10098984572146910405]\n    keep foo/1.txt\n    symlink bar/1.txt -\u003e ../foo/1.txt\n\n    # Reference:\n    # keep \u003ctarget\u003e = keep the target path as it is\n    # delete \u003ctarget\u003e = delete the target path\n    # symlink \u003ctarget\u003e [-\u003e \u003csrc\u003e] = Replace target with a symlink\n    # .       If 'src' is specified, it can either be an absolute or\n    # .       relative (to 'target'). Else one of the duplicates marked\n    # .       as 'keep' will be considered. If 'src' is not specified,\n    # .       a relative symlink will be created.\n    #\n    # This section is a comment and will be ignored by the tool\n```\n\nThis time, it found only 1 group of 2 duplicate files among which one\nis already a symlink to the other. Technically, there is no\nduplication anymore. If you wish to not include already deduplicated\ngroup such as this one, you can run the `find` command with the\n`--skip-deduped` flag.\n\nSymlink preferences\n-------------------\n\n### Implicit v/s Explicit symlink source paths\n\nIn the above example, we saw that to replace a file with a symlink we\nadded the `symlink` marker. On running the `apply` command,\n`bar/1.txt` was replaced with a symlink pointing to `foo/1.txt`.\n\nThis means `dupenukem` will use use the other duplicate file marked as\n`keep` as the symlink source path. But what if more than two\nduplicates are found, out of which 2 of them are marked as `keep`?\nConsider the following example:\n\n``` text\n    [..snip..]\n\n    [10098984572146910405]\n    keep foo/1.txt\n    symlink bar/1.txt\n    keep cat/one.txt\n\n    [..snip..]\n```\n\nIn this case, `dupenukem` will take the first entry from\nlexicographically sorted list of all files marked with `keep`. That\nwould be `cat/one.txt` in case of this example.\n\nSuppose the user wants that the symlink source path for `bar/1.txt`\nshould be `foo/1.txt` instead, they can explicitly mention it as\nfollows,\n\n``` text\n    [..snip..]\n\n    [10098984572146910405]\n    keep foo/1.txt\n    symlink bar/1.txt -\u003e ../foo/1.txt\n    keep cat/one.txt\n\n    [..snip..]\n```\n\nNote that the explicitly mentioned source path is relative to the\nsymlink (target) and not relative to the root directory.\n\n### Relative v/s absolute symlinks\n\nFor most use cases, relative symlinks are desirable. Hence the default\nbehaviour (in case of implicit symlinks) is to use relative source\npaths. But absolute symlinks are also supported - the user just needs\nto explicitly specify the absolute source path, similar to the\nprevious example:\n\n``` text\n    [..snip..]\n\n    [10098984572146910405]\n    keep foo/1.txt\n    symlink bar/1.txt -\u003e /Users/vineet/dpnktest/foo/1.txt\n    keep cat/one.txt\n\n    [..snip..]\n```\n\nOn running apply, `bar/1.txt` will be replaced with a symlink to the\nabsolute source path.\n\n``` shell\n    $ cd ~/dpnktest\n    $ readlink bar/1.txt\n    /Users/vineet/dpnktest/foo/1.txt\n```\n\nDeleting all files in a group\n-----------------------------\n\nBy default, deletion of all files in a group is not allowed. Hence,\nthe validation and apply steps would fail in case of such input. But\noften users end up noticing such files through `dupenukem`, hence this\nfunctionality is supported behind the command line flag\n`--allow-full-deletion`. Note that this flag needs to be specified for\nboth, validate and apply steps.\n\nExclusions\n----------\n\nBasic file exclusions by exact path are supported with the `--exclude`\nflag. For example, when used to scan the Dropbox folder, it makes\nsense to exclude the drop cache directories.\n\n``` shell\n    $ dupenukem find --exclude .dropbox.cache ~/Dropbox\n```\n\nHow are duplicate files identified?\n-----------------------------------\n\n`dupenukem` recursively traverses the root directory (in breadth-first\nmanner) and then finds duplicate files in 3 steps:\n\n1. First the file sizes are compared. All files with unique sizes are\n   discarded and only the rest go through to the next step. The\n   assumption is that duplicate files will have same sizes. As the\n   sizes are obtained from file metadata, this step is extremely fast\n   and significantly reduces the IO in the next step.\n\n2. In this step, files are grouped by 64-bit `xxh3` hashes of the file\n   content. The `xxh3` hashes are also used as the group identifiers\n   in the snapshot output.\n\n3. In the last step, it confirms that all files in a group\n   (i.e. having same xxh3 hashes) have the same `sha256` hashes as\n   well. This confirmation is optional but enabled by default. To\n   disable it, the `--quick` flag can be used with the `find` command.\n\nFuture improvements\n-------------------\n\n- Improve the `exclude` functionality - support exclusions based on\n  glob/patterns as well as min/max sizes (similar to rsync)\n- Use async programming where applicable\n- Add support for hardlinks\n- Add commands backup management - restoring, clean up etc.\n- May be support Windows at some point\n\nLicense\n-------\n\nMIT (See [LICENSE](LICENSE)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaiquevin%2Fdupenukem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnaiquevin%2Fdupenukem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaiquevin%2Fdupenukem/lists"}