{"id":25436681,"url":"https://github.com/robb-fr/fast-dupes-finder","last_synced_at":"2026-04-22T23:33:22.942Z","repository":{"id":276531889,"uuid":"929505831","full_name":"Robb-Fr/fast-dupes-finder","owner":"Robb-Fr","description":"This repository proposes clean, fast and shell based scripts for identifying finding duplicate files in a folder.","archived":false,"fork":false,"pushed_at":"2025-02-08T20:02:41.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-15T03:11:26.436Z","etag":null,"topics":["bash","duplicate-detection","duplicate-files","duplicate-images","duplicates","fdupes","fdupes-linux-utility"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Robb-Fr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-08T17:48:52.000Z","updated_at":"2025-02-11T12:49:54.000Z","dependencies_parsed_at":"2025-02-08T21:29:32.407Z","dependency_job_id":null,"html_url":"https://github.com/Robb-Fr/fast-dupes-finder","commit_stats":null,"previous_names":["robb-fr/fast-dupes-finder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Robb-Fr/fast-dupes-finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Robb-Fr%2Ffast-dupes-finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Robb-Fr%2Ffast-dupes-finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Robb-Fr%2Ffast-dupes-finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Robb-Fr%2Ffast-dupes-finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Robb-Fr","download_url":"https://codeload.github.com/Robb-Fr/fast-dupes-finder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Robb-Fr%2Ffast-dupes-finder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263468982,"owners_count":23471424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","duplicate-detection","duplicate-files","duplicate-images","duplicates","fdupes","fdupes-linux-utility"],"created_at":"2025-02-17T08:21:39.114Z","updated_at":"2026-04-22T23:33:22.860Z","avatar_url":"https://github.com/Robb-Fr.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fast-dupes-finder\n\nThis repository proposes clean, fast and shell based scripts for identifying\nfinding duplicate files in a folder.\n\nNo need to read this though, it's just information on the script, but feel free\nto just copy and paste it in you shell and observe the `./trash` folder filling\nwith files having duplicates in your `./` folder.\n\n## Where does this come from/what can it be used for\n\nBasically, I save my phone's photos on my NAS on which I cannot install APT\npackages or something like [fdupes](https://github.com/adrianlopezroche/fdupes).\n\nMany times I messed up my photos save and my Phone re-sent many picture to my\nNAS, but with different names, with no clear pattern: it could append\n`-1`, `-2`, ... to the name, but some photos already existed with `-1`, `-2`,\n... in their name without being duplicates.\n\nTherefore, I had to concretely compare files' contents to know if there were\nduplicate of each other. However, the folder contained over 9'000 files,\nrequiring a not so dumb algorithm to complete.\n\n## Why is this fast\n\n### Reducing the number of comparisons\n\nThe naive algorithm would be to take each file and compare it to every other:\n\n```pseudo-code\nFOR file1 IN folder:\n    FOR FILE2 IN folder:\n        IF file1 == file2:\n            OUTPUT \"file1 is a duplicate of file2\"\n        ENDIF\n    ENDFOR\nENDFOR\n```\n\nThis goes in a $O(N^2)$ basis, meaning that with my 9'000 files, there would be\naround $9'000^2$ file comparisons.\n\n#### What can be found quickly/cheaply\n\nIn order to reduce this, we need to identify when a comparison is not needed. To\nknow this \"in advance\", meaning before we have to actually compare the files,\nour only tools lie in the files' indexed information. These indexed information\nallow us to know more about the file without reading them.\n\nFor example the file size is an indexed information: using `ls -l` shows the\nfile size without requiring to read it, because our kind OS and file system\nsaves this information for us. This is instantaneous and required 0 file read,\nso cheap operation for us.\n\nAs you may see it coming, we will use files sizes to reduce the number of\ncomparisons: indeed, we know that if 2 files differ in size, they will not be\nidentical, no need to actually compare them to know that.\n\nHence, instead of comparing each file to each other in the folder, we will\nfirst find all sizes of each files (fast, `ls -l` more or less), sort all the\nsizes (fast, sorting 9'000 numbers once), go through this list of sorted sizes\nand if there are 2 successive sizes identical, keep it for later (fast, go once\nthrough a list of 9'000 numbers).\n\n#### Using the duplicate sizes to reduce the number of comparisons\n\nWith this reduced number of sizes, we cheaply identified all sizes for which\nthere are at least 2 files having this size.\n\nNow, instead of comparing each file to the others, we can go through each size,\nfind files of that size (fast), and compares these files among them.\n\n```pseudo-code\nFOR size IN duplicated_sizes:\n    FOR file1 WITH SIZE size:\n        FOR file2 WITH SIZE size NOT BEING file1:\n            IF file1 == file2:\n                OUTPUT \"file1 is a duplicate of file2\"\n            ENDIF\n        ENDFOR\n    ENDFOR\nENDFOR\n```\n\n#### How much faster can we expect to be?\n\nLet's say we actually have $M$ duplicate sizes in the folder\n($M\\leq \\frac{N}{2}$).\n\nAmong them, we need to identify how many \"honest\" file size collisions there\nare. Computing this can be boldly approximated with the\n[Birthday Paradox](https://en.wikipedia.org/wiki/Birthday_paradox#Approximations)\ncollision probability computation work. Following this, the number of \"people\"\nbecomes our number of files ($N$), while the number of \"days\" is the number of\npossibles picture sizes. This is not easy to fairly approximate and depend on\nthe structure of uploaded photos. If someone with cool knowledge in photography\nformat compression would like to help this is welcome. Assuming we can know the\nnumber of different sizes for our pictures ($d$), our probability of collision\napproximate to $p(n,d)\\approx1-e^{-\\frac{N^2}{2\\times d}}$, meaning our\nexpected number of collisions follows, by linearity, this approximation:\n$c\\approx N\\times(1-e^{-\\frac{N^2}{2\\times d}})$\n\nHopefully, this is very low, and allows us to ensure that when we compare 2\nfiles, we very likely compare 2 identical files, reducing the number of useless\ncomparisons. Empirically at least it's really fine.\n\n### Fast comparison\n\nLast but not least, we want to ensure that when we compare 2 files, we can very\nfast know if 2 files are different or not. I have seen some people online\nsuggesting hashing both files with MD5 and comparing the hashes. Although this\nprobably work, I see 2 issues:\n\n- hashing can have collisions (although unlikely but still, you don't want to\ndelete precious pictures by mistake)\n- hashing requires going through both files in whole\n\nTherefore, I prefer to use `cmp` which, conceptually, goes through both files at\nthe same time and stops as soon as it encounters 2 byte that differ. This solves\nboth the previous issues.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobb-fr%2Ffast-dupes-finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobb-fr%2Ffast-dupes-finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobb-fr%2Ffast-dupes-finder/lists"}