{"id":19509567,"url":"https://github.com/raspi/duplikaatti","last_synced_at":"2025-10-19T09:32:59.985Z","repository":{"id":57632537,"uuid":"140314047","full_name":"raspi/duplikaatti","owner":"raspi","description":"Remove duplicate files.","archived":false,"fork":false,"pushed_at":"2021-10-26T00:30:03.000Z","size":34,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T07:23:37.951Z","etag":null,"topics":["duplicate-files","duplicates","files","go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raspi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-09T16:33:11.000Z","updated_at":"2025-02-07T23:18:06.000Z","dependencies_parsed_at":"2022-08-31T16:31:39.836Z","dependency_job_id":null,"html_url":"https://github.com/raspi/duplikaatti","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fduplikaatti","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fduplikaatti/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fduplikaatti/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fduplikaatti/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raspi","download_url":"https://codeload.github.com/raspi/duplikaatti/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250926824,"owners_count":21509044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicate-files","duplicates","files","go","golang"],"created_at":"2024-11-10T23:12:29.370Z","updated_at":"2025-10-19T09:32:54.951Z","avatar_url":"https://github.com/raspi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# duplikaatti\n\n![GitHub All Releases](https://img.shields.io/github/downloads/raspi/duplikaatti/total?style=for-the-badge)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/raspi/duplikaatti?style=for-the-badge)\n![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/raspi/duplikaatti?style=for-the-badge)\n\n\nRemove duplicate files and do it fast. `duplikaatti` is designed to go through 50 TiB+ of data and hundreds of thousands of files and find duplicate files in few minutes.\n\n## Algorithm\n* Create file list of given directories \n  * do not add files with same identifier already added to the list (windows: file id, *nix: inode)\n  * do not add 0 byte files\n  * directories listed first has higher priority than the last\n* Remove all files from the list which do not share same file sizes (ie. there's only one 1000 byte file -\u003e remove)\n* Read first bytes of files and generate SHA256 sum of those bytes\n* Remove all hashes from the list which occured only once\n* Read last bytes of files and generate SHA256 sum of those bytes\n* Remove all hashes from the list which occured only once\n* Now finally hash the whole files that are left\n* Remove all hashes from the list which occured only once\n* Generate list of files to keep and what to remove\n  * use directory priority and file age to find what to keep \n    * oldest and highest priority files are kept\n* Finally, remove files from filesystem(s)\n\n## Usage\n```\nDuplicate file remover (version 1.0.0)\nRemoves duplicate files. Algorithm idea from rdfind.\n\nUsage of duplikaatti [options] \u003cdirectories\u003e:\n\nParameters:\n  -remove\n    \tActually remove files.\n\nExamples:\n  Test what would be removed:\n    duplikaatti /home/raspi/storage /mnt/storage\n  Remove files:\n    duplikaatti -remove /home/raspi/storage /mnt/storage\n```\n\nIdea inspired by https://github.com/pauldreik/rdfind\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraspi%2Fduplikaatti","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraspi%2Fduplikaatti","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraspi%2Fduplikaatti/lists"}