{"id":19509601,"url":"https://github.com/raspi/samanlainen","last_synced_at":"2025-04-15T19:32:08.485Z","repository":{"id":41488681,"uuid":"509858553","full_name":"raspi/samanlainen","owner":"raspi","description":"Delete duplicate files","archived":false,"fork":false,"pushed_at":"2023-04-01T19:30:18.000Z","size":56,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T00:11:13.143Z","etag":null,"topics":["duplicate-detection","duplicate-files","duplicates","files","rust"],"latest_commit_sha":null,"homepage":"https://github.com/raspi/samanlainen","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raspi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-02T20:47:09.000Z","updated_at":"2025-01-24T11:01:14.000Z","dependencies_parsed_at":"2025-01-09T00:46:55.784Z","dependency_job_id":null,"html_url":"https://github.com/raspi/samanlainen","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fsamanlainen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fsamanlainen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fsamanlainen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raspi%2Fsamanlainen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raspi","download_url":"https://codeload.github.com/raspi/samanlainen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249138590,"owners_count":21218913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicate-detection","duplicate-files","duplicates","files","rust"],"created_at":"2024-11-10T23:12:38.184Z","updated_at":"2025-04-15T19:32:08.459Z","avatar_url":"https://github.com/raspi.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# samanlainen\n\n![GitHub All Releases](https://img.shields.io/github/downloads/raspi/samanlainen/total?style=for-the-badge)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/raspi/samanlainen?style=for-the-badge)\n![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/raspi/samanlainen?style=for-the-badge)\n\nDelete duplicate files. Uses SHA512. Rewritten from [duplikaatti](https://github.com/raspi/duplikaatti) (Go) in Rust.\n\n## Usage\n\n```\n% samanlainen --help\nsamanlainen 0.2.1\nPekka Järvinen\nDelete duplicate files. Uses SHA512.\n\nUSAGE:\n    samanlainen [OPTIONS] \u003cPATHS\u003e...\n\nARGS:\n    \u003cPATHS\u003e...    Path(s) to scan for duplicate files\n\nOPTIONS:\n    -c, --count \u003cCOUNT\u003e              Minimum count of files considered duplicate (min. 2) [default:\n                                     2]\n    -C, --color \u003cCOLOR\u003e              Color [default: auto] [possible values: auto, off]\n        --delete-files               Delete files? If enabled, files are actually deleted\n    -h, --help                       Print help information\n    -m, --minsize \u003cMINSIZE\u003e          Minimum filesize to scan, supports EIC/SI units [default: 1B]\n    -M, --maxsize \u003cMAXSIZE\u003e          Maximum filesize to scan, supports EIC/SI units [default: 1EiB]\n    -s, --scansize \u003cSCANSIZE\u003e        Scan size used for scanning first and last bytes of file,\n                                     supports EIC/SI units [default: 1MiB]\n    -S, --sort-order \u003cSORT_ORDER\u003e    Sort order [default: i-node] [possible values: i-node,\n                                     filename, depth]\n    -v, --verbose                    Be verbose, -vvv... be very verbose\n    -V, --version                    Print version information\n```\n\n## Example run\n\n```shell\n% ls -la test\ntotal 20\ndrwxr-xr-x 2 raspi raspi 4096  2. 7. 02:36 .\ndrwxr-xr-x 7 raspi raspi 4096  2. 7. 23:57 ..\n-rw-r--r-- 1 raspi raspi 1000  2. 7. 02:36 random_copy2.dat\n-rw-r--r-- 1 raspi raspi 1000  2. 7. 02:36 random_copy.dat\n-rw-r--r-- 1 raspi raspi 1000  2. 7. 02:36 random.dat\n\n% sha1sum test/*\n105af7b371b01ee4bbb2dc7242b001dd61b07a26  test/random_copy2.dat\n105af7b371b01ee4bbb2dc7242b001dd61b07a26  test/random_copy.dat\n105af7b371b01ee4bbb2dc7242b001dd61b07a26  test/random.dat\n\n% samanlainen test\nNot deleting files (dry run), add --delete-files to actually delete files.\nFile sizes to scan: 1 B - 1152921504606846976 B (1.15 EB, 1 EiB)\nScan size for last and first bytes of files: 1048576 B (1.05 MB, 1 MiB)\nDirectories to scan:\n * /home/raspi/samanlainen/test\n\n(1 / 6) Generating file list based on file sizes...\n  File candidates: 3 Total size: 3000 B (3 kB, 2.93 kiB)\n(2 / 6) Eliminating candidates based on last 1048576 bytes of files...\n  File candidates: 3 Total size: 3000 B (3 kB, 2.93 kiB)\n(3 / 6) Eliminating candidates based on first 1048576 bytes of files...\n  File candidates: 3 Total size: 3000 B (3 kB, 2.93 kiB)\n(4 / 6) Hashing 3 files with size 1000 B (1 kB, 1000 B)  Total: 3000 B (3 kB, 2.93 kiB)...\n(5 / 6) Deleting duplicate files with checksum: 2a890a868d62a5c06a6354e023e6bf44016c963d376b2a736351348d5588f762f7199871d6a7ac09f01452846cbc45e63bd791bb30d483226203fa45f91bdca3\n   +keeping: /home/raspi/samanlainen/test/random.dat\n  -deleting: /home/raspi/samanlainen/test/random_copy.dat\n  -deleting: /home/raspi/samanlainen/test/random_copy2.dat\nCurrently removed 2 files totaling 2000 B (2 kB, 1.95 kiB)  Remaining: 0 files, 0 B\n(6 / 6) Removed 2 files totaling 2000 B (2 kB, 1.95 kiB)\n```\n\n## Algorithm\n\n1. Create file list of given directories\n    * do not add files with same identifier already added to the list (windows: file id, *nix: inode)\n    * do not add 0 byte files\n    * directories listed first has higher priority than the last\n1. Remove all files from the list which do not share same file sizes (ie. there's only one 1000 byte file -\u003e remove)\n1. Read last bytes of files and generate SHA512 sum of those bytes\n1. Remove all hashes from the list which occured only once\n1. Read first bytes of files and generate SHA512 sum of those bytes\n1. Remove all hashes from the list which occured only once\n1. Now finally hash the whole files that are left\n1. Remove all hashes from the list which occured only once\n1. Generate list of files to keep and what to remove\n    * use directory priority and file age to find what to keep\n        * oldest and highest priority files are kept\n1. Finally, remove files from filesystem(s)\n\n## Is it any good?\n\nYes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraspi%2Fsamanlainen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraspi%2Fsamanlainen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraspi%2Fsamanlainen/lists"}