{"id":17994990,"url":"https://github.com/totalkrill/filedupes","last_synced_at":"2025-07-01T10:35:39.693Z","repository":{"id":57629166,"uuid":"142753819","full_name":"TotalKrill/filedupes","owner":"TotalKrill","description":"Removes duplicates of files, treating one set as originals, and another set as possible dupes","archived":false,"fork":false,"pushed_at":"2019-05-02T21:54:46.000Z","size":42411,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T05:28:04.174Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TotalKrill.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-29T11:06:35.000Z","updated_at":"2019-05-02T21:54:48.000Z","dependencies_parsed_at":"2022-09-26T20:12:11.658Z","dependency_job_id":null,"html_url":"https://github.com/TotalKrill/filedupes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TotalKrill/filedupes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TotalKrill%2Ffiledupes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TotalKrill%2Ffiledupes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TotalKrill%2Ffiledupes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TotalKrill%2Ffiledupes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TotalKrill","download_url":"https://codeload.github.com/TotalKrill/filedupes/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TotalKrill%2Ffiledupes/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262944760,"owners_count":23388825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T20:17:25.786Z","updated_at":"2025-07-01T10:35:39.658Z","avatar_url":"https://github.com/TotalKrill.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# filedupes\n\n[![Latest version](https://img.shields.io/crates/v/filedupes.svg)](https://crates.io/crates/filedupes)\n![License](https://img.shields.io/crates/l/filedupes.svg)\n\nLibrary that can be used to lazily (not load entire file directly) check for duplicates on two vectors of filecontent\n\nUses file hashing code from [dupe-krill](https://github.com/kornelski/dupe-krill)\n\n## Example\n```\nuse filedupes::*;\n\nfn main() {\n    // get the files in the closest subfolder\n    let mut origs = get_files(\u0026\"./\",2);\n\n    // just do a quick removal of all original duplicates, sort will sort on size\n    origs.sort();\n    origs.dedup();\n\n    // compare to all files in all the subfolders\n    let mut checks = get_files(\u0026\"./\", std::usize::MAX);\n    println!(\"{} Originals checked against {} files\", origs.len(), checks.len());\n    let dupesets = check_duplicates(\u0026origs, \u0026mut checks);\n\n    println!( \"Amount of duplicates of first {}\", dupesets.first().unwrap().dupes.len() );\n}\n```\n\n## Binary\n\nthe crate contains a binary called ripdupe which is a small software tool that can check for duplicates\nand either remove them or just show and summarize them.\n\nExample searching for duplicates of cats.jpg\n\n    cargo run --release -- -o testdata/cats.jpg -c testdata/\n\nA small software tool that can compare one set of file, to find duplicates in another set. It has\nsome differences to similar tools due to the fact it can limit the traversal depth in folders, thus\nallow to only change or include files that not in subfolders.\n\nExample only taking the toplevel files in a folder, and looking for duplicates in the closest subfolders\n\n    cargo run --release -- -o testdata/ --odepth 1 -c testdata/ --cdepth 2\n\n## Warning\n\nI take no responsibility of files lost, since this program is specifically created to delete\nduplicate files\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftotalkrill%2Ffiledupes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftotalkrill%2Ffiledupes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftotalkrill%2Ffiledupes/lists"}