{"id":15443507,"url":"https://github.com/lemonyte/dff","last_synced_at":"2025-07-17T02:32:20.208Z","repository":{"id":52914360,"uuid":"345511514","full_name":"lemonyte/dff","owner":"lemonyte","description":"A dead-simple command line tool to find duplicate files on your computer.","archived":false,"fork":false,"pushed_at":"2023-10-16T08:48:00.000Z","size":45,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-19T21:17:36.108Z","etag":null,"topics":["cli","console","csharp","duplicate","file","files","hash","pyinstaller","python","rust","windows"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lemonyte.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-08T02:49:21.000Z","updated_at":"2025-03-19T13:34:32.000Z","dependencies_parsed_at":"2025-04-19T20:20:22.855Z","dependency_job_id":"0c6d9d05-c259-457c-a78b-2241408ce5ca","html_url":"https://github.com/lemonyte/dff","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/lemonyte/dff","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemonyte%2Fdff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemonyte%2Fdff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemonyte%2Fdff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemonyte%2Fdff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lemonyte","download_url":"https://codeload.github.com/lemonyte/dff/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemonyte%2Fdff/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265560768,"owners_count":23788234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","console","csharp","duplicate","file","files","hash","pyinstaller","python","rust","windows"],"created_at":"2024-10-01T19:35:17.151Z","updated_at":"2025-07-17T02:32:20.151Z","avatar_url":"https://github.com/lemonyte.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Duplicate File Finder\n\nA dead-simple command line tool to find duplicate files on your computer.\n\n\u003e Note: the repository is currently undergoing some restructuring.\n\u003e The main branch no longer contains the code.\n\u003e Please see the branches listed under [Implementations](#implementations) to view the code.\n\n## Implementations\n\nThis tool is implemented in multiple languages.\nIf you're looking for the code, please see the branches below for implementations.\n\n| Language | Branch | Status |\n| -------- | ------ | ------ |\n| Python | [python](https://github.com/lemonyte/dff/tree/python) | Implemented |\n| Rust | [rust](https://github.com/lemonyte/dff/tree/rust) | Implemented |\n| C# | [csharp](https://github.com/lemonyte/dff/tree/csharp) | In development |\n| Others | - | Planned |\n\n## Specification\n\nThe main branch contains an [abstract specification](src/spec.md) of the behavior and structure of the tool.\nThe specification is language-agnostic and can be used as a reference for implementing the tool in other languages.\n\n## How it works\n\nThe tool works by filtering files in three stages:\n\n1. By size\n2. By partial hash\n3. By full hash\n\n### Size\n\nA file size in bytes is queried for every file.\nFiles with matching sizes continue onto the next stage.\n\n### Partial hash\n\nA partial hash is calculated for every file.\nA partial hash is an MD5 hash of the first, last, and middle 64 kilobytes of a file.\nCalculating a partial hash before a full hash saves processing time by eliminating files with matching sizes but sufficiently different content.\nFiles with matching partial hashes continue onto the next stage.\nIf the file is smaller than 192 kilobytes (the minimum size needed for a partial hash) a full hash is calculated instead since it is equivalent.\n\n### Full hash\n\nA full MD5 hash is calculated for every file.\nFiles that have matching hashes are duplicates.\n\n## Disclaimer\n\nI am not liable for any data loss, damage, or any other consequences resulting from corruption or general use of this software.\nSecurity is not guaranteed.\nUse at your own risk.\n\n## License\n\n[MIT License](license.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemonyte%2Fdff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemonyte%2Fdff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemonyte%2Fdff/lists"}