https://github.com/localghost/dedup

Find and remove duplicated files.
https://github.com/localghost/dedup

deduplicate files

Last synced: about 2 months ago
JSON representation

Find and remove duplicated files.

• Host: GitHub
• URL: https://github.com/localghost/dedup
• Owner: localghost
• License: apache-2.0
• Created: 2022-06-12T19:51:23.000Z (almost 3 years ago)
• Default Branch: master
• Last Pushed: 2022-07-03T20:36:19.000Z (almost 3 years ago)
• Last Synced: 2025-02-07T21:37:23.591Z (3 months ago)
• Topics: deduplicate, files
• Language: Rust
• Homepage:
• Size: 9.77 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# dedup

Find and remove duplicated files.

`dedup` walks recursively a directory, computes SHA1 of each file and removes all but one file matching the same hash and leaves symbolic links in place of the removed files.

Computing hash is done in parallel on half of the available CPU cores on the machine. I have not yet checked whether it actually improves performance - wanted to try out [rayon](https://docs.rs/rayon/latest/rayon/).