{"id":21031982,"url":"https://github.com/navid2zp/dups","last_synced_at":"2025-05-15T12:31:27.612Z","repository":{"id":53533664,"uuid":"277811205","full_name":"Navid2zp/dups","owner":"Navid2zp","description":"A CLI tool to find/remove duplicate files supporting multi-core and different algorithms (MD5, SHA256, and XXHash).","archived":false,"fork":false,"pushed_at":"2024-05-02T22:26:47.000Z","size":64,"stargazers_count":25,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-03T09:12:24.223Z","etag":null,"topics":["duplicates","go","golang","md5","sha256","xxhash"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Navid2zp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-07T12:32:26.000Z","updated_at":"2025-01-07T09:38:50.000Z","dependencies_parsed_at":"2024-06-19T05:17:20.411Z","dependency_job_id":"45e98c9a-4534-410f-a1bb-801bfa1ed69f","html_url":"https://github.com/Navid2zp/dups","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Navid2zp%2Fdups","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Navid2zp%2Fdups/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Navid2zp%2Fdups/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Navid2zp%2Fdups/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Navid2zp","download_url":"https://codeload.github.com/Navid2zp/dups/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254341131,"owners_count":22054984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicates","go","golang","md5","sha256","xxhash"],"created_at":"2024-11-19T12:39:27.689Z","updated_at":"2025-05-15T12:31:26.603Z","avatar_url":"https://github.com/Navid2zp.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\t\u003cimg alt=\"dups\" src=\"https://raw.githubusercontent.com/Navid2zp/dups/master/dups-h.png\" /\u003e\n\u003c/p\u003e\n\n\n# dups\ndups is a CLI tool to find and remove duplicate files using different hash algorithms (MD5, SHA256 and XXHash) with multi-core support.\n\n### Install\nDownload binaries:\n\n[Release Page][1]\n\nTo use in a go project:\n```\ngo get github.com/Navid2zp/dups\n```\n\n### Usage\n\n#### CLI\n\nAvailable Commands:\n\n| Command | Description | \n|---|---|\n| clean | Finds duplicate files in a given path and deletes them.  |\n| scan |  Finds duplicate files in a given path but doesn't delete them. | \n| help |  Help about any command |\n\nFlags:\n\n| flag | Description | \n|---|---|\n| --algorithm string | algorithm to use (md5/sha256/xxhash) (default \"md5\")  |\n| -f, --flat |  flat output, no extra info (only prints duplicate files) | \n| -r, --full |  full search (search in sub-directories too) |\n| --min-size int | minimum file size to scan in bytes (default 10) |\n| -s, --single-core | use single cpu core |\n\n\n**Examples:**\n\nRemove duplicates bigger than 1KB using multiple cpu cores:\n```\ndups clean path/to/directory --min-size 1024\n```\n\nFind duplicates and write them into `file.txt`:\n```\ndups scan path/to/directory -f \u003e\u003e file.txt\n```\n\nFind and list duplicates using single cpu core and `XXHash` algorithm:\n```\ndups scan path/to/directory -s --algorithm xxhash\n```\n\n#### Go code:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/Navid2zp/dups\"\n)\n\nfunc main()  {\n\t// list all files including files in any sub-directory\n\tfiles, err := dups.GetFiles(\"path/to/directory\", true)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n        // group files based on their file size\n        fileGroups, totalFiles := dups.GroupFiles(files, 128)\n\n\t// collect hashes for groups with more than one file\n\t// singleThread: use a single thread\n\t// flatt: don't print the process bar or any other information\n\thashes := dups.CollectHashes(fileGroups, false, dups.XXHash, false, totalFiles)\n\tduplicates, filesCount, duplicatesCount := dups.GetDuplicates(hashes)\n\tfmt.Println(\"total number of files with duplicates:\", filesCount)\n\tfmt.Println(\"total number of duplicate files:\", duplicatesCount)\n\n\tfreedSize, deletedCount, err := dups.RemoveDuplicates(duplicates)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tfmt.Println(\"remove\", deletedCount, \"files\")\n\tfmt.Println(\"freed a total of \", freedSize, \"bytes\")\n}\n```\n\n#### Notes:\n\n- Use single core option (`-s`) if files are big (depending on your disk type).\n- Use XXHash algorithm for fast scanning and MD5/SHA256 for safest scanning or if the number of files is huge.\n\n#### Build from source:\n\n`go build -tags multicore` if you are building using Go \u003c 1.5 or edit `runtime.GOMAXPROCS()` manually to support multi-core.\n\n\nLicense\n----\n\n[Apache][2]\n\n\n[1]: https://github.com/Navid2zp/dups/releases\n[2]: https://github.com/Navid2zp/dups/blob/master/LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnavid2zp%2Fdups","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnavid2zp%2Fdups","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnavid2zp%2Fdups/lists"}