{"id":13548368,"url":"https://github.com/m-manu/go-find-duplicates","last_synced_at":"2025-05-16T18:08:45.593Z","repository":{"id":41276955,"uuid":"376264778","full_name":"m-manu/go-find-duplicates","owner":"m-manu","description":"Find duplicate files (photos, videos, music, documents) on your computer, portable hard drives etc.","archived":false,"fork":false,"pushed_at":"2025-02-15T10:33:26.000Z","size":61,"stargazers_count":280,"open_issues_count":0,"forks_count":24,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-04T01:38:09.759Z","etag":null,"topics":["command-line-tool","duplicate-files","file-scanner","go","golang","golang-application","media","utilities"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-manu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-12T10:42:51.000Z","updated_at":"2025-03-30T09:02:10.000Z","dependencies_parsed_at":"2025-03-21T00:01:41.224Z","dependency_job_id":"9b1654d3-9251-4a80-a07d-687b41baac28","html_url":"https://github.com/m-manu/go-find-duplicates","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-manu%2Fgo-find-duplicates","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-manu%2Fgo-find-duplicates/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-manu%2Fgo-find-duplicates/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-manu%2Fgo-find-duplicates/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-manu","download_url":"https://codeload.github.com/m-manu/go-find-duplicates/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248606860,"owners_count":21132428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","duplicate-files","file-scanner","go","golang","golang-application","media","utilities"],"created_at":"2024-08-01T12:01:09.421Z","updated_at":"2025-04-12T17:38:12.234Z","avatar_url":"https://github.com/m-manu.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Go Find Duplicates\n\n[![build-and-test](https://github.com/m-manu/go-find-duplicates/actions/workflows/build-and-test.yml/badge.svg)](https://github.com/m-manu/go-find-duplicates/actions/workflows/build-and-test.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/m-manu/go-find-duplicates)](https://goreportcard.com/report/github.com/m-manu/go-find-duplicates)\n[![Go Reference](https://pkg.go.dev/badge/github.com/m-manu/go-find-duplicates.svg)](https://pkg.go.dev/github.com/m-manu/go-find-duplicates)\n[![License](https://img.shields.io/badge/License-Apache%202-blue.svg)](./LICENSE)\n\n## Introduction\n\nA blazingly-fast simple-to-use tool to find duplicate files (photos, videos, music, documents etc.) on your computer,\nportable hard drives etc.\n\n**Note**:\n\n* This tool just *reads* your files and creates a 'duplicates report' file\n* It does **not** delete or otherwise modify your files in any way 🙂\n* So, it's very safe to use 👍\n\n## How to install?\n\n1. Install Go version at least **1.22**\n    * See: [Go installation instructions](https://go.dev/doc/install)\n2. Run command:\n   ```bash\n   go install github.com/m-manu/go-find-duplicates@latest\n   ```\n3. Add following line in your `.bashrc`/`.zshrc` file:\n   ```bash\n   export PATH=\"$PATH:$HOME/go/bin\"\n   ```\n\n## How to use?\n\n### Run directly (preferred)\n\n```bash\ngo-find-duplicates {dir-1} {dir-2} ... {dir-n}\n```\n\n### Command line options\n\nRunning `go-find-duplicates --help` displays following:\n\n```\ngo-find-duplicates is a tool to find duplicate files and directories\n\nUsage:\n  go-find-duplicates [flags] \u003cdir-1\u003e \u003cdir-2\u003e ... \u003cdir-n\u003e\n\nwhere,\n  arguments are readable directories that need to be scanned for duplicates\n\nFlags (all optional):\n  -x, --exclusions string   path to file containing newline-separated list of file/directory names to be excluded\n                            (if this is not set, by default these will be ignored:\n                            .DS_Store, System Volume Information, $RECYCLE.BIN etc.)\n  -h, --help                display help\n  -m, --minsize uint        minimum size of file in KiB to consider (default 4)\n  -o, --output string       following modes are accepted:\n                            print = just prints the report without creating any file\n                             text = creates a text file in the output directory with basic information\n                              csv = creates a csv file in the output directory with detailed information\n                             json = creates a JSON file in the output directory with basic information\n                             (default \"text\")\n  -f, --outputfile string   output file path (will be created, but directory needs to be writeable)\n  -p, --parallelism uint8   extent of parallelism (defaults to number of cores minus 1)\n  -q, --quiet               quiet mode: no output on stdout/stderr, except for duplicates/errors\n  -t, --thorough            apply thorough check of uniqueness of files\n                            (caution: this makes the scan very slow!)\n      --version             display version (1.8.0) and exit (useful for incorporating this in scripts)\n\nFor more details: https://github.com/m-manu/go-find-duplicates\n```\n\n### Run via Docker\n\n```bash\ndocker run --rm -v /Volumes/PortableHD:/mnt/PortableHD manumk/go-find-duplicates:latest go-find-duplicates -o print /mnt/PortableHD\n```\n\nIn above command:\n\n* option `--rm` removes the container when it exits\n* option `-v` is mounts host directory `/Volumes/PortableHD` as `/mnt/PortableHD` inside the container\n\n## How does this identify duplicates?\n\n**By default**, this tool identifies duplicates if _all_ of the following conditions match:\n\n1. file extension is same\n2. file size is same\n3. CRC32 hash of \"crucial bytes\" is same\n\nIf above default isn't enough for your requirements, you could use the command line option `--thorough` to switch to\nSHA-256 hash of *entire file contents*. But remember, with this, scan becomes much slower!\n\nWhen tested on my portable hard drive containing \u003e172k files (videos, audio files, images and documents), with and\nwithout `--thorough` option, the results were same!\n\n## How to build?\n\n```shell\ngo build \n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-manu%2Fgo-find-duplicates","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-manu%2Fgo-find-duplicates","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-manu%2Fgo-find-duplicates/lists"}