{"id":19853856,"url":"https://github.com/magnetikonline/identix","last_synced_at":"2025-05-02T01:30:24.792Z","repository":{"id":147825648,"uuid":"41595005","full_name":"magnetikonline/identix","owner":"magnetikonline","description":"Python utility which will recursively scan one or more given directories for duplicate files.","archived":false,"fork":false,"pushed_at":"2025-02-21T01:09:23.000Z","size":50,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-06T20:24:54.987Z","etag":null,"topics":["duplicate-files","python","sha-1","utility"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/magnetikonline.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-08-29T13:52:07.000Z","updated_at":"2025-02-21T01:09:27.000Z","dependencies_parsed_at":"2024-11-12T14:07:53.318Z","dependency_job_id":"f3274b17-65c8-40ac-9026-bd7bb2f4f387","html_url":"https://github.com/magnetikonline/identix","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnetikonline%2Fidentix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnetikonline%2Fidentix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnetikonline%2Fidentix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magnetikonline%2Fidentix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/magnetikonline","download_url":"https://codeload.github.com/magnetikonline/identix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251969230,"owners_count":21673180,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicate-files","python","sha-1","utility"],"created_at":"2024-11-12T14:07:50.348Z","updated_at":"2025-05-02T01:30:24.512Z","avatar_url":"https://github.com/magnetikonline.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# identix\n\nPython utility which will recursively scan one or more given directories for duplicate files.\n\n- [What is a duplicate?](#what-is-a-duplicate)\n- [Usage](#usage)\n- [Examples](#examples)\n\n## What is a duplicate?\n\nFiles are considered duplicate based on their identical binary representation:\n\n- Files are scanned and grouped by file size to quickly rule out non-duplicates.\n- Grouped files then have SHA-1 hashes calculated - those that match are duplicates.\n\nFiles to consider can optionally be filtered based on:\n\n- One or more glob filespecs.\n- Minimum file size.\n\n## Usage\n\n```\nusage: identix.py [-h] [--include [INCLUDE [INCLUDE ...]]]\n                  [--min-size MIN_SIZE] [--progress]\n                  [--report-file REPORT_FILE]\n                  [--report-file-format {text,JSON}]\n                  scandir [scandir ...]\n\nRecursively scan one or more directories for duplicate files.\n\npositional arguments:\n  scandir               source directory/directories for scanning\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --include [INCLUDE [INCLUDE ...]]\n                        glob filespec(s) to include in scan, if omitted all\n                        files are considered\n  --min-size MIN_SIZE   minimum filesize considered\n  --progress            show progress during file diffing\n  --report-file REPORT_FILE\n                        send duplicate report to file, rather than console\n  --report-file-format {text,JSON}\n                        format of duplicate report file\n```\n\nNotes:\n\n- The `--include` argument evaluates *filename only*, so expects globs such as `*.jpg` or `image*.png`.\n- Omitting `--report-file` output file argument will display results directly on the console\n- Option `--report-file-format` enables `--report-file` as `JSON` - format example:\n\n\t```json\n\t[\n\t  {\n\t    \"sha-1\": \"xxxxx\",\n\t    \"size\": 12345,\n\t    \"fileList\": [\"/path/to/file\",\"/path/to/another/file\"]\n\t  },\n\t  {\n\t    \"sha-1\": \"yyyyy\",\n\t    \"size\": 6789,\n\t    \"fileList\": [\"/path/to/yet/another/file\",\"/one/more/file\"]\n\t  },\n\t]\n\t```\n\n## Examples\n\nScan for duplicates greater than or equal to `2048` bytes in the directories of `/dupe/path/one` and `/dupe/path/two`:\n\n```sh\n$ ./identix.py \\\n  --min-size 2048 \\\n    -- /dupe/path/one /dupe/path/two\n```\n\nFind duplicates that match file globs of `*.jpg` and `*.png` in `/my/images`, write results to `/path/to/report.txt` and display processing progress to console:\n\n```sh\n$ ./identix.py \\\n  --include \"*.jpg\" \"*.png\" \\\n  --progress \\\n  --report-file /path/to/report.txt \\\n    -- /my/images\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmagnetikonline%2Fidentix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmagnetikonline%2Fidentix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmagnetikonline%2Fidentix/lists"}