{"id":44726311,"url":"https://github.com/barrust/dup-file-finder","last_synced_at":"2026-02-15T17:46:10.971Z","repository":{"id":332530561,"uuid":"1124423207","full_name":"barrust/dup-file-finder","owner":"barrust","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-08T00:26:24.000Z","size":51,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-31T22:36:41.632Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/barrust.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-29T02:04:40.000Z","updated_at":"2026-01-08T00:26:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/barrust/dup-file-finder","commit_stats":null,"previous_names":["barrust/dup-file-finder"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/barrust/dup-file-finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrust%2Fdup-file-finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrust%2Fdup-file-finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrust%2Fdup-file-finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrust%2Fdup-file-finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/barrust","download_url":"https://codeload.github.com/barrust/dup-file-finder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrust%2Fdup-file-finder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29486040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T15:33:17.885Z","status":"ssl_error","status_checked_at":"2026-02-15T15:32:53.698Z","response_time":118,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-15T17:46:10.498Z","updated_at":"2026-02-15T17:46:10.958Z","avatar_url":"https://github.com/barrust.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Duplicate File Finder\n\n![License](https://img.shields.io/badge/license-MIT-blue.svg)\n[![GitHub release](https://img.shields.io/github/release/barrust/dup-file-finder.svg)](https://github.com/barrust/dup-file-finder/releases)\n[![Build Status](https://github.com/barrust/dup-file-finder/workflows/Python%20package/badge.svg)](https://github.com/barrust/dup-file-finder/actions?query=workflow%3A%22Python+package%22)\n[![PyPI Release](https://badge.fury.io/py/dup-file-finder.svg)](https://pypi.org/project/dup-file-finder/)\n[![Downloads](https://pepy.tech/badge/dup-file-finder)](https://pepy.tech/project/dup-file-finder)\n\nA Python library to find and manage duplicate files. It scans directories, identifies duplicate files using hash algorithms, stores the information in a SQLite database, and provides tools to manage and delete duplicates.\n\n## Features\n\n- 🔍 **Fast duplicate detection** using SHA256 or MD5 hashing\n- 💾 **SQLite database** for storing file information\n- 🗑️ **Safe deletion** with dry-run mode and confirmation prompts\n- 🖥️ **Command-line interface** for easy automation\n- 📊 **Statistics** about scanned files and duplicates\n\n## Documentation\n\n[Documentation is hosted on readthedocs.org](https://dup-file-finder.readthedocs.io/en/latest/)\n\n## Installation\n\nFrom PyPi\n\n```bash\npip install dup-file-finder\n```\n\nOr install from source:\n\n```bash\ngit clone https://github.com/barrust/dup-file-finder.git\ncd dup-file-finder\npip install -e .\n```\n\n## Quick Start\n\n### Using the CLI\n\n#### 1. Scan a directory\n```bash\ndupFileFinder scan /path/to/directory\n```\n\n#### 2. Find duplicates\n```bash\ndupFileFinder find --show-all\n```\n\n#### 3. View statistics\n```bash\ndupFileFinder stats\n```\n\n#### 4. Delete duplicates (dry run)\n```bash\ndupFileFinder delete --dry-run\n```\n\n#### 5. Delete duplicates (for real)\n```bash\ndupFileFinder delete --confirm\n```\n\n### Using as a Library\n\n```python\nfrom dup_file_finder import DuplicateFileFinder\n\n# Initialize the finder\nfinder = DuplicateFileFinder(db_path=\"my_duplicates.db\")\n\n# Scan a directory\ncount = finder.scan_directory(\"/path/to/directory\", recursive=True)\nprint(f\"Scanned {count} files\")\n\n# Find duplicates\nduplicates = finder.find_duplicates()\nfor hash_val, files in duplicates.items():\n    print(f\"Duplicate group: {files}\")\n\n# Get statistics\nstats = finder.get_statistics()\nprint(f\"Total files: {stats['total_files']}\")\nprint(f\"Duplicate files: {stats['duplicate_files']}\")\n\n# Get statistics by file extension\next_stats = finder.get_statistics_by_extension()\nfor ext, data in ext_stats.items():\n    print(f\"{ext}: {data['count']} files, {data['total_size_bytes']} bytes\")\n\n# Delete duplicates (dry run first!)\ndeleted = finder.delete_duplicates(keep_first=True, dry_run=True)\nprint(f\"Would delete: {deleted}\")\n\n# Actually delete\ndeleted = finder.delete_duplicates(keep_first=True, dry_run=False)\nprint(f\"Deleted: {deleted}\")\n```\n\n## CLI Commands\n\n### `scan`\nScan a directory for files and store them in the database.\n\n```bash\ndupFileFinder scan /path/to/directory [--no-recursive]\n```\n\nOptions:\n- `--no-recursive`: Don't scan subdirectories\n\n### `find`\nFind and display duplicate files.\n\n```bash\ndupFileFinder find [--show-all]\n```\n\nOptions:\n- `--show-all`: Display all duplicate files (default: show summary)\n\n### `delete`\nDelete duplicate files.\n\n```bash\ndupFileFinder delete [--keep-first|--keep-last] [--dry-run|--confirm]\n```\n\nOptions:\n- `--keep-first`: Keep the first file alphabetically (default)\n- `--keep-last`: Keep the last file alphabetically\n- `--dry-run`: Show what would be deleted without deleting (default)\n- `--confirm`: Actually delete files\n\n### `stats`\nDisplay statistics about scanned files.\n\n```bash\ndupFileFinder stats [--by-extension]\n```\n\nOptions:\n- `--by-extension`: Show statistics grouped by file extension\n\n### `clear`\nClear all data from the database.\n\n```bash\ndupFileFinder clear --confirm\n```\n\n## Database\n\nBy default, dupFileFinder uses a SQLite database file named `deduper.db` in the current directory. You can specify a custom database path:\n\n```bash\ndupFileFinder --db /path/to/custom.db scan /directory\n```\n\nThe database stores:\n- File paths (absolute paths)\n- File hashes (SHA256 by default)\n- File sizes\n- File extensions (for filtering and statistics)\n- Scan timestamps\n\n**Note:** If you have an existing database from an earlier version without the extension column, you'll need to rebuild it by clearing and rescanning your files.\n\n## Safety Features\n\n- **Dry run mode** by default for deletions\n- **Confirmation prompts** for destructive operations\n- **Keeps one copy** of each duplicate file\n- **Error handling** for inaccessible files\n- **Database transactions** for data integrity\n\n## Example Usage\n\nSee `example.py` for a complete working example. Run it with:\n\n```bash\npython example.py\n```\n\n## License\n\nMIT License - see LICENSE file for details.\n\n## Author\n\nTyler Barrus\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarrust%2Fdup-file-finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbarrust%2Fdup-file-finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarrust%2Fdup-file-finder/lists"}