{"id":24746536,"url":"https://github.com/ahmadasjad/duplicate-finder","last_synced_at":"2026-05-07T00:39:10.510Z","repository":{"id":273713088,"uuid":"920625432","full_name":"ahmadasjad/duplicate-finder","owner":"ahmadasjad","description":"Easily find and remove duplicate files to keep your local system or Google Drive clean—featuring a cleaner, more efficient approach that works seamlessly both on your local machine and in the cloud.","archived":false,"fork":false,"pushed_at":"2025-07-06T02:05:01.000Z","size":130,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-06T02:36:02.538Z","etag":null,"topics":["cleaning","deduplication","duplicate-finder","file-cleaner","file-management","google-drive"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahmadasjad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-22T13:44:18.000Z","updated_at":"2025-07-06T01:47:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"feb5df2e-aed7-476b-8118-25b2138b001c","html_url":"https://github.com/ahmadasjad/duplicate-finder","commit_stats":null,"previous_names":["ahmadasjad/duplicate-finder"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ahmadasjad/duplicate-finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmadasjad%2Fduplicate-finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmadasjad%2Fduplicate-finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmadasjad%2Fduplicate-finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmadasjad%2Fduplicate-finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahmadasjad","download_url":"https://codeload.github.com/ahmadasjad/duplicate-finder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmadasjad%2Fduplicate-finder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267967720,"owners_count":24173566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cleaning","deduplication","duplicate-finder","file-cleaner","file-management","google-drive"],"created_at":"2025-01-28T04:29:05.372Z","updated_at":"2026-05-07T00:39:10.483Z","avatar_url":"https://github.com/ahmadasjad.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Duplicate File Finder\n\nA Streamlit-based application to find and manage duplicate files across multiple storage providers including local filesystem, Google Drive, OneDrive, and Dropbox.\n\n## Table of Contents\n- [Features](#features)\n  - [Safety Features](#safety-features)\n- [Storage Providers](#storage-providers)\n  - [Provider Features](#provider-features)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Requirements](#requirements)\n- [File Metadata](#file-metadata)\n- [Development](#development)\n  - [Architecture](#architecture)\n    - [Adding New Providers](#adding-new-providers)\n  - [Project Structure](#project-structure)\n  - [Running Tests](#running-tests)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n- **Multi-Provider Support**: Scan files from local filesystem and cloud storage providers\n- **Modular Architecture**: Extensible storage provider system with factory pattern\n- **File Preview**: View images and PDFs directly in the app\n- **Detailed Metadata**: File name, path, size, extension, creation/modification dates\n- **Docker Support**: Easy deployment with Docker Compose\n- **Provider Selection**: Dropdown interface to choose between storage providers\n\n### Safety Features\n- **Deletion Protection**: Prevents removing all files in a duplicate group\n- **Preview Before Action**: Visual confirmation of files before deletion\n- **Explicit Confirmation**: Requires user confirmation for destructive operations\n- **Read-Only Mounts**: Docker volumes mounted read-only by default for safety\n- **Provider Isolation**: Each storage provider operates independently\n\n## Storage Providers\n\nThe application supports multiple storage providers through a modular architecture:\n\n- **Local Filesystem**: Scan directories on your local machine or mounted drives\n- **Google Drive**: Scan files in your Google Drive (authentication required)\n- **OneDrive**: Scan files in your Microsoft OneDrive (authentication required)\n- **Dropbox**: Scan files in your Dropbox (authentication required)\n\nEach provider is implemented as a separate module with a common interface, making it easy to add new providers.\n\n### Provider Features\n\n#### Google Drive Integration\n- Browser-based OAuth2 authentication\n- File metadata caching for faster rescans\n- Support for Team Drives and shared folders\n- Safe file operations (moves to trash instead of permanent deletion)\n- See [Google Drive Setup Guide](docs/GOOGLE_DRIVE_SETUP.md) for detailed instructions\n\n## Installation\n\nFor detailed installation instructions, including Google Colab, Docker, and manual installation methods, please see the [Installation Guide](docs/INSTALLATION.md).\n\nAvailable installation methods:\n- Google Colab (quick start with no local setup)\n- Docker (recommended for most users)\n- Manual installation (for development)\n\n## Usage\n\nFor detailed usage instructions, including best practices, troubleshooting, and advanced features, please see the [Usage Guide](docs/USAGE.md).\n\nQuick start:\n1. Select a storage provider (Local Filesystem, Google Drive, OneDrive, Dropbox)\n2. Configure access and scan settings\n3. Review and manage duplicate files with built-in safety features\n4. Preview and verify files before any actions\n5. Safely remove unnecessary duplicates\n\n## Requirements\n\n- Python 3.8+\n- Streamlit\n- Pillow (for image previews)\n- pdfplumber (for PDF previews)\n- Docker \u0026 Docker Compose (for containerized deployment)\n\n\n\n## File Metadata\n\nFor each file, the application displays:\n- File name and full path\n- File extension and MIME type\n- File size (human-readable format)\n- Creation and modification timestamps\n- Provider-specific metadata (when available)\n\n## Development\n\n### Architecture\n\nThe application uses a modular storage provider architecture:\n\n```\napp/storage_providers/\n├── __init__.py          # Package initialization\n├── base.py              # BaseStorageProvider abstract class\n├── factory.py           # StorageProviderFactory for provider management\n├── local_filesystem.py  # Local file system implementation\n├── google_drive.py      # Google Drive implementation (placeholder)\n├── onedrive.py          # OneDrive implementation (placeholder)\n├── dropbox.py           # Dropbox implementation (placeholder)\n└── README.md            # Provider architecture documentation\n```\n\n**Key Components:**\n- **BaseStorageProvider**: Abstract base class defining the common interface\n- **StorageProviderFactory**: Factory pattern for creating and managing providers\n- **Provider Implementations**: Each cloud service has its own dedicated module\n- **Legacy Compatibility**: The main `storage_providers.py` imports from the new structure\n\n#### Adding New Providers\n\nTo add a new storage provider:\n\n1. Create a new file in `app/storage_providers/`\n2. Inherit from `BaseStorageProvider`\n3. Implement required methods: `connect()`, `list_files()`, `delete_file()`\n4. Add to the factory in `factory.py`\n5. Update UI dropdown in `app/ui.py`\n\n### Project Structure\n```\nduplicate-finder/\n├── app/                     # Main application code\n│   ├── storage_providers/   # Modular provider system\n│   ├── main.py             # Streamlit entry point\n│   ├── ui.py               # User interface components\n│   ├── file_operations.py  # File handling utilities\n│   └── utils.py            # Helper functions\n├── test_data/              # Sample files for testing\n├── docker-compose.yml      # Docker configuration\n├── Dockerfile              # Container build instructions\n├── requirements.txt        # Python dependencies\n└── start-docker.sh         # Quick start script\n```\n\n### Running Tests\n```bash\n# Manual testing with sample data\ncd test_data\n# Create some duplicate files for testing\n\n# Run the app and test provider functionality\nstreamlit run app/main.py\n```\n\n## Contributing\n\nContributions are welcome! Please open an issue or pull request on GitHub.\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmadasjad%2Fduplicate-finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahmadasjad%2Fduplicate-finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmadasjad%2Fduplicate-finder/lists"}