{"id":31581378,"url":"https://github.com/cqllum/cloudstore","last_synced_at":"2025-10-05T21:56:50.501Z","repository":{"id":318059194,"uuid":"1067342884","full_name":"cqllum/CloudStore","owner":"cqllum","description":"Flask-based multi-cloud file storage system with automatic replication across Local, Mega, Dropbox, AWS S3, and Google Cloud. Features web UI, user management, and RESTful API.","archived":false,"fork":false,"pushed_at":"2025-10-04T19:03:07.000Z","size":93,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-04T21:20:25.200Z","etag":null,"topics":["aws","distributed-storage","distributed-storage-system","distributed-systems","dropbox","flask","gcp","mega","python","s3","storage","whitelabel"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cqllum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-30T18:11:43.000Z","updated_at":"2025-10-04T19:03:11.000Z","dependencies_parsed_at":"2025-10-04T21:20:29.109Z","dependency_job_id":null,"html_url":"https://github.com/cqllum/CloudStore","commit_stats":null,"previous_names":["cqllum/cloudstore"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/cqllum/CloudStore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqllum%2FCloudStore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqllum%2FCloudStore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqllum%2FCloudStore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqllum%2FCloudStore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cqllum","download_url":"https://codeload.github.com/cqllum/CloudStore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqllum%2FCloudStore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278526241,"owners_count":26001326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","distributed-storage","distributed-storage-system","distributed-systems","dropbox","flask","gcp","mega","python","s3","storage","whitelabel"],"created_at":"2025-10-05T21:56:46.848Z","updated_at":"2025-10-05T21:56:50.496Z","avatar_url":"https://github.com/cqllum.png","language":"HTML","readme":"\u003cimg src=\"https://i.imgur.com/7rFn1Ab.png\"\u003e\n\n# CloudStore - Decentralized File Storage System\n\n\u003e **🚧 Work in Progress** - This project is actively being developed and improved.\n\nA Flask-based distributed file storage system that provides a unified interface for storing files across multiple cloud providers and storage nodes with automatic replication and load balancing.\n\n## Features\n\n- **Multi-Cloud Storage** - Support for Local, Mega.io, Dropbox, AWS S3, and Google Cloud Storage\n- **Automatic Replication** - Configurable file replication across multiple storage brokers (this can be replicated in settings)\n- **Smart Load Balancing** - Intelligent file distribution based on broker usage and availability\n- **User Management** - Complete authentication system with admin controls\n- **Virtual File System** - Hierarchical directory structure with folder management\n- **Real-time Monitoring** - Storage usage tracking and broker health monitoring\n- **Web Interface** - Modern responsive UI for file management\n- **RESTful API** - Complete API for programmatic access\n- **Whitelabel Support** - Customizable branding and themes\n\n## Installation Guide\n\n### Prerequisites\n- Python 3.7 or higher\n- pip package manager\n- Git (for cloning)\n\n### Step-by-Step Installation\n\n1. **Clone the repository**:\n```bash\ngit clone https://github.com/cqllum/Cloudstore\ncd Cloudstore\n```\n\n2. **Create virtual environment** (recommended):\n```bash\npython -m venv cloudstore-env\n\n# On Windows:\ncloudstore-env\\Scripts\\activate\n\n# On macOS/Linux:\nsource cloudstore-env/bin/activate\n```\n\n3. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n4. **Configure environment**:\n```bash\ncp .env.example .env\n```\n\nEdit `.env` file with your settings:\n```bash\nSECRET_KEY=your-unique-secret-key-here\nPORT=5000\nDEBUG=False\n```\n\n5. **Initialize the database**:\n```bash\npython app.py\n```\nThe application will automatically create the SQLite database and default admin user.\n\n6. **Access the application**:\n   - Open browser: http://localhost:5000\n   - Login with: `admin/admin`\n   - Change default password immediately\n\n### First-Time Setup\n\n1. **Login as admin** and change the default password\n2. **Add storage brokers** (Local storage is created automatically)\n3. **Create regular users** if needed\n4. **Configure system settings** (replication, themes, etc.)\n\n### Production Deployment\n\n1. **Set environment variables**:\n```bash\nexport SECRET_KEY=\"your-production-secret-key\"\nexport DEBUG=False\nexport PORT=5000\n```\n\n2. **Use production WSGI server**:\n```bash\npip install gunicorn\ngunicorn -w 4 -b 0.0.0.0:5000 app:app\n```\n\n3. **Set up reverse proxy** (nginx recommended)\n4. **Configure SSL/TLS** for HTTPS\n5. **Set up regular backups** of the SQLite database\n\n## Dashboard Pages Overview\n\n### Login Page (`/`)\nLogin/registration\n\u003cimg src=\"https://i.imgur.com/r44PzvK.png\"\u003e\n\n\n### User Dashboard (`/dashboard`)\nThe main file management interface for regular users:\n\u003cimg src=\"https://i.imgur.com/GcYS2Mm.png\"\u003e\n\n\n**Features:**\n- **File Grid/List/Detail Views** - Switch between different file display modes\n- **Upload Files** - Drag-and-drop or browse to upload files\n- **Create Files/Folders** - Built-in text editor for creating new files\n- **Search Functionality** - Find files across your storage\n- **File Operations** - Download, rename, move, copy, delete files\n- **Breadcrumb Navigation** - Easy folder navigation\n- **Storage Usage** - Real-time storage quota tracking\n- **Upload Queue** - Monitor file upload progress\n- **Drag \u0026 Drop** - Move files between folders\n- **Context Menus** - Right-click for quick actions\n- **File Metadata** - View file details and replica locations\n\n\n### Admin Panel (`/admin`)\nComprehensive administration interface:\n\u003cimg src=\"https://i.imgur.com/nM4dyJ8.png\"\u003e\n\n**User Management:**\n- Create, edit, and delete user accounts\n- Set admin privileges\n- View user storage usage and file counts\n- Browse individual user files\n- Reset user passwords\n\n**System Monitoring:**\n- Real-time broker status monitoring\n- Storage usage across all brokers\n- System health indicators\n- Background task monitoring\n\n### Storage Browser (`/browse`)\nDirect storage broker management (Admin only):\n\u003cimg src=\"https://i.imgur.com/Yn601d8.png\"\u003e\n\n**Features:**\n- Browse raw storage on each broker\n- View actual files stored on cloud providers\n- Delete orphaned files\n- Monitor broker connectivity\n- Inspect storage paths and file distribution\n\n### Broker Management (`/manage`)\nStorage provider configuration (Admin only):\n\u003cimg src=\"https://i.imgur.com/FsT8RY2.png\"\u003e\n\u003cimg src=\"https://i.imgur.com/UmCTSex.png\"\u003e\n\n**Broker Operations:**\n- Add new storage brokers (Local, Mega, Dropbox, S3, GCP)\n- Edit broker credentials and settings\n- Enable/disable brokers\n- Test broker connectivity\n- View broker storage statistics\n- Remove unused brokers\n\n### Storage Space Monitor (`/storage_space`)\nDetailed storage analytics (Admin only):\n\u003cimg src=\"https://i.imgur.com/uvlRV18.png\"\u003e\n\n**Analytics:**\n- Total storage across all brokers\n- Virtual vs actual storage usage\n- Per-broker utilization charts\n- Storage efficiency metrics\n- Replication overhead analysis\n\n### Admin Settings (`/admin/settings_page`)\nSystem configuration interface:\n\u003cimg src=\"https://i.imgur.com/wsyDPvx.png\"\u003e\n\n**System Settings:**\n- Replication count (how many copies of each file)\n- Broker refresh interval (health check frequency)\n- Storage quotas per user\n\n**Whitelabel Customization:**\n- Site name and branding\n- Custom CSS styling\n- Color scheme configuration\n- Theme presets\n- Logo and favicon upload\n\n### User File Viewer (`/admin/user_files_view/\u003cuser_id\u003e`)\nAdmin view of individual user files:\n\u003cimg src=\"https://i.imgur.com/uJYOJ4H.png\"\u003e\n\n**File Management:**\n- View all files for specific user\n- See file replica locations\n- Delete user files\n- Monitor user storage patterns\n- Troubleshoot file access issues\n\n## Configuration\n\n### Environment Variables\n```bash\nSECRET_KEY=your-secret-key-here\nPORT=5000\nDEBUG=False\n```\n\n### Storage Brokers\n\n#### Local Storage\n```json\n{\n  \"name\": \"Local Storage\",\n  \"type\": \"local\",\n  \"config\": {\n    \"path\": \"storage/local\"\n  }\n}\n```\n\n#### Mega.io\n```json\n{\n  \"name\": \"Mega Storage\",\n  \"type\": \"mega\",\n  \"config\": {\n    \"email\": \"your-email@example.com\",\n    \"password\": \"your-password\"\n  }\n}\n```\n\n#### Dropbox\n```json\n{\n  \"name\": \"Dropbox\",\n  \"type\": \"dropbox\",\n  \"config\": {\n    \"access_token\": \"your-dropbox-access-token\"\n  }\n}\n```\n\n## API Reference\n\n### Authentication\n```bash\n# Register user\nPOST /register\n{\n  \"username\": \"user\",\n  \"password\": \"password\"\n}\n\n# Login\nPOST /login\n{\n  \"username\": \"user\",\n  \"password\": \"password\"\n}\n```\n\n### Storage Management\n```bash\n# Add storage broker\nPOST /add_broker\n{\n  \"name\": \"My Storage\",\n  \"type\": \"local\",\n  \"config\": {\"path\": \"/storage\"}\n}\n\n# Upload file\nPOST /upload\nContent-Type: multipart/form-data\nfile: \u003cfile-data\u003e\nvirtual_path: \"/documents/\"\n\n# Download file\nGET /download_file?filename=document.pdf\u0026path=/documents/\n\n# Search files\nGET /search?q=document\u0026path=/documents/\n\n# List files\nGET /files?path=/documents/\n```\n\n### File Operations\n```bash\n# Create folder\nPOST /create_folder\n{\n  \"path\": \"/new-folder/\"\n}\n\n# Rename file\nPOST /rename_file\n{\n  \"old_name\": \"old.txt\",\n  \"new_name\": \"new.txt\",\n  \"virtual_path\": \"/\"\n}\n\n# Delete file\nDELETE /delete_file\n{\n  \"filename\": \"document.pdf\",\n  \"virtual_path\": \"/documents/\"\n}\n\n# Move file\nPOST /move_file\n{\n  \"filename\": \"document.pdf\",\n  \"old_path\": \"/documents/\",\n  \"new_path\": \"/archive/\"\n}\n```\n\n## Architecture\n\n### System Components\n1. **Flask Web Server** - HTTP API and web interface\n2. **SQLite Database** - Metadata storage and user management\n3. **Storage Brokers** - Pluggable cloud storage adapters\n4. **Replication Engine** - Automatic file distribution and redundancy\n5. **Background Workers** - Broker health monitoring and maintenance\n\n### Database Schema\n- **users** - User accounts and authentication\n- **brokers** - Storage provider configurations\n- **files** - File metadata and virtual paths\n- **file_replicas** - File location tracking across brokers\n- **settings** - System configuration\n\n### Storage Flow\n1. File uploaded via API/UI\n2. System selects optimal brokers based on load balancing\n3. File replicated across configured number of brokers\n4. Metadata stored in database with replica locations\n5. Background workers monitor broker health\n\n## Administration\n\n### Admin Features\n- User management (create, edit, delete users)\n- Storage broker configuration\n- System settings (replication count, refresh intervals)\n- Storage usage monitoring\n- Theme customization\n\n### System Settings\n- **Replica Count**: Number of copies per file (default: 2)\n- **Broker Refresh**: Health check interval in minutes (default: 5)\n- **Storage Limit**: Per-user storage quota (default: 5GB)\n\n## Development\n\n### Adding New Storage Brokers\n\n1. **Implement CloudBroker interface**:\n```python\nclass CustomBroker(CloudBroker):\n    def upload_file(self, file_data, filename):\n        # Upload implementation\n        return storage_path\n    \n    def download_file(self, storage_path):\n        # Download implementation\n        return file_data\n    \n    def delete_file(self, storage_path):\n        # Delete implementation\n        pass\n    \n    def list_files(self, path=''):\n        # List implementation\n        return file_list\n    \n    def get_storage_info(self):\n        # Storage info implementation\n        return {'total': 0, 'used': 0, 'available': 0}\n```\n\n2. **Register in broker factory**:\n```python\n# In cloud_brokers.py\nbrokers = {\n    'local': LocalBroker,\n    'custom': CustomBroker,\n    # ...\n}\n```\n \n\n## Security\n\n- Password hashing \n- Session-based authentication\n- File access control per user\n- Admin privilege separation\n- Secure file upload handling\n\n## Limitations\n\n- 5GB storage limit per user (configurable)\n- SQLite database (suitable for small-medium deployments)\n- No end-to-end encryption (files stored as-is on brokers)\n- Limited concurrent user support\n\nFYI - currently this is a POC / beta product - encryption should be added and better distribution of secured areas on brokers should be introduced.\nThis shouldn't be used for anything sensitive, merely just a way to remove clutter on your own storage areas.\n\nThis goal here is to have a whitelabel solution that can be integrated into your own platforms (i.e hosting companies) and have sufficient user management whilst having good security practices.\nI will continue to work on this over time but feel free to contribute, and also raise issues where you have a problem.\n\nAlso note that the architecture here (specifiaclly SQLite was to demonstrate and build the product quickly - production environments should and will not be using this on the release of that..)\n\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests if applicable\n5. Submit a pull request\n\n \n  \n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcqllum%2Fcloudstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcqllum%2Fcloudstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcqllum%2Fcloudstore/lists"}