{"id":25013826,"url":"https://github.com/kenzycodex/distributed-file-storage","last_synced_at":"2025-12-30T22:40:02.236Z","repository":{"id":275589177,"uuid":"926498692","full_name":"kenzycodex/distributed-file-storage","owner":"kenzycodex","description":"This project focuses on developing a Server Load Balancer Distributed File Storage, Sharing, and Management System to efficiently handle file operations across multiple storage containers. The system ensures optimized file management, secure access control, and dynamic workload distribution using advanced load-balancing techniques.","archived":false,"fork":false,"pushed_at":"2025-02-22T13:22:29.000Z","size":713,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-22T14:28:31.052Z","etag":null,"topics":["docker","java","javafx","mysql","server","spring","sql"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kenzycodex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-03T11:12:58.000Z","updated_at":"2025-02-22T13:22:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"bc5fede7-1354-48fe-996c-7641069111e0","html_url":"https://github.com/kenzycodex/distributed-file-storage","commit_stats":null,"previous_names":["kenzycodex/distributed-file-storage"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenzycodex%2Fdistributed-file-storage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenzycodex%2Fdistributed-file-storage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenzycodex%2Fdistributed-file-storage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenzycodex%2Fdistributed-file-storage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kenzycodex","download_url":"https://codeload.github.com/kenzycodex/distributed-file-storage/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246277370,"owners_count":20751549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","java","javafx","mysql","server","spring","sql"],"created_at":"2025-02-05T07:16:00.581Z","updated_at":"2025-12-30T22:40:02.231Z","avatar_url":"https://github.com/kenzycodex.png","language":"Java","readme":"# Distributed Storage System\r\n\r\nA scalable distributed file storage system with load balancing capabilities. This system consists of multiple microservices designed to work together:\r\n\r\n- **Load Balancer**: Manages requests and distributes them across storage nodes\r\n- **File Storage Microservices**: Handle file upload, storage, and retrieval\r\n- **MySQL Database**: Stores metadata and system information\r\n\r\n## Table of Contents\r\n\r\n1. [System Architecture](#system-architecture)\r\n2. [Prerequisites](#prerequisites)\r\n3. [Setup Instructions](#setup-instructions)\r\n    - [Development Environment](#development-environment)\r\n    - [Production Environment](#production-environment)\r\n4. [Environment Configuration](#environment-configuration)\r\n5. [API Endpoints](#api-endpoints)\r\n6. [Monitoring and Management](#monitoring-and-management)\r\n7. [Troubleshooting](#troubleshooting)\r\n\r\n## System Architecture\r\n\r\nThe system follows a microservices architecture:\r\n\r\n- **Load Balancer**: Routes client requests to the appropriate storage nodes using various strategies (round-robin, least-connection, etc.)\r\n- **Storage Containers**: Manage file operations and store actual file data\r\n- **MySQL Database**: Centralized metadata storage for the entire system\r\n- **Health Checks**: Regular monitoring of node status and availability\r\n\r\n## Prerequisites\r\n\r\n- Java 17 or higher\r\n- Maven 3.6 or higher\r\n- MySQL 8.0 (development mode) or Docker (production mode)\r\n- Docker and Docker Compose (for production deployment)\r\n- Minimum 8GB RAM recommended\r\n- At least 10GB free disk space\r\n\r\n## Setup Instructions\r\n\r\n### Development Environment\r\n\r\n#### 1. Database Setup\r\n\r\nIn development mode, you need to create a MySQL database on your local MySQL server:\r\n\r\n```sql\r\nCREATE DATABASE loadbalancer;\r\nCREATE USER 'loadbalancer'@'localhost' IDENTIFIED BY 'loadbalancer';\r\nGRANT ALL PRIVILEGES ON loadbalancer.* TO 'loadbalancer'@'localhost';\r\nFLUSH PRIVILEGES;\r\n```\r\n\r\n#### 2. Clone the Repository\r\n\r\n```bash\r\ngit clone \u003crepository-url\u003e\r\ncd distributed-storage-system\r\n```\r\n\r\n#### 3. Start the Load Balancer\r\n\r\n```bash\r\ncd load-balancer\r\n./start-dev.bat  # Windows\r\n# OR\r\n./start-dev.sh   # Linux/Mac\r\n```\r\n\r\nThis script will:\r\n- Build the application using Maven\r\n- Start the application with the 'dev' profile\r\n- The load balancer will be available at http://localhost:8080\r\n\r\n#### 4. Start the File Storage Service\r\n\r\nOpen a new terminal and run:\r\n\r\n```bash\r\ncd file-storage\r\n./start-dev.bat  # Windows\r\n# OR\r\n./start-dev.sh   # Linux/Mac\r\n```\r\n\r\nThis will start a file storage service instance at http://localhost:8081/api/v1\r\n\r\nYou can start multiple storage instances by manually setting different ports, e.g.:\r\n```bash\r\nmvnw spring-boot:run -Dspring-boot.run.profiles=dev -Dserver.port=8082\r\n```\r\n\r\n### Production Environment\r\n\r\nIn production mode, all services run as Docker containers. The system uses Docker Compose to manage all components.\r\n\r\n#### 1. Configure Environment Variables\r\n\r\nReview and modify the `.env` file in the project root as needed:\r\n\r\n```\r\n# Example of .env configuration\r\nMYSQL_ROOT_PASSWORD=secure_password\r\nMYSQL_USER=loadbalancer\r\nMYSQL_PASSWORD=loadbalancer\r\nSTORAGE_CAPACITY=2147483648  # 2GB per container\r\n```\r\n\r\n#### 2. Start the System\r\n\r\nFrom the project root directory:\r\n\r\n```bash\r\ndocker-compose up -d\r\n```\r\n\r\nThis will:\r\n- Build all necessary Docker images\r\n- Start the MySQL database container\r\n- Create 4 file storage service containers\r\n- Start the load balancer container\r\n- Configure all networking and volumes\r\n\r\nThe load balancer will be accessible at http://localhost:8080 and will distribute requests to the storage containers.\r\n\r\n#### 3. Scale the System (Optional)\r\n\r\nTo adjust the number of storage containers:\r\n\r\n```bash\r\ndocker-compose up -d --scale storage-container=6\r\n```\r\n\r\n## Environment Configuration\r\n\r\n### Key Configuration Files\r\n\r\n- `.env`: Main environment variables for Docker deployment\r\n- `application.yml`: Common application settings for all environments\r\n- `application-dev.yml`: Development-specific settings\r\n- `application-prod.yml`: Production-specific settings\r\n\r\n### Important Configuration Parameters\r\n\r\n#### Load Balancer\r\n\r\n| Parameter | Description | Default |\r\n|-----------|-------------|---------|\r\n| `LOAD_BALANCER_STRATEGY` | Strategy for distributing requests | round-robin |\r\n| `HEALTH_CHECK_INTERVAL` | Time between node health checks (ms) | 30000 |\r\n| `JAVA_OPTS` | JVM configuration for load balancer | -Xmx1g -Xms512m |\r\n\r\n#### Storage Containers\r\n\r\n| Parameter | Description | Default |\r\n|-----------|-------------|---------|\r\n| `STORAGE_CAPACITY` | Storage space per container (bytes) | 1073741824 (1GB) |\r\n| `MAX_FILE_UPLOAD_SIZE` | Maximum file size for uploads | 100MB |\r\n| `CONTAINER_ID` | Unique ID for each container | Automatically assigned |\r\n\r\n## API Endpoints\r\n\r\n### Load Balancer Endpoints\r\n\r\n- `GET /api/v1/health`: System health check\r\n- `POST /api/v1/nodes/register`: Register a new storage node\r\n- `GET /api/v1/metrics`: System performance metrics\r\n\r\n### File Storage Endpoints\r\n\r\n- `POST /api/v1/files`: Upload a file\r\n- `GET /api/v1/files/{id}`: Download a file\r\n- `DELETE /api/v1/files/{id}`: Delete a file\r\n- `GET /api/v1/health`: Container health status\r\n\r\n## Monitoring and Management\r\n\r\nThe system includes built-in monitoring through Spring Boot Actuator:\r\n\r\n- Health endpoints: `/actuator/health`\r\n- Metrics: `/actuator/metrics`\r\n- Prometheus integration: `/actuator/prometheus`\r\n\r\nFor log access in production:\r\n```bash\r\ndocker logs load-balancer\r\ndocker logs storage-container-1\r\n```\r\n\r\n## Troubleshooting\r\n\r\n### Common Issues\r\n\r\n1. **Database Connection Failures**\r\n    - In development mode: Ensure MySQL is running and accessible\r\n    - In production mode: Check logs for MySQL container startup issues\r\n\r\n   ```bash\r\n   # Check MySQL container status\r\n   docker logs mysql-db\r\n   ```\r\n\r\n2. **Storage Container Registration Failures**\r\n    - Verify network connectivity between containers\r\n    - Check if the load balancer is running before storage containers\r\n\r\n   ```bash\r\n   # Inspect container network\r\n   docker network inspect distributed_storage_network\r\n   ```\r\n\r\n3. **File Upload Errors**\r\n    - Verify storage capacity hasn't been exceeded\r\n    - Check max file size configuration matches your needs\r\n\r\n### Restarting Services\r\n\r\nIn development mode:\r\n```bash\r\n# Load Balancer\r\ncd load-balancer\r\n./start-dev.bat  # or ./start-dev.sh\r\n\r\n# File Storage\r\ncd file-storage\r\n./start-dev.bat  # or ./start-dev.sh\r\n```\r\n\r\nIn production mode:\r\n```bash\r\n# Restart all services\r\ndocker-compose restart\r\n\r\n# Restart specific service\r\ndocker-compose restart load-balancer\r\ndocker-compose restart storage-container-1\r\n```\r\n\r\n### Database Initialization\r\n\r\nIf the database schema needs to be reset:\r\n\r\nIn development mode:\r\n```sql\r\nDROP DATABASE loadbalancer;\r\nCREATE DATABASE loadbalancer;\r\n```\r\n\r\nIn production mode:\r\n```bash\r\n# Remove volumes and restart\r\ndocker-compose down -v\r\ndocker-compose up -d\r\n```\r\n\r\n## Advanced Configuration\r\n\r\n### Custom Load Balancing Strategies\r\n\r\nThe system supports multiple load balancing strategies:\r\n- `round-robin`: Distribute requests evenly in sequence\r\n- `least-connection`: Send to the node with fewest active connections\r\n- `shortest-job-next`: Optimize for task completion time\r\n- `first-come-first-serve`: Simple queue-based allocation\r\n- `weighted-round-robin`: Weighted distribution based on node capacity\r\n\r\nTo change the strategy, modify the `.env` file:\r\n```\r\nLOAD_BALANCING_STRATEGY=least-connection\r\n```\r\n\r\n### Scaling Storage Capacity\r\n\r\nTo increase individual container storage capacity, update the `.env` file:\r\n```\r\nSTORAGE_CAPACITY=5368709120  # 5GB per container\r\n```\r\n\r\nIn production, you may need to recreate the containers after changing this value:\r\n```bash\r\ndocker-compose down\r\ndocker-compose up -d\r\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenzycodex%2Fdistributed-file-storage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkenzycodex%2Fdistributed-file-storage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenzycodex%2Fdistributed-file-storage/lists"}