https://github.com/kenzycodex/distributed-file-storage
This project focuses on developing a Server Load Balancer Distributed File Storage, Sharing, and Management System to efficiently handle file operations across multiple storage containers. The system ensures optimized file management, secure access control, and dynamic workload distribution using advanced load-balancing techniques.
https://github.com/kenzycodex/distributed-file-storage
docker java javafx mysql server spring sql
Last synced: 3 months ago
JSON representation
This project focuses on developing a Server Load Balancer Distributed File Storage, Sharing, and Management System to efficiently handle file operations across multiple storage containers. The system ensures optimized file management, secure access control, and dynamic workload distribution using advanced load-balancing techniques.
- Host: GitHub
- URL: https://github.com/kenzycodex/distributed-file-storage
- Owner: kenzycodex
- Created: 2025-02-03T11:12:58.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-22T13:22:29.000Z (about 1 year ago)
- Last Synced: 2025-02-22T14:28:31.052Z (about 1 year ago)
- Topics: docker, java, javafx, mysql, server, spring, sql
- Language: Java
- Homepage:
- Size: 696 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Distributed Storage System
A scalable distributed file storage system with load balancing capabilities. This system consists of multiple microservices designed to work together:
- **Load Balancer**: Manages requests and distributes them across storage nodes
- **File Storage Microservices**: Handle file upload, storage, and retrieval
- **MySQL Database**: Stores metadata and system information
## Table of Contents
1. [System Architecture](#system-architecture)
2. [Prerequisites](#prerequisites)
3. [Setup Instructions](#setup-instructions)
- [Development Environment](#development-environment)
- [Production Environment](#production-environment)
4. [Environment Configuration](#environment-configuration)
5. [API Endpoints](#api-endpoints)
6. [Monitoring and Management](#monitoring-and-management)
7. [Troubleshooting](#troubleshooting)
## System Architecture
The system follows a microservices architecture:
- **Load Balancer**: Routes client requests to the appropriate storage nodes using various strategies (round-robin, least-connection, etc.)
- **Storage Containers**: Manage file operations and store actual file data
- **MySQL Database**: Centralized metadata storage for the entire system
- **Health Checks**: Regular monitoring of node status and availability
## Prerequisites
- Java 17 or higher
- Maven 3.6 or higher
- MySQL 8.0 (development mode) or Docker (production mode)
- Docker and Docker Compose (for production deployment)
- Minimum 8GB RAM recommended
- At least 10GB free disk space
## Setup Instructions
### Development Environment
#### 1. Database Setup
In development mode, you need to create a MySQL database on your local MySQL server:
```sql
CREATE DATABASE loadbalancer;
CREATE USER 'loadbalancer'@'localhost' IDENTIFIED BY 'loadbalancer';
GRANT ALL PRIVILEGES ON loadbalancer.* TO 'loadbalancer'@'localhost';
FLUSH PRIVILEGES;
```
#### 2. Clone the Repository
```bash
git clone
cd distributed-storage-system
```
#### 3. Start the Load Balancer
```bash
cd load-balancer
./start-dev.bat # Windows
# OR
./start-dev.sh # Linux/Mac
```
This script will:
- Build the application using Maven
- Start the application with the 'dev' profile
- The load balancer will be available at http://localhost:8080
#### 4. Start the File Storage Service
Open a new terminal and run:
```bash
cd file-storage
./start-dev.bat # Windows
# OR
./start-dev.sh # Linux/Mac
```
This will start a file storage service instance at http://localhost:8081/api/v1
You can start multiple storage instances by manually setting different ports, e.g.:
```bash
mvnw spring-boot:run -Dspring-boot.run.profiles=dev -Dserver.port=8082
```
### Production Environment
In production mode, all services run as Docker containers. The system uses Docker Compose to manage all components.
#### 1. Configure Environment Variables
Review and modify the `.env` file in the project root as needed:
```
# Example of .env configuration
MYSQL_ROOT_PASSWORD=secure_password
MYSQL_USER=loadbalancer
MYSQL_PASSWORD=loadbalancer
STORAGE_CAPACITY=2147483648 # 2GB per container
```
#### 2. Start the System
From the project root directory:
```bash
docker-compose up -d
```
This will:
- Build all necessary Docker images
- Start the MySQL database container
- Create 4 file storage service containers
- Start the load balancer container
- Configure all networking and volumes
The load balancer will be accessible at http://localhost:8080 and will distribute requests to the storage containers.
#### 3. Scale the System (Optional)
To adjust the number of storage containers:
```bash
docker-compose up -d --scale storage-container=6
```
## Environment Configuration
### Key Configuration Files
- `.env`: Main environment variables for Docker deployment
- `application.yml`: Common application settings for all environments
- `application-dev.yml`: Development-specific settings
- `application-prod.yml`: Production-specific settings
### Important Configuration Parameters
#### Load Balancer
| Parameter | Description | Default |
|-----------|-------------|---------|
| `LOAD_BALANCER_STRATEGY` | Strategy for distributing requests | round-robin |
| `HEALTH_CHECK_INTERVAL` | Time between node health checks (ms) | 30000 |
| `JAVA_OPTS` | JVM configuration for load balancer | -Xmx1g -Xms512m |
#### Storage Containers
| Parameter | Description | Default |
|-----------|-------------|---------|
| `STORAGE_CAPACITY` | Storage space per container (bytes) | 1073741824 (1GB) |
| `MAX_FILE_UPLOAD_SIZE` | Maximum file size for uploads | 100MB |
| `CONTAINER_ID` | Unique ID for each container | Automatically assigned |
## API Endpoints
### Load Balancer Endpoints
- `GET /api/v1/health`: System health check
- `POST /api/v1/nodes/register`: Register a new storage node
- `GET /api/v1/metrics`: System performance metrics
### File Storage Endpoints
- `POST /api/v1/files`: Upload a file
- `GET /api/v1/files/{id}`: Download a file
- `DELETE /api/v1/files/{id}`: Delete a file
- `GET /api/v1/health`: Container health status
## Monitoring and Management
The system includes built-in monitoring through Spring Boot Actuator:
- Health endpoints: `/actuator/health`
- Metrics: `/actuator/metrics`
- Prometheus integration: `/actuator/prometheus`
For log access in production:
```bash
docker logs load-balancer
docker logs storage-container-1
```
## Troubleshooting
### Common Issues
1. **Database Connection Failures**
- In development mode: Ensure MySQL is running and accessible
- In production mode: Check logs for MySQL container startup issues
```bash
# Check MySQL container status
docker logs mysql-db
```
2. **Storage Container Registration Failures**
- Verify network connectivity between containers
- Check if the load balancer is running before storage containers
```bash
# Inspect container network
docker network inspect distributed_storage_network
```
3. **File Upload Errors**
- Verify storage capacity hasn't been exceeded
- Check max file size configuration matches your needs
### Restarting Services
In development mode:
```bash
# Load Balancer
cd load-balancer
./start-dev.bat # or ./start-dev.sh
# File Storage
cd file-storage
./start-dev.bat # or ./start-dev.sh
```
In production mode:
```bash
# Restart all services
docker-compose restart
# Restart specific service
docker-compose restart load-balancer
docker-compose restart storage-container-1
```
### Database Initialization
If the database schema needs to be reset:
In development mode:
```sql
DROP DATABASE loadbalancer;
CREATE DATABASE loadbalancer;
```
In production mode:
```bash
# Remove volumes and restart
docker-compose down -v
docker-compose up -d
```
## Advanced Configuration
### Custom Load Balancing Strategies
The system supports multiple load balancing strategies:
- `round-robin`: Distribute requests evenly in sequence
- `least-connection`: Send to the node with fewest active connections
- `shortest-job-next`: Optimize for task completion time
- `first-come-first-serve`: Simple queue-based allocation
- `weighted-round-robin`: Weighted distribution based on node capacity
To change the strategy, modify the `.env` file:
```
LOAD_BALANCING_STRATEGY=least-connection
```
### Scaling Storage Capacity
To increase individual container storage capacity, update the `.env` file:
```
STORAGE_CAPACITY=5368709120 # 5GB per container
```
In production, you may need to recreate the containers after changing this value:
```bash
docker-compose down
docker-compose up -d
```