https://github.com/arpan491/distributed-filesystem
An open-source, scalable, decentralized, robust, heterogeneous file storage solution which is fault tolerant, replicated, distributed and lets you upload, download, and see the catalog of other cluster with low latency and LRU cache capabilities.
https://github.com/arpan491/distributed-filesystem
distributed-file-system grpc-python heartbeat-service lru-cache multithreading poxy-server protobuf3 python3 raft-consensus-algorithm
Last synced: 10 months ago
JSON representation
An open-source, scalable, decentralized, robust, heterogeneous file storage solution which is fault tolerant, replicated, distributed and lets you upload, download, and see the catalog of other cluster with low latency and LRU cache capabilities.
- Host: GitHub
- URL: https://github.com/arpan491/distributed-filesystem
- Owner: arpan491
- Created: 2024-09-14T12:57:10.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-09-14T16:56:46.000Z (over 1 year ago)
- Last Synced: 2024-10-18T13:14:25.730Z (about 1 year ago)
- Topics: distributed-file-system, grpc-python, heartbeat-service, lru-cache, multithreading, poxy-server, protobuf3, python3, raft-consensus-algorithm
- Language: Python
- Homepage:
- Size: 1.61 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Distributed File Storage
This project presents an open-source, scalable, and robust distributed storage solution. It enables multiple servers to collaborate, forming a dynamic 'overlay' network.
Key features include:
- Platform independence
- Fault tolerance and recovery
- Adaptive task distribution
- Extensibility
- Resilience
- Concurrent request handling
- Distributed consensus (RAFT)
- Data redundancy
- Enhanced caching
- Optimized search functionality
##### Supported user services (expandable): -
1. File Upload (all formats)
2. File Download
3. File Search
4. File Listing (user-specific)
5. File Deletion
6. File Modification
The system facilitates multi-client data access and supports CRUD operations. Files are segmented into chunks, distributed across remote machines for parallel processing.
****************************************************************************************************************
### Architecture Diagram
****************************************************************************************************************
### Technology Stack
****************************************************************************************************************
### Setup Instructions
- Install dependencies from requirements.txt
- Configure server IPs and parameters in config.yaml
- Update iptable.txt with cluster IPs
#### Launch Supernode
```
python3 supernode.py
```
#### Launch Cluster Nodes
```
python3 server.py four
```
```
python3 server.py five
```
```
python3 server.py six
```
#### Launch Client
```
python3 client.py
```