https://github.com/knu-k/large-file-processing-demo
The project takes a solution approach to how to handle large files.
https://github.com/knu-k/large-file-processing-demo
chunked-uploads http large-file-upload mp4-video spring
Last synced: about 2 months ago
JSON representation
The project takes a solution approach to how to handle large files.
- Host: GitHub
- URL: https://github.com/knu-k/large-file-processing-demo
- Owner: knu-k
- Created: 2024-10-10T06:06:52.000Z (8 months ago)
- Default Branch: demo-1
- Last Pushed: 2024-10-11T20:51:27.000Z (7 months ago)
- Last Synced: 2025-04-01T19:48:12.374Z (about 2 months ago)
- Topics: chunked-uploads, http, large-file-upload, mp4-video, spring
- Language: Java
- Homepage:
- Size: 320 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# large-file-processing-demo
## 1. Design
### 1-1. Flow Diagram
1. Split MP4 files into chunks.
2. Send chunk file from client to server
3. Integrate chunk files
### 1-2. Goal
- Enhance the efficiency of processing and transferring large files by effectively splitting MP4 files into chunks for seamless transmission and integration.
- Improve the user experience by allowing users to upload files without delays, ensuring a smooth and responsive interface.
- Optimize server resources to maintain stable and efficient file processing, minimizing downtime and bottlenecks.
### 1-3. Key Points
1. Efficiently handle large volumes of data using a message broker for low overhead and increased throughput.
2. Utilize asynchronous and non-blocking operations to achieve optimal speed and responsiveness.
3. Implement sequence numbers in chunk files to guarantee the correct order during integration, preventing data integrity issues.
## 2. Implementation
### 2-1. Speed comparison between chunked and regular methods (Based on 2GB file)
* **Chunked Upload(Promise parallel processing)**
>average latency : 19000 ms
* **Non Chunked Upload**
>average latency : 25000 ms
#### Conclusion
Transmitting files in chunks uses less bandwidth compared to sending the entire file at once, and it also allows for parallel processing, resulting in faster speeds.
## 3. Reference
* frontend : https://github.com/KNU-K/large-file-processing-fe-demo