Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/arvryna/blazer
concurrent file downloader
https://github.com/arvryna/blazer
concurrency downloader fast go golang wget-alternative
Last synced: about 1 month ago
JSON representation
concurrent file downloader
- Host: GitHub
- URL: https://github.com/arvryna/blazer
- Owner: arvryna
- License: apache-2.0
- Created: 2021-07-27T13:26:01.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-03-31T18:39:43.000Z (over 2 years ago)
- Last Synced: 2024-11-01T19:35:53.636Z (about 1 month ago)
- Topics: concurrency, downloader, fast, go, golang, wget-alternative
- Language: Go
- Homepage:
- Size: 325 KB
- Stars: 16
- Watchers: 2
- Forks: 3
- Open Issues: 23
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- awesome-blazingly-fast - blazer - blazing fast file downloader (Go)
README
# Blazer - Concurrent file downloader
A CLI tool to download files from anywhere concurrently using the power of go-routines written using stdlib without any dependencies.
## Features
- Control thread count
- Resume from interruption
- File integrity check - SHA256
## Install
3 different ways:
- ``` go get -u github.com/arvryna/blazer ```
- Download specific version from releases: https://github.com/arvryna/blazer/releases
- Build from source:
```
git clone [email protected]:arvryna/blazer.git
make package
```
## Usage
``` blazer -url=example.com/1.mp4 -t=10 ```
## Flags
```
blazer -h
Usage of blazer:
-checksum string
checksum SHA256(currently supported) to verify file
-out string
output path to store the downloaded file
-t int
Thread count - Number of concurrent downloads (default 10)
-url string
Valid URL to download
-v prints current version of blazer
```
## Benchmarks
| Name |Size | Blazer | cURL | Wget |
| -----------|--------| ----------- | ---- | ----- |
| Debian ISO | 300 MB | 1min 10 sec (25 threads)| 2min 40 sec | 3 min 10 sec |
| Windows-10 | 5.4 GB | 20min 52sec (25 threads)| 46min 52 sec | 40 min 25 sec|
## Demo
[](https://asciinema.org/a/DInboSaUY2Ik9JIOcY4vZHRY9)
# Usage examples:
- Integration with node.js server [here](https://github.com/teyalite/client-sever-multithreading-downloader)
# FAQ
## How blazer works ?
Blazer makes use of [RFC7233](https://datatracker.ietf.org/doc/html/rfc7233) to identify the size of the resource to download, and then initiates partial downloads concurrently with the help of multiple go-routines, once all segements are downloaded, we finally merge all the individual segments into the final file. Number of segments = thread count passed as param to blazer CLI
## How file resumption work ?
* If download is either interrupted manually or because of any network errors (Timeout RCP(connection reset by peer) - may happen if "-t" is a large number than server can handle) then those segments may fail and you may have to restart the download with same URL and thread count for retry to work because, temp folder name is a hash of URL and thread count.
* When download is re-initiated, blazer downloads only the segments that were not successful during the previous attempt. Because a temproary download cache is stored in the current directory of download, which will be used to restart from where it was left off.
* File resumption won't work if the number of threads used is different from the previous attempt.
# Roadmap:
* Possiblity of adding a config file for ~/.blazerc in home directory, to add new features like storing history of downloads
* Ability to perform concurrent file uploads
* Figuring out optimal thread count at runtime by considering various factors like network speed, server bandwidth, etc.,
* Showing an interactive progress bar with higher accuracy
* Add more algorithms for file integrity check