https://github.com/ckampfe/s3dl
Download files from S3 in parallel
https://github.com/ckampfe/s3dl
Last synced: 8 months ago
JSON representation
Download files from S3 in parallel
- Host: GitHub
- URL: https://github.com/ckampfe/s3dl
- Owner: ckampfe
- License: bsd-3-clause
- Created: 2021-04-12T05:35:14.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2021-12-12T03:02:57.000Z (over 4 years ago)
- Last Synced: 2025-07-28T10:50:42.144Z (11 months ago)
- Language: Rust
- Size: 165 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# s3dl
Download files from S3 in parallel
## install
```
$ cargo install --git https://github.com/ckampfe/s3dl --branch main
```
## example
```
$ cat my_s3_files.txt
a.stl
b.stl
c.png
d.png
e.rs
$ mkdir out
$ s3dl --bucket mybucket --keys-file my_s3_files.txt -o out
```
## use/options
```
s3dl 0.1.0
Download files from S3 in parallel
USAGE:
s3dl [FLAGS] [OPTIONS] --bucket --keys-file --out-path
FLAGS:
-h, --help Prints help information
-d, --ordered Force keys to download in the order in which they appear in `keys_file`. By default, keys are
downloaded in a nondeterministic order
-V, --version Prints version information
OPTIONS:
-b, --bucket The target S3 bucket
-e, --event-format
The logging format [default: full] [possible values: full, compact, pretty, json]
-f, --keys-file
A path to a newline-separated file of AWS S3 keys to download. The keys should be relative, like
`a/path/to/a/file.jpg`
-x, --on-existing-file
What to do when attempting to download a file that already exists locally [default: skip] [possible values:
skip, overwrite, error]
-o, --out-path Where the downloaded files should be written
-p, --parallelism
The maximum number of inflight requests. Defaults to (number of cpus * 10)
```
## todo
- [ ] quiet mode?