https://github.com/meowgorithm/pipedream
Easy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible services
https://github.com/meowgorithm/pipedream
cli digitalocean-spaces go multipart-uploads s3
Last synced: 10 months ago
JSON representation
Easy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible services
- Host: GitHub
- URL: https://github.com/meowgorithm/pipedream
- Owner: meowgorithm
- License: mit
- Created: 2020-08-10T00:27:08.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-08-15T20:17:01.000Z (over 3 years ago)
- Last Synced: 2025-02-28T07:51:44.220Z (10 months ago)
- Topics: cli, digitalocean-spaces, go, multipart-uploads, s3
- Language: Go
- Homepage:
- Size: 53.7 KB
- Stars: 19
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Pipe Dream
==========
Easy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible
services. Available as a CLI and Go library.
## CLI
### Usage
```bash
# Set your secrets, region and endpoint in the environment
export ACCESS_KEY="..."
export SECRET_KEY="..."
export ENDPOINT="sfo2.digitaloceanspaces.com" # for AWS set REGION
# Pipe in data or redirect in a file
pipedream --bucket images --path pets/puppy.jpg < puppy.jpg
# Get fancy
export now=$(date +"%Y-%m-%d_%H:%M:%S_%Z")
cat /data/dump.rdb | gzip | pipedream --bucket backups --path dump-$now.rdb.gz
# For more info
pipedream -h
```
### Installation
Download a build from the [releases][releases] page. macOS, Linux and Windows
builds are available for a variety of architectures.
macOS users can also use Homebrew:
```
brew tap meowgorithm/tap && brew install meowgorithm/tap/pipedream
```
Or you can just use `go get`:
```bash
go get github.com/meowgorithm/pipedream/pipedream
```
[releases]: https://github.com/meowgorithm/pipedream/releases
## Library
The library uses an event based model, sending events through a channel.
```go
import "github.com/meowgorithm/pipedream"
// Create a new multipart upload object
m := pipedream.MultipartUpload{
AccessKey: os.Getenv("ACCESS_KEY"),
SecretKey: os.Getenv("SECRET_KEY"),
Endpoint: "sfo2.digitaloceanspaces.com", // you could use Region for AWS
Bucket: "my-fave-bucket",
}
// Get an io.Reader, like an *os.File or os.Stdout
f, err := os.Open("big-redis-dump.rdb")
if err != nil {
fmt.Printf("Rats: %v\n", err)
os.Exit(1)
}
defer f.Close()
// Send up the data. Pipdream returns a channel where you can listen for events
ch := m.Send(f, "backups/dump.rdb")
done := make(chan struct{})
// Listen for activity. For more detailed reporting, see the docs
go func() {
for {
e := <-ch
switch e.(type) {
case pipedream.Complete:
fmt.Println("It worked!")
close(done)
return
case pipedream.Error:
fmt.Println("Rats, it didn't work.")
close(done)
return
}
}
}()
<-done
```
[Full source][example] of this example. For an example with more detailed
reporting, see the source code in the [CLI][cli].
[example]: https://github.com/meowgorithm/pipedream/blob/master/example/main.go
[cli]: https://github.com/meowgorithm/pipedream/tree/master/pipedream
## Awknowledgements
Thanks to to Apoorva Manjunath‘s [S3 multipart upload example](https://github.com/apoorvam/aws-s3-multipart-upload)
for the S3 implementation details.
## License
[MIT](https://github.com/meowgorithm/pipedream/raw/master/LICENSE)