{"id":13430704,"url":"https://github.com/meowgorithm/pipedream","last_synced_at":"2025-03-18T06:31:08.185Z","repository":{"id":54418257,"uuid":"286340225","full_name":"meowgorithm/pipedream","owner":"meowgorithm","description":"Easy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible services","archived":false,"fork":false,"pushed_at":"2022-08-15T20:17:01.000Z","size":55,"stargazers_count":19,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T07:51:44.220Z","etag":null,"topics":["cli","digitalocean-spaces","go","multipart-uploads","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/meowgorithm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-10T00:27:08.000Z","updated_at":"2024-07-19T17:52:50.000Z","dependencies_parsed_at":"2022-08-13T15:01:01.634Z","dependency_job_id":null,"html_url":"https://github.com/meowgorithm/pipedream","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meowgorithm%2Fpipedream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meowgorithm%2Fpipedream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meowgorithm%2Fpipedream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meowgorithm%2Fpipedream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/meowgorithm","download_url":"https://codeload.github.com/meowgorithm/pipedream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243910392,"owners_count":20367537,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","digitalocean-spaces","go","multipart-uploads","s3"],"created_at":"2024-07-31T02:00:56.937Z","updated_at":"2025-03-18T06:31:07.725Z","avatar_url":"https://github.com/meowgorithm.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"Pipe Dream\n==========\n\n\u003cp\u003e\n    \u003ca href=\"https://github.com/meowgorithm/pipedream/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/release/meowgorithm/pipedream.svg\" alt=\"Latest Release\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pkg.go.dev/github.com/meowgorithm/pipedream?tab=doc\"\u003e\u003cimg src=\"https://godoc.org/github.com/golang/gddo?status.svg\" alt=\"GoDoc\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/meowgorithm/pipedream/actions\"\u003e\u003cimg src=\"https://github.com/meowgorithm/pipedream/workflows/build/badge.svg\" alt=\"Build Status\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nEasy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible\nservices. Available as a CLI and Go library.\n\n## CLI\n\n### Usage\n\n```bash\n# Set your secrets, region and endpoint in the environment\nexport ACCESS_KEY=\"...\"\nexport SECRET_KEY=\"...\"\nexport ENDPOINT=\"sfo2.digitaloceanspaces.com\" # for AWS set REGION\n\n# Pipe in data or redirect in a file\npipedream --bucket images --path pets/puppy.jpg \u003c puppy.jpg\n\n# Get fancy\nexport now=$(date +\"%Y-%m-%d_%H:%M:%S_%Z\")\ncat /data/dump.rdb | gzip | pipedream --bucket backups --path dump-$now.rdb.gz\n\n# For more info\npipedream -h\n```\n\n### Installation\n\nDownload a build from the [releases][releases] page. macOS, Linux and Windows\nbuilds are available for a variety of architectures.\n\nmacOS users can also use Homebrew:\n\n```\nbrew tap meowgorithm/tap \u0026\u0026 brew install meowgorithm/tap/pipedream\n```\n\nOr you can just use `go get`:\n\n```bash\ngo get github.com/meowgorithm/pipedream/pipedream\n```\n\n[releases]: https://github.com/meowgorithm/pipedream/releases\n\n## Library\n\nThe library uses an event based model, sending events through a channel.\n\n```go\nimport \"github.com/meowgorithm/pipedream\"\n\n// Create a new multipart upload object\nm := pipedream.MultipartUpload{\n    AccessKey: os.Getenv(\"ACCESS_KEY\"),\n    SecretKey: os.Getenv(\"SECRET_KEY\"),\n    Endpoint:  \"sfo2.digitaloceanspaces.com\", // you could use Region for AWS\n    Bucket:    \"my-fave-bucket\",\n}\n\n// Get an io.Reader, like an *os.File or os.Stdout\nf, err := os.Open(\"big-redis-dump.rdb\")\nif err != nil {\n    fmt.Printf(\"Rats: %v\\n\", err)\n    os.Exit(1)\n}\ndefer f.Close()\n\n// Send up the data. Pipdream returns a channel where you can listen for events\nch := m.Send(f, \"backups/dump.rdb\")\ndone := make(chan struct{})\n\n// Listen for activity. For more detailed reporting, see the docs\ngo func() {\n    for {\n        e := \u003c-ch\n        switch e.(type) {\n        case pipedream.Complete:\n            fmt.Println(\"It worked!\")\n            close(done)\n            return\n        case pipedream.Error:\n            fmt.Println(\"Rats, it didn't work.\")\n            close(done)\n            return\n        }\n    }\n}()\n\n\u003c-done\n```\n\n[Full source][example] of this example. For an example with more detailed\nreporting, see the source code in the [CLI][cli].\n\n[example]: https://github.com/meowgorithm/pipedream/blob/master/example/main.go\n[cli]: https://github.com/meowgorithm/pipedream/tree/master/pipedream\n\n## Awknowledgements\n\nThanks to to Apoorva Manjunath‘s [S3 multipart upload example](https://github.com/apoorvam/aws-s3-multipart-upload)\nfor the S3 implementation details.\n\n## License\n\n[MIT](https://github.com/meowgorithm/pipedream/raw/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeowgorithm%2Fpipedream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeowgorithm%2Fpipedream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeowgorithm%2Fpipedream/lists"}