https://github.com/seqsense/s3sync
♻️ Golang utility for syncing between s3 and local, similar to `aws s3 sync`
https://github.com/seqsense/s3sync
aws-sdk-go golang s3
Last synced: 5 months ago
JSON representation
♻️ Golang utility for syncing between s3 and local, similar to `aws s3 sync`
- Host: GitHub
- URL: https://github.com/seqsense/s3sync
- Owner: seqsense
- License: apache-2.0
- Created: 2019-06-28T08:49:34.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2026-01-10T02:31:24.000Z (6 months ago)
- Last Synced: 2026-01-11T00:32:42.339Z (5 months ago)
- Topics: aws-sdk-go, golang, s3
- Language: Go
- Homepage: https://pkg.go.dev/github.com/seqsense/s3sync
- Size: 314 KB
- Stars: 80
- Watchers: 7
- Forks: 27
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# s3sync
[](https://codecov.io/gh/seqsense/s3sync)
> Golang utility for syncing between s3 and local
## Migration guide
- [v2](MIGRATION.md#v2)
# Usage
Use `New` to create a manager, and `Sync` function syncs between s3 and local filesystem.
```go
import (
"github.com/aws/aws-sdk-go-v2/config"
"github.com/seqsense/s3sync/v2"
)
func main() {
ctx := context.TODO()
// Load AWS config
cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("ap-northeast-1"))
if err != nil {
panic(err)
}
syncManager := s3sync.New(cfg)
// Sync from s3 to local
syncManager.Sync(ctx, "s3://yourbucket/path/to/dir", "local/path/to/dir")
// Sync from local to s3
syncManager.Sync(ctx, "local/path/to/dir", "s3://yourbucket/path/to/dir")
// Sync from s3 to s3
syncManager.Sync(ctx, "s3://yourbucket/path/to/dir", "s3://anotherbucket/path/to/dir")
}
```
## Sets the custom logger
You can set your custom logger.
```go
import "github.com/seqsense/s3sync/v2"
...
s3sync.SetLogger(&CustomLogger{})
...
```
The logger needs to implement `Logf` methods. See the godoc for details.
## Sets up the parallelism
You can configure the number of parallel jobs for sync. Default is 16.
```
s3sync.New(cfg, s3sync.WithParallel(16)) // This is the same as default.
s3sync.New(cfg, s3sync.WithParallel(1)) // You can sync one by one.
```
# License
Apache 2.0 License. See [LICENSE](https://github.com/seqsense/s3sync/blob/master/LICENSE).