https://github.com/maknahar/s3copy
An AWS lambda function to duplicate objects from n to m AWS S3 Buckets where n, m >= 1
https://github.com/maknahar/s3copy
amazon-web-services aws-lambda aws-s3 copy duplicate-files duplication go golang lambda
Last synced: 11 days ago
JSON representation
An AWS lambda function to duplicate objects from n to m AWS S3 Buckets where n, m >= 1
- Host: GitHub
- URL: https://github.com/maknahar/s3copy
- Owner: maknahar
- License: mit
- Created: 2018-03-19T19:20:42.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-04-10T17:17:03.000Z (about 7 years ago)
- Last Synced: 2024-08-03T23:28:58.340Z (10 months ago)
- Topics: amazon-web-services, aws-lambda, aws-s3, copy, duplicate-files, duplication, go, golang, lambda
- Language: Go
- Homepage:
- Size: 8.16 MB
- Stars: 1
- Watchers: 1
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-golang-repositories - s3copy
README
# s3copy
A lambda function to copy objects from n to m S3 Buckets
where n, m >= 1
## Deployment
- Download the deployment file from
https://github.com/maknahar/s3copy/releases
- Upload the deployment file to lambda
- Set the configuration
or
- Checkout the latest release code
`go get github.com/maknahar/copys3`
- Build and generate the deployment file
`GOOS=linux go build -o main main.go && zip deployment.zip main`
- Upload the deployment file to lambda
- Set the configuration
**Handler function name:** `main`
## Config
This Lambda Function expect the list of source and
destination buckets in a JSON file.
### Config File Format
Config File is a map of input s3 bucket and detail to copy
changes to destination.
The function can take events from an SQS as well if
source bucket put all event to an SQS.
The function can take events from an SQS where SQS is
subscribed to an SNS and source bucket publish a message
to an SNS on every change.
To Support SQS processing add SQS URL `sqs` and
SQS region `sqsRegion` in the config file. Otherwise,
Leave them empty `""` or remove the keys.
```
{
"source-bucket": {
"region": "us-east-1",
"sqs": "URL of SQS where all S3 change events are stored",
"sqsRegion": "us-east-1",
"destinations": [
"destination-bucket-1",
"destination-bucket-2"
]
},
"another-source-bucket": {
"region": "us-east-1",
"sqs": "SQS URL",
"sqsRegion": "us-east-1",
"destinations": [
"destination-bucket-1",
"destination-bucket-3"
]
}
}
```
With above configuration, any object put in `source-bucket`
will be copied to `destination-bucket-1` & `destination-bucket-2`
and any object put in `another-source-bucket` will be
copied to `destination-bucket-1` & `destination-bucket-3`.
### Env Var
This Lambda function can be configured in two ways.
- Either give a public url of config file via CONFIG_FILE
- or provide base64 encoded value of configuration via
CONFIG. You can online tools like www.base64encode.org
to encode config content.
If both are provided CONFIG_FILE value will take precedence.
- CONFIG_FILE : URL of configuration file
- CONFIG : Base64 Encoded string of content of CONFIG_FILE
## NOTE:
Make sure Lambda function have required access to source
and destination bucket.
## Contribution
All contributions are welcome. Either via PR or Issue.