https://github.com/yandex-cloud-examples/yc-s3-data-sync-to-aws
Синхронизация данных между S3 хранилищами в Yandex Cloud и AWS.
https://github.com/yandex-cloud-examples/yc-s3-data-sync-to-aws
aws aws-lambda data-sync object-storage s3 s3-storage serverless terraform yandex-cloud yandex-function yandexcloud
Last synced: 4 months ago
JSON representation
Синхронизация данных между S3 хранилищами в Yandex Cloud и AWS.
- Host: GitHub
- URL: https://github.com/yandex-cloud-examples/yc-s3-data-sync-to-aws
- Owner: yandex-cloud-examples
- License: apache-2.0
- Created: 2024-03-08T08:53:15.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-06-06T11:44:22.000Z (12 months ago)
- Last Synced: 2025-06-06T12:38:29.965Z (12 months ago)
- Topics: aws, aws-lambda, data-sync, object-storage, s3, s3-storage, serverless, terraform, yandex-cloud, yandex-function, yandexcloud
- Language: Python
- Homepage:
- Size: 65.4 KB
- Stars: 1
- Watchers: 10
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Description
This scenario explains how to leverage [AWS Lambda](https://aws.amazon.com/lambda/) functions and [Yandex Cloud Functions](https://cloud.yandex.com/en-ru/docs/functions/) to sync newly created objects on [AWS S3](https://aws.amazon.com/s3/) and [Yandex Object Storage](https://cloud.yandex.com/en-ru/docs/storage/).
The example uses the same function code for both sides, as Yandex Cloud Functions and AWS Lambda runtimes are compatible.
If you look at the code, you’ll notice that it uses the same S3 API for both sides, as the AWS S3 and Yandex Storage APIs are compatible.
# Limitations
Please note that this example syncs only newly created objects. If you want some solution that helps with more scenarios out of the box please refer to Multi-cloud storage with Yandex Cloud and Amazon S3 guide.
# Syncing two S3 buckets
Review the following example to see how syncing works.
You will need:
- Accounts in AWS and Yandex Cloud
- Bash
- Terraform 1.1.5
- [s3cmd](https://s3tools.org/download)
- jq
Configure the AWS site:
- Configure the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
Configure Yandex Cloud:
- Configure the [YC CLI](https://cloud.yandex.com/docs/cli/quickstart)
- Prepare the Terraform environment:
```
export YC_TOKEN=$(yc iam create-token)
export YC_CLOUD_ID=$(yc config get cloud-id)
export YC_FOLDER_ID=$(yc config get folder-id)
export TF_VAR_folder_id=$YC_FOLDER_ID
```
## Quick start
### Initiate an example playbook
```
cd example
terraform init
terraform apply
```
### Check results
We will use:
- `aws` tool for AWS site
- `s3cmd` tool for Yandex Cloud site
Prepare an environment:
```
BUCKET_NAME=$(terraform output -raw bucket_name)
mv s3cfg ~/.s3cfg
```
Make sure that AWS and Yandex Cloud buckets are empty:
```
aws s3 ls $BUCKET_NAME
s3cmd ls s3://$BUCKET_NAME
```
Put the file into the Yandex Cloud bucket:
```
s3cmd put sync.zip s3://$BUCKET_NAME
upload: 'sync.zip' -> 's3://yc-s3-sync-a6v4g3vlra/sync.zip' [1 of 1]
2551 of 2551 100% in 0s 24.73 KB/s done
```
Check that the file is in place at Yandex Cloud:
```
s3cmd ls s3://$BUCKET_NAME
2020-12-08 18:41 2551 s3://yc-s3-sync-a6v4g3vlra/sync.zip
```
Check that the file is successfully synced to the AWS S3:
```
aws s3 ls $BUCKET_NAME
2020-12-08 21:42:14 2551 sync.zip
```
### To destroy everything quickly
Delete all files at AWS and Yandex Cloud buckets:
```
aws s3 rm s3://$BUCKET_NAME/sync.zip
s3cmd rm s3://$BUCKET_NAME/sync.zip
rm ~/.s3cfg
```
Destroy Terraform resources:
```
terraform destroy
```