https://github.com/botpress/aws-s3-signed-uploads-example
S3 Uploads with Signed URLs
https://github.com/botpress/aws-s3-signed-uploads-example
Last synced: 11 months ago
JSON representation
S3 Uploads with Signed URLs
- Host: GitHub
- URL: https://github.com/botpress/aws-s3-signed-uploads-example
- Owner: botpress
- Created: 2022-05-11T13:09:53.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2024-05-06T13:12:43.000Z (about 2 years ago)
- Last Synced: 2025-02-15T14:47:34.493Z (over 1 year ago)
- Language: TypeScript
- Size: 103 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# S3 Uploads with Signed URLs
## What is this?
This repo contains examples of uploading files to a S3 bucket using Signed URLs
## Getting started
1. Install: `yarn`
2. Deploy the CDK app contained in `src/cdk-example`. From the `src/cdk-example` directory, run `npx cdk deploy --all --profile {your AWS profile} --require-approval never --progress events` .
- Note the stack's outputs (`AccessKeyOutput`, `BucketNameOutput`, `SecretAccessKeyOutput`), they'll be useful in the following steps
- This stack contains a IAM User that can upload files to a S3 bucket
3. Test singlepart uploads: from the repo's root, run `AWS_REGION={region where you deployed the CDK app} AWS_ACCESS_KEY_ID={value of AccessKeyOutput} AWS_SECRET_ACCESS_KEY={value of SecretAccessKeyOutput} BUCKET_NAME={value of BucketNameOutput} npx ts-node src/singlepart/index.ts path/to/a/file/to/upload.jpg`
- After the command completes, you'll find the uploaded file in the S3 bucket created in step 2
4. Test multipart uploads: from the repo's root, run `AWS_REGION={region where you deployed the CDK app} AWS_ACCESS_KEY_ID={value of AccessKeyOutput} AWS_SECRET_ACCESS_KEY={value of SecretAccessKeyOutput} BUCKET_NAME={value of BucketNameOutput} npx ts-node src/multipart/index.ts path/to/a/file/to/upload.jpg`
- After the command completes, you'll find the uploaded file in the S3 bucket created in step 2
## About uploads in S3
To upload data in a S3 bucket, 2 methods exist:
1. Single PUT object request (I call this "singlepart upload")
2. Multipart upload
### Comparing singlepart and multipart uploads
#### Singlepart uploads
- Works with files <= 5GB in size
- Simpler to implement (only 1 request to S3)
- If the request is interrupted, it has to be started all over
#### Multipart uploads
- Works with files <= 5TB in size
- More complex to implement (multiple requests to S3)
- Improved throughput – You can upload parts in parallel to improve throughput
- Quick recovery from any network issues – Smaller part size minimizes the impact of restarting a failed upload due to a network error.
- Pause and resume object uploads – You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload.
- Begin an upload before you know the final object size – You can upload an object as you are creating it.