An open API service indexing awesome lists of open source software.

https://github.com/andresweitzel/crud_amazon_s3_aws

βœ… CRUD model for object management with Amazon S3 from AWS implemented with Systems Manager Parameter Store, S3 Bucket, Api-Gateway, Serverless-Framework, Lambda, NodeJs, aws-sdk-v3, among others.
https://github.com/andresweitzel/crud_amazon_s3_aws

amazon-lambda amazon-s3 api-gateway aws aws-lambda aws-sdk-v3 nodejs parameter-store serverless serverless-framework serverless-offline

Last synced: about 1 month ago
JSON representation

βœ… CRUD model for object management with Amazon S3 from AWS implemented with Systems Manager Parameter Store, S3 Bucket, Api-Gateway, Serverless-Framework, Lambda, NodeJs, aws-sdk-v3, among others.

Awesome Lists containing this project

README

          



![Index app](./doc/assets/CRUD_Bucket_s3.drawio.png)



















# CRUD Bucket S3 AWS ![(status-completed)](./doc/assets/icons/badges/status-completed.svg)


CRUD model for object handling implemented with Systems Manager Parameter Store, S3 Bucket, Api-Gateway, Serverless-Framework, Lambda, NodeJs, aws-sdk-v3, among others others. AWS services are deployed locally. The project code and documentation (except technical docs) have been developed in English.

* [Postman Collection](./src/postman/collection/CRUD_Bucket_S3_AWS.postman_collection.json)
* [Functionality test playlist](https://www.youtube.com/playlist?list=PLCl11UFjHurDPyOkEXOR6JO-vUnYqd1FW)



## Index πŸ“œ

View details

`Latest update: 19/02/26`

### Section 1) Description, Configuration, and Technologies

* [1.0) Project Description.](#10-description-)
* [1.1) Project Execution .](#11-project-execution-)
* [1.2) Project setup from scratch](#12-project-setup-from-scratch-)
* [1.3) Technologies.](#13-technologies-)

### Section 2) Endpoints and Examples

* [2.0) Endpoints and resources.](#20-endpoints-and-resources-)

### Section 3) Functionality Testing and References

* [3.0) Functionality Testing.](#30-functionality-testing-)
* [3.1) References.](#31-references-)




## Section 1) Description, configuration and technologies

### 1.0) Description [πŸ”](#index-)

View details

### 1.0.0) General Description

* This app is divided into several functionalities/components. The first component or connection layer (/bucket) is the interaction with aws-sdk and the bucket. It is modularized in such a way that we have .js files for creating s3 clients, bucket reading, bucket writing, etc.
Then for the application layer (/helpers) we have header validations, request body, date formats, authentication, etc.
Next, the controller/view layer (/controllers) is defined by the CRUD operations possible in each of the defined lambdas.

### 1.0.1) Architecture and Operation Description

* The image of the aws architecture used describes the general flow of the app. Any request to the bucket starts from a client (Postman, server, etc.).
* `Step 0` : This request is received by the api-gateway and will only be validated if the correct x-api-key is found within the headers of said request.
* `Steps 1A, 1B, etc.` : All these steps correspond to an endpoint with its specific resource. For example, for uploadObject (1A) it is http://localhost:4000/dev/upload-object .... check these endpoints in [endpoints section](#section-2-endpoints-and-examples). Each lambda performs x-api-key and Bearer token verification, among others.
* `Steps 2` : The lambdas perform the corresponding ssm validations with the System Manager Parameter Store.. they validate tokens, values ​​for the s3 bucket, etc.
* `Steps 3` : The lambdas perform the necessary requests and operations against the s3 bucket (reading, updating, deleting and inserting objects).
* `Clarifications` : This operation is emulated within the same network and in a local environment with the corresponding serverless plugins.


### 1.1) Project Execution [πŸ”](#index-)

View details

* Once a work environment has been created through an ide, we clone the project

```git
git clone https://github.com/andresWeitzel/CRUD_Bucket_S3_AWS
```

* We position ourselves on the project

```git
cd 'projectName'
```

* We install the LTS version of [Nodejs(v18)](https://nodejs.org/en/download)
* We install the Serverless Framework globally if we haven't already. I recommend version three since it's free and doesn't ask for credentials. The latest version (four) can be used without any problems, although it requires a paid version.
```git
npm install -g serverless@3
```
* We check the installed version of Serverless

```git
sls -v
```

* We install all the necessary packages

```git
npm i
```

* The ssm variables used in the project are kept to simplify the configuration process of the project. It is recommended to add the corresponding file (serverless\_ssm.yml) to the .gitignore.
* The following script configured in the package.json of the project is in charge of
* Starting serverless-offline (serverless-offline)

```git
"scripts": {
"serverless-offline": "sls offline start",
"start": "npm run serverless-offline"
},
```

* We run the app from the terminal.

```git
npm start
```

* If we get a message indicating that port 9324 is already in use, we can terminate all dependent processes.
```git
npx kill-port 9324
```
* We launch the app from terminal.
```git
npm start
```


### 1.2) Setting up the project from scratch [πŸ”](#Γ­ndice-)

View details

* We create a work environment through an IDE, after creating a folder we position ourselves on it

```git
cd 'projectName'
```

* We install the LTS version of [Nodejs(v18)](https://nodejs.org/en/download)
* We install the Serverless Framework globally if we haven't already. I recommend version three since it's free and doesn't ask for credentials. The latest version (four) can be used without any problems, although it requires a paid version.
```git
npm install -g serverless@3
```

* Check the installed Serverless version

```git
sls -v
```

* Initialize a serverless template

```git
serverless create --template aws-nodejs
```

* Initialize an npm project

```git
npm init -y
```

* Install local S3

```git
npm install serverless-s3-local --save-dev
```

* Install the s3 Client

```git
npm install @aws-sdk/client-s3
```

* Install serverless offline

```git
npm i serverless-offline --save-dev
```

* Install serverless ssm

```git
npm i serverless-offline-ssm --save-dev
```

* The ssm variables used in the project are kept to simplify the configuration process of the project. It is recommended to add the corresponding file (serverless\_ssm.yml) to the .gitignore.
* The following script configured in the package.json of the project is in charge of
* Starting serverless-offline (serverless-offline)

```git
"scripts": {
"serverless-offline": "sls offline start",
"start": "npm run serverless-offline"
},
```

* We run the app from the terminal.

```git
npm start
```

* If we get a message indicating that port 9324 is already in use, we can terminate all dependent processes.
```git
npx kill-port 9324
```
* We launch the app from terminal.
```git
npm start
```


### 1.3) Technologies [πŸ”](#index-)

View details

| Technology | Version | Purpose |
|------------|---------|---------|
| [SDK](https://www.serverless.com/framework/docs/guides/sdk/) | 4.3.2 | Automatic Module Injection for Lambdas |
| [Serverless Framework Core v3](https://www.serverless.com//blog/serverless-framework-v3-is-live) | 3.23.0 | AWS Core Services |
| [Systems Manager Parameter Store (SSM)](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) | 3.0 | Management of Environment Variables |
| [Amazon Api Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) | 2.0 | API Manager, Authentication, Control and Processing |
| [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) | 3.0 | Object Container |
| [NodeJS](https://nodejs.org/en/) | 14.18.1 | JS Library |
| [VSC](https://code.visualstudio.com/docs) | 1.72.2 | IDE |
| [Postman](https://www.postman.com/downloads/) | 10.11 | Http Client |
| [CMD](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cmd) | 10 | Command Prompt |
| [Git](https://git-scm.com/downloads) | 2.29.1 | Version Control |

### Plugins

| Plugin | Description |
|--------|-------------|
| [Serverless Plugin](https://www.serverless.com/plugins/) | Libraries for Modular Definition |
| [serverless-offline](https://www.npmjs.com/package/serverless-offline) | This serverless plugin emulates AWS Ξ» and API Gateway on-premises |
| [serverless-offline-ssm](https://www.npmjs.com/package/serverless-offline-ssm) | Finds environment variables that match SSM parameters at build time and replaces them from a file |
| [serverless-s3-local](https://www.serverless.com/plugins/serverless-s3-local) | Serverless plugin to run S3 clones locally |

### Extensions

| Extension |
|-----------|
| Prettier - Code formatter |
| YAML - Autoformatter .yml (alt+shift+f) |



## Section 2) Endpoints and Examples.

### 2.0) Endpoints and Resources [πŸ”](#index-)

View details

### 2.1.0) Variables in Postman

| Variable | Initial Value | Current Value |
|----------|---------------|---------------|
| base_url | http://localhost:4000 | http://localhost:4000 |
| x-api-key | f98d8cd98h73s204e3456998ecl9427j | f98d8cd98h73s204e3456998ecl9427j |
| bearer_token | Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c | Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c |



### 2.1.1) Upload an object to the s3 bucket

#### Request | Code snippet

```postman
curl --location 'http://localhost:4000/dev/upload-object' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data '{
"type":"image",
"format":"png",
"description":"5000 Γ— 3061 png",
"url":"https://www.bing.com/images/search?view=detailV2&ccid=Tf4BFI68&id=D66EF5BFB7DA0A645A70240C32CB8664E8F8BF09&thid=OIP.Tf4BFI6846neirVSebC0vAHaEi&mediaurl=https%3a%2f%2flogos-download.com%2fwp-content%2fuploads%2f2016%2f09%2fNode_logo_NodeJS.png&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.4dfe01148ebce3a9de8ab55279b0b4bc%3frik%3dCb%252f46GSGyzIMJA%26pid%3dImgRaw%26r%3d0&exph=3061&expw=5000&q=jpg+nodejs&simid=608055434302923247&FORM=IRPRST&ck=2FF3D39CAEF945F20B996CF6042F88A6&selectedIndex=1&ajaxhist=0&ajaxserp=0"
}'
```

#### Response

```postman
{
"message": {
"type": "image",
"format": "png",
"description": "5000 Γ— 3061 png",
"url": "https://www.bing.com/images/search?view=detailV2&ccid=Tf4BFI68&id=D66EF5BFB7DA0A645A70240C32CB8664E8F8BF09&thid=OIP.Tf4BFI6846neirVSebC0vAHaEi&mediaurl=https%3a%2f%2flogos-download.com%2fwp-content%2fuploads%2f2016%2f09%2fNode_logo_NodeJS.png&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.4dfe01148ebce3a9de8ab55279b0b4bc%3frik%3dCb%252f46GSGyzIMJA%26pid%3dImgRaw%26r%3d0&exph=3061&expw=5000&q=jpg+nodejs&simid=608055434302923247&FORM=IRPRST&ck=2FF3D39CAEF945F20B996CF6042F88A6&selectedIndex=1&ajaxhist=0&ajaxserp=0",
"uuid": 104851112
}
}
```



### 2.1.2) Get an object from the bucket based on its uuid

#### Request | Code snippet

```postman
curl --location 'http://localhost:4000/dev/get-object/103053674' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data ''
```

#### Response

```postman
{
"message": {
"type": "image",
"format": "jpg",
"description": "1000 Γ— 1261 png",
"url": "https://www.bing.com/images/search?view=detailV2&ccid=Tf4BFI68&id=D66EF5BFB7DA0A645A70240C32CB8664E8F8BF09&thid=OIP.Tf4BFI6846neirVSebC0vAHaEi&mediaurl=https%3a%2f%2flogos-download.com%2fwp-content%2fuploads%2f2016%2f09%2fNode_logo_NodeJS.png&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.4dfe01148ebce3a9de8ab55279b0b4bc%3frik%3dCb%252f46GSGyzIMJA%26pid%3dImgRaw%26r%3d0&exph=3061&expw=5000&q=jpg+nodejs&simid=608055434302923247&FORM=IRPRST&ck=2FF3D39CAEF945F20B996CF6042F88A6&selectedIndex=1&ajaxhist=0&ajaxserp=0",
"uuid": 103053674
}
}
```



### 2.1.3) Update an object in the s3 bucket

#### Request | Code snippet

```postman
curl --location --request PUT 'http://localhost:4000/dev/edit-object/104851112' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'Content-Type: application/json' \
--data '{
"type":"image",
"format":"jpg",
"description":"1200 Γ— 1201 png",
"url":"https://www.bing.com/images/search?view=detailV2&ccid=Tf4BFI68&id=D66EF5BFB7DA0A645A70240C32CB8664E8F8BF09&thid=OIP.Tf4BFI6846neirVSebC0vAHaEi&mediaurl=https%3a%2f%2flogos-download.com%2fwp-content%2fuploads%2f2016%2f09%2fNode_logo_NodeJS.png&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.4dfe01148ebce3a9de8ab55279b0b4bc%3frik%3dCb%252f46GSGyzIMJA%26pid%3dImgRaw%26r%3d0&exph=3061&expw=5000&q=jpg+nodejs&simid=608055434302923247&FORM=IRPRST&ck=2FF3D39CAEF945F20B996CF6042F88A6&selectedIndex=1&ajaxhist=0&ajaxserp=0"
}'
```

#### Response

```postman
{
"message": {
"type": "image",
"format": "jpg",
"description": "1200 Γ— 1201 png",
"url": "https://www.bing.com/images/search?view=detailV2&ccid=Tf4BFI68&id=D66EF5BFB7DA0A645A70240C32CB8664E8F8BF09&thid=OIP.Tf4BFI6846neirVSebC0vAHaEi&mediaurl=https%3a%2f%2flogos-download.com%2fwp-content%2fuploads%2f2016%2f09%2fNode_logo_NodeJS.png&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.4dfe01148ebce3a9de8ab55279b0b4bc%3frik%3dCb%252f46GSGyzIMJA%26pid%3dImgRaw%26r%3d0&exph=3061&expw=5000&q=jpg+nodejs&simid=608055434302923247&FORM=IRPRST&ck=2FF3D39CAEF945F20B996CF6042F88A6&selectedIndex=1&ajaxhist=0&ajaxserp=0",
"uuid": 104851112
}
}
```



### 2.1.4) Delete an object from the bucket

#### Request | Code snippet

```postman
curl --location --request DELETE 'http://localhost:4000/dev/delete-object/104851112' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c' \
--header 'x-api-key: f98d8cd98h73s204e3456998ecl9427j' \
--header 'Content-Type: application/json'
```

#### Response

```postman
{
"message": "Removed object with uuid 104851112 successfully."
}
```



## Section 3) Functionality Testing and References

### 3.0) Functionality Testing [πŸ”](#index-)

View details

#### [Watch video](https://www.youtube.com/playlist?list=PLCl11UFjHurDPyOkEXOR6JO-vUnYqd1FW)



### 3.1) References [πŸ”](#index-)

View details

#### Bucket Configuration

* [s3-example](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-configuring-buckets.html)
* [s3-examples official](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-node-examples.html)

#### Tools

* [AWS Design Tool app.diagrams.net](https://app.diagrams.net/?splash=0\&libs=aws4)

#### AWS-SDK

* [Official Doc](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html)

#### API Gateway

* [Best Api-Gateway Practices](https://docs.aws.amazon.com/whitepapers/latest/best-practices-api-gateway-private-apis-integration/rest-api.html)
* [Api-key creation custom](https://towardsaws.com/protect-your-apis-by-creating-api-keys-using-serverless-framework-fe662ad37447)

#### Libraries

* [Field validation](https://www.npmjs.com/package/node-input-validator)