Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/focusaurus/hexagonal-lambda

Sample of an AWS lambda application using hexagonal architecture
https://github.com/focusaurus/hexagonal-lambda

Last synced: about 1 year ago
JSON representation

Sample of an AWS lambda application using hexagonal architecture

Awesome Lists containing this project

README 

          
# Hexagonal Lambda



This is an example application repository for implementing [hexagonal architecture](https://web.archive.org/web/20180822100852/http://alistair.cockburn.us/Hexagonal+architecture) as described by Alistair Cockburn (also called "ports and adapters") on top of Amazon Web Services Lambda.



It is intended to be a reference/example project implementation. While small, it is hopefully realistic and reasonably comprehensive. It takes into account development tooling, testing, deployment, security, developer documentation, and API documentation.



## How to develop locally via docker (initial setup)



* Install [docker](https://docs.docker.com/install/)

* Build the image

  * `./bin/docker-build.sh`

* Get a shell for development in docker

  * `./bin/docker-run.sh`

* Now you're in the dev container and should have the prerequisite development tools with the correct versions available (node, terraform, AWS CLI etc)

* Install npm deps and run tests

  * `npm install && npm test`



## How to develop locally without docker (initial setup)



* Install prerequisites

  * node and npm

    * See `.nvmrc` file for correct node version

    * Using [https://github.com/creationix/nvm](nvm) recommended but optional

  * zip

  * pass: `brew install pass`

  * AWS CLI: `brew install awscli`

    * **OR** `virtualenv python && ./python/bin/pip install awscli`

* Clone the git repo if you haven't already and `cd` into the root directory

* Run `npm install && npm run lint && npm test`

* Set up your `local/env.sh` based on the template below. More details on configuration further down in this document.



```

export AWS_DEFAULT_REGION='us-example-1'

export PASS_ENV='hexagonal-lambda-dev'

export HL_DEPLOY='dev'

export HL_HTTPBIN_URL='https://httpbin.org'

export TF_VAR_httpbin_url="${HL_HTTPBIN_URL}"

```



* To use that for terminal development we do `source ./local/env.sh`;



* Set up pgp and a private key for working with `pass`

  * There's some docs missing here on initially setting up a PGP key if you've never had one before and initializing your password store/repo. I'm currently pondering different alternatives for this so bear with me while I figure that out. Start with `gpg --full-generate-key`.

* Set up the following shell alias (run in project root directory):

  * `alias run-pass="${PWD}/bin/run-pass.sh"`



## How to do typical development



These steps are the same with or without docker.



* Build lambdas: `npm run build`

* Run lint: `npm run lint`

* Run tests: `npm test`

  * Run a single test file: `NODE_ENV=test tap code/get-hex/unit-tests-tap.js`

  * Run some tests in a file: `NODE_ENV=test tap --grep=example code/get-hex/unit-tests-tap.js`

    * This will only run tests whose description includes "example"

  * Run a few test files: `NODE_ENV=test tap code/get-hex/unit-tests-tap.js code/post-up/unit-tests-tap.js`

  * debug a single test file: `NODE_ENV=test node --inspect-brk=0.0.0.0:9229 code/get-hex/unit-tests-tap.js`

* Run a lambda locally: `node code/get-hex/run.js`

  * Edit the sample event object in the code as needed before calling the lambda handler to simulate a specific case of interest

* Run code coverage: `npm run coverage`

* Preview terraform: `(cd terraform/dev && run-pass terraform plan)`

* Provision for real: `(cd terraform/dev && run-pass terraform apply)`

* Run smoke (integration/system) tests

  * `export HL_API_URL=$(cd terraform/dev && run-pass terraform output api_url)`

    * where "terraform/dev" is the desired deployment to test

  * `npm run smoke` with the appropriate values for your deployment

* Trigger an API Gateway deployment: `run-pass npm run deploy-apig dev`

  * Substitute `demo` for `dev` to target that deployment

  * Note our terraform-triggered deployments currently have an ordering issue where they fire before other APIG changes are done, so manually deploying is sometimes required.

* Build OpenAPI JSON for documentation: `run-pass npm run openapi`

  * Spits out JSON to stdout. Copy/paste to a swagger UI if you want a pretty site.

  * `run-pass npm run openapi demo` if you want to set the demo deployment as the base URL

* Update secrets

  * for dev: `(cd secrets/dev && PASSWORD_STORE_DIR=. pass edit secrets.sh)`

  * for demo same put replace dev with demo



## Filesystem Layout



This project follows the same [underlying principles](https://github.com/focusaurus/express_code_structure#underlying-principles-and-motivations) I describe in my "Express Code Structure" sample project. Terraform doesn't play well with this as it requires grouping all `.tf` files in the same directory, so those are in a separate directory.



**File Naming Conventions**



* `lamba.js` AWS lambda handler modules

* `*-tap.js` tap unit test files

* `smoke-tests.js` smoke test files

* `openapi.js` Open API documentation as an object



## Lambda Organization



* Each lambda handler goes in its own directory under `code/name-of-lambda`

  * This directory contains

  * The lambda code itself goes in `lambda.js`

    * The handler function is exported as `exports.handler`

* I use the `mintsauce` middleware npm package to allow me to concisely mix and match reusable middlewares across all my lambdas

  * The middleware pattern from express is proven effective, but it has drawbacks around implicit middleware interdependencies and run order

  * Try not to over-rely on `call.local` shared state

* All lambda code is easy to test and develop on

  * Lambda tests are fully runnable offline

  * Lambda handlers are fully runnable local talking to AWS/Internet services

  * The whole test suite is fast to run

  * It's easy to run a single test file

  * It's easy to run a small group of test files

  * It's easy to run some or all tests under the devtools debugger

  * It's easy to run a lambda handler under the devtools debugger

* Each lambda has a corresponding `-tap.js` file for the unit tests

* Each lambda has a corresponding `terraform/*.tf` that defines the terraform configuration for that lambda function, and a corresponding API Gateway method as needed



## Input Validation



All key data shapes including end user input and external service responses is modeled as JSON schema and validated immediately upon arrival into the system. JSON schema has broad tool support (OpenAPI, many npm packages, etc) and is also cross-language. Currently we use `ajv` for validation as it is quite thorough and the error messages are good, although it's API is awkward. For each data shape, we have examples easily available for unit tests and ad-hoc developer testing convenience.



The `code/core/schemas.js` module provides some helper functions to make JSON schema easier to work with including `check(input)` and `example()` helper functions as properties on the schema object itself.



## Lambda Error Reporting



For lambdas triggered by API Gateway, most errors are "soft errors" and should be done via `callback(null, res);` where `res.statusCode` is the appropriate HTTP 400/500 value. I only pass an error as the first callback argument for programmer/deployment errors that will require developer/admin attention to fix. Examples would be invalid lambda environment variables or IAM errors accessing AWS resources. But an external service failing, invalid end user input, anything that might resolve itself with time should be considered "success" from the lambda callback perspective.



## Configuration



Non-secret configuration can be set as environment variables prefixed with `HL_` to distinguish them from the other variables present in your environment. Use the `local/env.sh` file to configure them into your shell. This file is excluded from git to allow each developer the ability to set distinct personal settings if desired.



Like external input, configuration data is considered external and thus we define the expected schema in JSON schema and validate it as early as possible and refuse to process invalid configuration. The code takes configuration key/value string settings from environment variables (both for local development and when running in lambda), and validates the configuration is sufficient before using that data.



When tests are run (`NODE_ENV=test`) a realistic but neutered/harmless ("example.com" etc) test configuration is forceably set so the test environment is consistent.



## Credentials and Secrets



Developers will be working with a set of AWS credentials when interacting with AWS APIs. These are security sensitive and must remain confidential and thus there's a fairly byzantine system described here about how we try to secure them. We generally don't want them available to be stolen by shoulder surfing or unauthorized access to your laptop or stolen by malicious install scripts during `npm install`. Thus the are encrypted at rest using the [pass](https://www.passwordstore.org/) password storage system. This essentially makes a database of secrets encrypted via [OpenPGP](https://en.wikipedia.org/wiki/Pretty_Good_Privacy#OpenPGP) and associated tools such as gnupg, gpg-agent, etc.



When running commands interacting with AWS such as `aws` or `terraform`, we'll use `./bin/run-pass.sh` in combination with gpg-agent to decrypt your permanent AWS credentials, which are then used with the AWS session token service to generate a set of temporary credentials, which we then expose to the subcommand actually doing the work as environment variables in a single short-lived subprocess.



Initial setup of secrets was done basically as follows:



```

cat <