https://github.com/accenture/serverless-ephemeral
This is a Serverless Framework plugin that helps bundling any stateless zipped library to AWS Lambda.
https://github.com/accenture/serverless-ephemeral
Last synced: about 1 year ago
JSON representation
This is a Serverless Framework plugin that helps bundling any stateless zipped library to AWS Lambda.
- Host: GitHub
- URL: https://github.com/accenture/serverless-ephemeral
- Owner: Accenture
- Created: 2017-09-12T19:33:08.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2023-01-03T18:48:22.000Z (over 3 years ago)
- Last Synced: 2025-04-19T14:35:55.622Z (about 1 year ago)
- Language: JavaScript
- Homepage:
- Size: 887 KB
- Stars: 67
- Watchers: 31
- Forks: 15
- Open Issues: 23
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Serverless Ephemeral
[![NPM Version][npm-image]][npm-url]
[![Build Status][travis-image]][travis-url]
Serverless Ephemeral (or Serephem) is a [Serverless Framework plugin](https://serverless.com/framework/docs/providers/aws/guide/plugins/) that helps bundle any stateless library into a Lambda deployment artifact.
## Pre-requirements
- Node >= 6.9
- Serverless Framework >= 1.12.0
- Docker (with docker compose)
## Examples
- [TensorFlow Lambda (CPU only, Python 2.7)](examples/tensorflow-lambda): Pulls in or builds (via Docker) a TensorFlow package. For reference on how to manually build a TensorFlow package for Lambdas, see [docs/build-tensorflow-package.md](docs/build-tensorflow-package.md).
- [Image Processor](examples/image-processor): Builds and adds a Python [Pillow](https://python-pillow.org/) package in order to resize an image uploaded to an S3 bucket.
## Add the plugin
1. Install it
```bash
npm i --save-dev serverless-ephemeral
```
1. Add it to your `serverless.yml` file and exclude the `.ephemeral` directory
```yaml
plugins:
- serverless-ephemeral
package:
exclude:
- package.json
- package-lock.json
- node_modules/**
- .ephemeral/**
```
1. Add the `.ephemeral` directory to `.gitignore`
```text
# Serverless Framework
.serverless
.ephemeral
```
## Configuration
The configuration for the Ephemeral plugin is set inside the `custom` section of the `serverless.yml` file. In it, you can define the list of stateless libraries you wish to pull into the final Lambda artifact.
There are two types of configuration:
- [Build a library during runtime](#build-a-library)
- [Download a library](#download-a-library)
Both can be enhanced with [global configuration options](#global-options).
### Build a library
You can build a specific library during runtime. This is achieved via a Docker container that outputs a zip library.
The Serepehm plugin provides some useful packagers out of the box. However, you can create your own packager via Docker files.
#### Serephem packagers
You can use one of the Docker packagers provided with the Serephem plugin.
##### TensorFlow
```yaml
custom:
ephemeral:
libraries:
- packager:
name: tensorflow
version: 1.8.0
```
- **packager.name** is required. This is the packager name identifier for TensorFlow: **tensorflow**
- **packager.version** is required. This will determine which TensorFlow version you want to build.
#### Build your own packager
You can create your own packager via Docker. To do so:
1. Create a directory where you will store all your Docker files:
```bash
mkdir my-packager
cd my-packager
```
1. Create a `docker-compose.yml` file. For example:
```yaml
version: "3"
services:
packager:
build: .
```
Keep note of the name of your packager service, in this case `packager`.
1. Create a `Dockerfile` and any other support files. For example:
**`Dockerfile`**
```yaml
FROM amazonlinux
COPY scripts/build.sh scripts/build.sh
RUN yum -y install zip && \
chmod +x scripts/build.sh
CMD [ "scripts/build.sh" ]
```
**`scripts/build.sh`**
```bash
# create zip destination directory
mkdir -p /tmp/lambda-libraries
# download library files
mkdir /tmp/files
cd /tmp/files
curl http://example.com/file-1.py --output file-1.py
curl http://example.com/file-2.py --output file-2.py
zip -9rq /tmp/lambda-libraries/library-a.zip *
```
**IMPORTANT**: the container must generate a zip file containing the stateless library files. Thus:
- Your container must zip the stateless library files.
- You must create a directory where the final zip(s) will be stored. This directory will be mounted to the Serephem's libraries directory, so add only the necessary zip files.
- It is recommended that your Docker container extends from `amazonlinux` image to maximize compatibility with the Lambda environment.
1. Add this configuration to your `serverless.yml`:
```yaml
custom:
ephemeral:
libraries:
- packager:
compose: my-packager/docker-compose.yml
service: packager
output: /tmp/lambda-libraries/library-a.zip
```
Notice how each of the values correspond to a setting previously created:
- `compose`: points to your Docker compose file, inside the directory you created
- `service`: the name of the service inside the `docker-compose.yml` file
- `output`: the output path for the zip file in the Docker container
### Download a library
You can download a previously zipped library that contains all the necessary files and automatically add it to your Lambda deployment.
```yaml
custom:
ephemeral:
libraries:
- url: https://xxxxx.s3.amazonaws.com/my-library.zip
```
- **url** is required. This is the packaged library you want to include. The library must be a zip file.
> Documentation explaining how to create a deployable TensorFlow zipped package can be found here: [docs/build-tensorflow-package.md](docs/build-tensorflow-package.md). This approach can be used as a base to create other stateless libraries.
### Global options
```yaml
custom:
ephemeral:
libraries:
- packager:
name: tensorflow
version: 1.8.0
directory: tfpackage
- url: https://xxxxx.s3.amazonaws.com/my-library.zip
nocache: true
```
- **directory** is optional. When ommitted, the package contents will be unzipped at service root level. If entered, a new folder will be created at that level using the specified name and everything will be unzipped there. The folder can only be named using alphanumeric characters and the symbols `. _ -`
- **nocache** is optional. When ommitted or set to _false_, it will use the locally cached copy of the library. Otherwise, if set to _true_, it will re-fetch (download or build) the library every time the service is packaged.
> Note: the **forceDownload** option has been deprecated in favor of **nocache**
## Deploy
Deploy your service normally with the `serverless deploy` (or `sls deploy`) command. If you use the `-v` option, Ephemeral will show more information about the process.
```bash
sls deploy -v
```
> If the Serverless deployment is timing out, use the `AWS_CLIENT_TIMEOUT` environment variable:
### The .ephemeral directory
During the deployment process, the `.ephemeral` directory will be created. The purpose of this directory is:
- Saving the libraries' zip files inside the `.ephemeral/lib` folder
- Bundling the libraries and the Serverless Lambda function file(s) inside the `.ephemeral/pkg` folder
---
## Development
This plugin is created with Node and uses the Serverless Framework hooks to execute the necessary actions.
### Getting started
1. Clone this repo
2. Install dependencies via `npm i`
### Running Lint
The plugin code uses the AirBnB ESLint rule set with some enhancements (see `.eslintrc` file). To run the linter:
```bash
npm run lint
```
### Tests
The unit tests are coded with [Ava](https://github.com/avajs/ava) and [SinonJS](http://sinonjs.org/docs/). They can be found inside the `spec` folder.
To run the tests:
```bash
npm test
```
To run tests on "watch" mode and add verbosity:
```bash
npm test -- --watch -v
```
[npm-image]: https://img.shields.io/npm/v/serverless-ephemeral.svg
[npm-url]: https://npmjs.org/package/serverless-ephemeral
[travis-image]: https://img.shields.io/travis/Accenture/serverless-ephemeral/master.svg
[travis-url]: https://travis-ci.org/Accenture/serverless-ephemeral
### Test via examples
Refer to the [`examples`](examples) directory, for instance the [`TensorFlow example`](examples/tensorflow-lambda/README.md).