{"id":19732238,"url":"https://github.com/lifeomic/lambda-tools","last_synced_at":"2025-04-30T02:32:03.235Z","repository":{"id":33118485,"uuid":"135757304","full_name":"lifeomic/lambda-tools","owner":"lifeomic","description":"Common utilities for Lambda testing and development","archived":false,"fork":false,"pushed_at":"2025-03-11T23:31:23.000Z","size":5186,"stargazers_count":24,"open_issues_count":12,"forks_count":7,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-27T14:56:22.727Z","etag":null,"topics":["aws","lambda","serverless","team-mobile-backend"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lifeomic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-06-01T19:46:28.000Z","updated_at":"2025-01-03T20:54:24.000Z","dependencies_parsed_at":"2023-02-18T02:46:05.987Z","dependency_job_id":"aae4b1bf-e20e-442d-9b5b-ae2337076ccf","html_url":"https://github.com/lifeomic/lambda-tools","commit_stats":{"total_commits":540,"total_committers":28,"mean_commits":"19.285714285714285","dds":0.6574074074074074,"last_synced_commit":"11d2feae78966cb82ac3b050d6de5c7de6a092ad"},"previous_names":[],"tags_count":101,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lifeomic%2Flambda-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lifeomic%2Flambda-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lifeomic%2Flambda-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lifeomic%2Flambda-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lifeomic","download_url":"https://codeload.github.com/lifeomic/lambda-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251629197,"owners_count":21618120,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","lambda","serverless","team-mobile-backend"],"created_at":"2024-11-12T00:25:23.622Z","updated_at":"2025-04-30T02:32:02.518Z","avatar_url":"https://github.com/lifeomic.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lambda-tools\n\n[![npm](https://img.shields.io/npm/v/@lifeomic/lambda-tools.svg)](https://www.npmjs.com/package/@lifeomic/lambda-tools)\n[![Build Status](https://github.com/lifeomic/lambda-tools/actions/workflows/release.yaml/badge.svg)](https://github.com/lifeomic/lambda-tools/actions/workflows/release.yaml)\n[![Coverage Status](https://coveralls.io/repos/github/lifeomic/lambda-tools/badge.svg?branch=master)](https://coveralls.io/github/lifeomic/lambda-tools?branch=master)\n![Dependabot Badge](https://flat.badgen.net/dependabot/lifeomic/lambda-tools?icon=dependabot)\n\n`lambda-tools` provides a set of utilities that are useful for development and\ntesting of Lambda based services. The functionality is divided into several\nmain categories so that features may be adopted and used as needed. Working code\nexamples can be found in the [examples](./examples) directory.\n\n```\nnpm install --save @lifeomic/lambda-tools\n```\n\n## DynamoDB\n\nMany services use [DynamoDB][dynamodb] as their primary storage solution.\nTesting service code against a DynamoDB storage layer requires either mocking\nthe DynamoDB interface (using something like [aws-sdk-mock][aws-sdk-mock]) or\npointing the test code at a provisioned DynamoDB instance. AWS has published\n[DynamoDB Local][dynamodb-local] so that testing can be done without having to\nuse real AWS resources. DynamoDB Local has also been published as a\n[community docker image][dynamodb-image] making testing even easier to do using\ntools like [docker-compose][docker-compose] and [dockerode][dockerode].\n\n`lambda-tools` supports both methods of integrating with DynamoDB. For simple\nunit tests, the `dynamodb` helper can be used to provision and link DynamoDB\ndocker containers with [ava][ava] test suites. The helper will define all\nrelevant environment variables and will inject a preconfigured\n[DyanomDB client][dynamodb-client] into the test context. The setup and tear\ndown will also create and destroy tables, as defined in the schema, between\ntest cases. The helper will also automatically handle port binding differences\nbetween regular and nested Docker environments.\n\n`lambda-tools` managed containers are able to join an existing docker-compose\nnetwork allowing them to reuse an existing DynamoDB container by simply setting\nthe appropriate environment variables. If the `dynamodb` helper sees the\n`DYNAMODB_ENDPOINT` environment variable, it will not provision a new DynamoDB\ncontainer and will instead point all clients at and perform all table\nmanipulations in the referenced DynamoDB instance. In this case AWS specific\nenvironment variables, like `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`,\nwill also be used when constructing the test clients.\n\n### `dynamodb.tableSchema(schema)`\n\nDeclares the table schema that should be used by all test cases using the\n`useDynamoDB()` helper. A `schema` a list of parameter objects accepted by\n[`AWS.DynamoDB.createTable()`][dynamodb-create-table].\n\n### `dynamodb.useDynamoDB(test, useUniqueTables)`\n\nPrepares an [ava][ava] test suite for use with DynamoDB. The test context will\ninclude the following clients on the `dynamodb` attribute:\n\n| Attribute        | Description/Type            |\n|------------------|-----------------------------|\n| documentClient   | AWS.DynamoDB.DocumentClient |\n| dynamoClient     | AWS.DynamoDB                |\n| streamsClient    | AWS.DynamoDBStreams         |\n| tableNames       | Map of base table name to uuid table name. E.g. `users` to `users-abcdef12345`|\n| uniqueIdentifier | The unique identifier appended to each table name. E.g. `abcdef12345`|\n| config           | The aws dynamodb config object, useful for passing to dynamoose or other dynamo wrappers.|\n\nIf `useUniqueTables` is true, dynamically generated table names will be used, in\nthe form of `\u003ctableNameProvidedInSchema\u003e-\u003cuuid\u003e`. The unique table name can be\nfetched from the `tableNames` map. Otherwise, the table name will be the default\nprovided in the schema. This allows tests to be run in parallel.\n\n\n## Kinesis\n\nMany services use [Kinesis][kinesis] as a message processing system.\nTesting service code against a Kinesis service layer requires either mocking\nthe Kinesis interface (using something like [aws-sdk-mock][aws-sdk-mock]) or\npointing the test code at a provisioned Kinesis instance. LocalStack has published\n[LocalStack Docker][localstack-docker] so that testing can be done without having to\nuse real AWS resources, making testing even easier to do using\ntools like [docker-compose][docker-compose] and [dockerode][dockerode].\n\n`kinesis-tools` supports both methods of integrating with Kinesis. For simple\nunit tests, the `kinesis` helper can be used to provision and link a Kinesis\ndocker container with [ava][ava] test suites. The helper will define all\nrelevant environment variables and will inject a preconfigured\n[Kinesis client][kinesis-client] into the test context. The setup and tear\ndown will also create and destroy streams, as defined in a provided stream name array, between\ntest cases. The helper will also automatically handle port binding differences\nbetween regular and nested Docker environments.\n\n`lambda-tools` managed containers are able to join an existing docker-compose\nnetwork allowing them to reuse an existing Kinesis instance by simply setting\nthe appropriate environment variables. If the `kinesis` helper sees the\n`KINESIS_ENDPOINT` environment variable, it will not provision a new Kinesis\nservice and will instead point all clients at and perform all stream\nmanipulations in the referenced Kinesis instance. In this case AWS specific\nenvironment variables, like `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`,\nwill also be used when constructing the test clients.\n\n### `kinesis.streams(streams)`\n\nDeclares the list of stream names that should be used by all test cases using the\n`useKinesis()` helper.  Each stream will be created with one shard.\n\n### `kinesis.useKinesis(test, streamName)`\n\nPrepares an [ava][ava] test suite for use with a provided Kinesis instance. The test context will\ninclude a AWS.Kinesis client on the `kinesis` attribute.\n\nThe streamName is used to create a new kinesis stream before all tests, and delete the stream at the end.\nThis method can be used to create multiple streams.\n\n### `kinesis.useKinesisDocker(test, useUniqueStreams)`\n\nPrepares an [ava][ava] test suite for use with Kinesis. The test context will\ninclude the following clients on the `kinesis` attribute:\n\n| Attribute        | Description/Type            |\n|------------------|-----------------------------|\n| kinesisClient    | AWS.Kinesis |\n| streamNames      | Map of base stream name to uuid stream name. E.g. `users` to `users-abcdef12345`|\n| uniqueIdentifier | The unique identifier appended to each stream name. E.g. `abcdef12345`|\n| config           | The aws kinesis config object, useful for passing to AWS.Kineis, or other kinesis wrappers.|\n\nIf `useUniqueStreams` is true, dynamically generated stream names will be used, in\nthe form of `\u003cstreamNameProvided\u003e-\u003cuuid\u003e`. The unique stream name can be\nfetched from the `streamNames` map. Otherwise, the stream name will be the default\nprovided in the streams array. This allows tests to be run in parallel.\n\n# Kinesis Tools\n\n`kinesis.tools` contains some simple kinesis tools to iterate a stream.\n\n## `KinesisIterator`\n\nThe `KinesisIterator` class provides a simple method to get the records from a stream by creating the stream iterator,\nand reusing it to get records.\n\n### Iterator `config`\n| Attribute     | Description/Type |\n|---------------|------------------|\n| kinesisClient | AWS.Kinesis |\n| streamName | the name of the kinesis stream |\n\n### `new KinesisIterator(config)`\n\n### `async init()`\nThe init function creates the stream iterator so it can get records from the stream.\n\n### `async next(limit)`\nFetches the next batch of records from the stream.  It auto updates it's position in the stream, and returns the KinesisIterator\n\n| Attribute     | Description/Type |\n|---------------|------------------|\n| limit | an optional limit to how many records to return.  Max is 10,000 |\n\n### `records`\nThe records returned from the last `KinesisIterator.next()` call.\n\n### `response`\nThe complete response from the last `KinesisIterator.next()` call.\n\n### `static async newIterator(config)`\nThe static `KinesisIterator.newIterator` function creates a new `KinesisIterator`, and calls the init function.\n\n## `async getStreamRecords(config)`\nA convenience method to get a single batch of records from a stream.\n\n\n## `createLambdaEvent(records)`\nConvert an array of kinesis records into a lambda trigger event.\n\n\n| Attribute      | Description/Type |\n|----------------|------------------|\n| SequenceNumber | number or string |\n| PartitionKey   | The partition key of the record |\n| SequenceNumber | number\n| Data           | The data buffer returned by `kinesis.getRecords(...)` |\n\n## `kinesisLambdaTrigger({lambdaHandler, kinesisIterator, limit})`\nWill iterate through a kinesis stream, and pass the events to the lambdaHandler.\n\n\n| Attribute       | Description/Type |\n|-----------------|------------------|\n| lambdaHandler   | A function to call the lambda function.  It will be called with { Records } |\n| kinesisIterator | KinesisIterator that is already provisioned to iterate through all of the records |\n| limit           | number. optional to limit how big each batch should be |\n\n\n## GraphQL\n\n`lambda-tools` provides helpers for wrapping [`Koa`][koa] instances in a client\nthat can easily make GraphQL requests. GraphQL reports application errors as\npart of the response payload rather than through the use of HTTP status codes.\nThe `graphql` helpers provides specialized assertions that can be used to\ncheck the status of a GraphQL response.\n\n### `graphql.assertError(response, path, messageTest)`\n\nAsserts that an error was returned on the response. `response` is the response\nobject returned from the helper client. `path` is an object path used to select\nthe error to test (GraphQL can return multiple errors for a single query).\n`messageTest` is either a string or a function used to test the error message.\nWhen a string is used the error message must be equal to the string. When a\nfunction is used the function must return true if the message meets\nexpectations.\n\n### `graphql.assertSuccess(response)`\n\nAsserts that no errors are included on the response. `response` is the response\nobject returned from the helper client.\n\n### `graphql.setupGraphQL(fn)`\n\nPrepares a Koa app instance for use by the `useGraphQL()` helper. `fn` is\ninvoked with the `test` instance and must return a Koa application.\n\n### `graphql.useGraphQL(test, options)`\n\nPrepares an [ava][ava] test suite for use with theh GraphQL helper client. The\ntest context will be augmented with a `graphql(query, variables)` method that\nwill use [supertest][supertest] to POST data to the `options.url` endpoint and\nreturn the response. The default endpoint is `/graphql`.\n\n## MockServer Lambda\n\nA collection of helper methods to mock and verify Lambda invocations based on [MockServer](https://mock-server.com/)\n\n\n## Lambda\n\nReplicating the Lambda runtime environment in a local test framework is a\nnon-trivial thing to do. `lambda-tools` provides helper functions that can\neither provision managed containers on a developer's behalf or reuse containers\nthat have been provisioned externally (using something like\n[docker-compose][docker-compose]). It is also possible to use a managed Lambda\ncontainer with other existing infrastructure managed by docker-compose.\n\n### `lambda.useComposeContainer(options)`\n\nConfigures the `lambda` helper to use an existing container managed by\ndocker-compose in all test suites leveraging the `useLambda()` helper. This\nhelper depends on the `COMPOSE_PROJECT_NAME` environment variable being set in\nthe test process. The following options must be provided:\n\n - **handler** -- the reference to the Lambda handler function in the form\n   `\u003cmodule\u003e.\u003cfunction name\u003e`\n - **service** -- the name of the compose service providing the Lambda runtime\n\n### `lambda.useNewContainer(options)`\n\nConfigures the `lambda` helper to provision a new Docker container managed by\n`lambda-tools` for use in test cases. The following options are supported:\n\n - **environment** -- a map of environment variables to be defined in the\n   Lambda execution environment\n - **mountpoint** -- _required._ The directory that should be used as the\n   Lambda task root. This should contain the Lambda code bundle.\n - **handler** - _required._ The reference to the Lambda handler function in\n   the form `\u003cmodule\u003e.\u003cfunction name\u003e`\n - **image** - the docker image used to provide the Lambda runtime. By default\n   `lambci/lambda:nodejs10.x` is used.\n - **useComposeNetwork** - a flag indicating if the container should be attached\n   to a docker-compose managed network. By default the container uses a\n   dedicated isolated network. If set to `true`, the `COMPOSE_PROJECT_NAME`\n   environment variable must also be available in the test process.\n\n### `lambda.useLambda(test, options)`\n\nPrepares an [ava][ava] test suite for use with a Lambda runtime. `options` may\nbe used to provide or override any options from `useComposeContainer()` or\n`useNewContainer()`. The test context will be augmented with a `lambda`\nattribute. This attribute is an [alpha][alpha] client instance that will invoke\nthe configured Lambda function and has been augmented with two additional\nmethods. The `graphql(path, query, variables, config)` method will execute\na GraphQL query against the function where `path` is the URL path to POST to,\n`query` and `variables` define the GraphQL request, and `config` provides any\nadditional `alpha` parameters (like request headers). The `raw(event, context)`\nmethod allows a raw Lambda event to be passed to the function and will return\nthe raw response object.\n\n### Lambda Webpack Bundle CLI\n\nBuilding code bundles that are optimized for the Lambda runtime can be a\ntedious exercise. In order to share code and learning in this area across\nseveral projects, `lambda-tools` provides a `lambda-tools-build` command that\nwill generate Lambda code bundles. The CLI is capable of building a single\nbundle or multiple bundles and includes source maps, transpiling, minification,\nand relevant polyfills. When building a single bundle the output may also be\nzipped so that it is ready for upload to the Lambda environment. The CLI\ndocumentation may be accessed using the `lambda-tool-build --help` command.\n\n**Build all lambda functions within a directory:**\n\n```bash\nlambda-tools-build -z -s my-service -n 8.10 -o ./dist/lambdas ./src/lambdas\n```\n\nYour `./src/lambdas` directory should look similar to:\n\n- `./src/lambdas/func1/index.js`\n- `./src/lambdas/func2/index.ts`\n- `./src/lambdas/func3.js`\n- `./src/lambdas/func4.ts`\n\nThis will produce the following zip files:\n\n- `./dist/lambdas/func1.js.zip`\n- `./dist/lambdas/func2.js.zip`\n- `./dist/lambdas/func3.js.zip`\n- `./dist/lambdas/func4.js.zip`\n\n**Build a single lambda function and provide a name for the file:**\n\n```bash\n lambda-tools-build -z -s my-service -n 8.10 -o ./dist/lambdas ./src/lambdas/my-function/index.ts:my-function.js\n ```\n\n This will produce the following zip files:\n\n- `./dist/lambdas/my-function.js.zip`\n\nYou will also find the following intermediate files:\n\n- `./dist/lambdas/my-function.js`\n- `./dist/lambdas/my-function.js.map`\n\n**Build a TypeScript lambda function with a custom tsconfig**\n\n```bash\n lambda-tools-build -t tsconfig-prod.json -o ./dist src/service.ts\n```\n\n**Development mode:**\n\n```bash\n WEBPACK_MODE=development lambda-tools-build -z -s my-service -n 8.10 -o ./dist/lambdas ./src/lambdas/my-function/index.ts:my-function.js\n ```\n\n The `WEBPACK_MODE=development` environment variable will prevent\n minification in the final output bundle.\n\n## Debugging\n\nTo enable debug level logging we are using the [debug][debug] library to create the log lines.\n\nAvailable flags are\n- `lambda-tools:lambda`\n- `lambda-tools:docker`\n- `lambda-tools:localstack`\n- `lambda-tools:webpack`\n\n## Concurrency\n\nIf you are experiencing Docker networking errors, it might be because of the high level of concurrency\nin the kinesis, docker, or localstack setup processes.  To specify how many connections to make at once\nset the `LAMBDA_TOOLS_CONCURRENCY` environment variable to some integer value.\n\n\n[alpha]: https://bitbucket.org/lifeomic/alpha/src/master/ \"alpha\"\n[ava]: https://github.com/avajs/ava \"Ava\"\n[aws-sdk-mock]: https://github.com/dwyl/aws-sdk-mock \"aws-sdk-mock\"\n[debug]: https://github.com/visionmedia/debug \"debug\"\n[docker-compose]: https://docs.docker.com/compose/ \"Docker Compose\"\n[dockerode]: https://github.com/apocas/dockerode \"Docker + Node = Dockerode\"\n[dynamodb]: https://aws.amazon.com/documentation/dynamodb/ \"DynamoDB\"\n[dynamodb-client]: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html \"DynamoDB Client\"\n[dynamodb-create-table]: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#createTable-property \"DynamoDB Client: Create Table\"\n[dynamodb-image]: https://hub.docker.com/r/amazon/dynamodb-local \"DynamoDB Docker Image\"\n[dynamodb-local]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html \"DynamoDB Local\"\n[koa]: http://koajs.com/ \"koa\"\n[supertest]: https://github.com/visionmedia/supertest \"supertest\"\n[kinesis]: https://aws.amazon.com/documentation/kinesis/ \"Kinesis\"\n[kinesis-client]: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Kinesis.html \"Kinesis Client\"\n[localStack]: https://github.com/localstack/localstack \"LocalStack\"\n[localStack-docker]: https://hub.docker.com/r/localstack/localstack/ \"LocalStack Docker\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flifeomic%2Flambda-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flifeomic%2Flambda-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flifeomic%2Flambda-tools/lists"}