{"id":15712435,"url":"https://github.com/razor-x/blobpack","last_synced_at":"2025-10-09T22:15:34.651Z","repository":{"id":47175781,"uuid":"404589928","full_name":"razor-x/blobpack","owner":"razor-x","description":"Packaging Benthos configs for AWS Lambda was too exciting, now it's acceptably boring.","archived":false,"fork":false,"pushed_at":"2024-09-10T06:24:39.000Z","size":3903,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-30T20:52:08.659Z","etag":null,"topics":["benthos","serverless"],"latest_commit_sha":null,"homepage":"http://io.evansosenko.com/blobpack/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/razor-x.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-09T04:53:12.000Z","updated_at":"2024-10-14T19:56:45.000Z","dependencies_parsed_at":"2024-09-10T08:17:55.916Z","dependency_job_id":"63540651-5498-486c-a7ff-0c8ddbb8d644","html_url":"https://github.com/razor-x/blobpack","commit_stats":{"total_commits":330,"total_committers":1,"mean_commits":330.0,"dds":0.0,"last_synced_commit":"b756d4b6d5a90777e735dcafacaeac5ea26beb22"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/razor-x%2Fblobpack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/razor-x%2Fblobpack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/razor-x%2Fblobpack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/razor-x%2Fblobpack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/razor-x","download_url":"https://codeload.github.com/razor-x/blobpack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251810897,"owners_count":21647626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benthos","serverless"],"created_at":"2024-10-03T21:16:12.252Z","updated_at":"2025-10-09T22:15:29.621Z","avatar_url":"https://github.com/razor-x.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ![Blobpack](./blobpack.svg) Blobpack\n\n[![npm](https://img.shields.io/npm/v/blobpack.svg)](https://www.npmjs.com/package/blobpack)\n[![GitHub Actions](https://github.com/razor-x/blobpack/workflows/main/badge.svg)](https://github.com/razor-x/blobpack/actions)\n\nPackaging [Benthos] configs for [AWS Lambda] was too exciting, now it's acceptably boring.\n\n_If all this still sounds too exciting and you just want to deploy Benthos on AWS Lambda,\njump over to the [Serverless Benthos Project Skeleton] or the [Benthos Plugin Project Skeleton]._\n\n[Benthos]: https://www.benthos.dev/\n[AWS Lambda]: https://aws.amazon.com/lambda/\n[Serverless Benthos Project Skeleton]: https://github.com/makenew/serverless-benthos\n[Benthos Plugin Project Skeleton]: https://github.com/makenew/benthos-plugin\n\n## Description\n\nBenthos on AWS Lambda requires either a single config.yaml file inside its\ndeployed .zip artifact or the config passed as an environment variable.\nSince AWS Lambda limits its environment variable size to 4 KB,\neven simple Benthos configs are too large to deploy.\n\nThis tool allows you to write and test Benthos configs as you normally would,\ni.e., splitting and sharing resources across multiple files.\nFor each config you want to deploy, Blobpack merges\nthe selected Benthos YAML files into a single config.yaml and\npackages it into a .zip file with Benthos for deployment to AWS Lambda.\n\nThis can be used with any tool that can deploy .zip artifacts to AWS Lambda.\nSince [Serverless] and the [AWS CDK] are both popular deployment\nsolutions that support Node.js, this tool is distributed as an npm package.\n\nThis package exposes the CLI command blobpack\nand its [underlying JavaScript API](http://io.evansosenko.com/blobpack/).\n\n[Serverless]: https://serverless.com/\n[AWS CDK]: https://aws.amazon.com/cdk/\n\n## Installation\n\nAdd this as a dependency to your project using [npm] with\n\n```\n$ npm install --save-dev blobpack\n```\n\n[npm]: https://www.npmjs.com/\n\n### Usage\n\n#### Download the Benthos Lambda Archive\n\nIn order to create Benthos artifact to deploy to AWS Lambda,\nthe upstream Benthos lambda archive must be downloaded locally to `tmp`.\n\nTo have this happen automatically after `npm install`,\nadd this to your `package.json`,\n\n```json\n{\n  \"scripts\": {\n    \"postinstall\": \"blobpack install\"\n  },\n  \"blobpack\": {\n    \"name\": \"benthos-lambda\",\n    \"version\": \"4.10.0\",\n    \"platform\": \"linux_amd64\",\n    \"src\": \"https://github.com/benthosdev/benthos/releases/download\"\n  }\n}\n```\n\n#### Create Serverless Artifacts\n\nFirst, add a new build step and ensure it runs before deployment,\n\n```json\n{\n  \"scripts\": {\n    \"blobpack\": \"blobpack\"\n  }\n}\n```\n\nAssuming you want to deploy the below Serverless function,\nyou will need to tell blobpack to generate the `boring.zip` artifact.\n\n_Tip: you can reuse the same artifact for multiple functions._\n\n```yaml\nboring:\n  handler: benthos-lambda\n  package:\n    artifact: dist/boring.zip\n    individually: true\n    exclude: ['*/**']\n    include: []\n```\n\nFirst, add the `artifacts` section to the `blobpack` config.\nThis will generate a new artifact to `dist/boring.zip` which uses\n`config/boring.yaml` and intelligently merges resources in both\n`resources/outputs.yaml` and `node_modules/@my-org/blobd/resources/logger.yaml`.\n\n_Tip: any top level keys which are not of type `*_resources` will still be included.\nIf two files have the same key, the last one wins._\n\n```json\n{\n  \"blobpack\": {\n    \"artifacts\": [\n      {\n        \"name\": \"boring\",\n        \"resources\": [\"outputs\"],\n        \"node_modules/@my-org/blobd/resources\": [\"logger\"]\n      }\n    ]\n  }\n}\n```\n\n_Tip: put your common resources in an npm package like @my-org/blobd._\n\nIf you only need to package a single config file into the artifact,\nyou can use this shorthand,\n\n```json\n{\n  \"blobpack\": {\n    \"artifacts\": [\"boring\"]\n  }\n}\n```\n\nIf you want to merge a common set of resources into every config,\nyou can use the `include` property,\n\n```json\n{\n  \"blobpack\": {\n    \"include\": {\n      \"resources\": [\"logger\"]\n    },\n    \"artifacts\": [\"boring\"]\n  }\n}\n```\n\n#### Bento support\n\nIf you need Serverless [Bento], set the `src` and `checksumPrefix` properties,\n\n```json\n{\n  \"blobpack\": {\n    \"name\": \"benthos-lambda\",\n    \"version\": \"1.2.0\",\n    \"platform\": \"linux_amd64\",\n    \"checksumPrefix\": \"bento\",\n    \"src\": \"https://github.com/warpstreamlabs/bento/releases/download\"\n  }\n}\n```\n\n[Bento]: https://warpstreamlabs.github.io/bento/\n\n### CLI\n\n```\n\n  Usage: blobpack [command] [options]\n\n\n  Commands:\n\n    install        Download Benthos .zip\n    build          Build .zip artifacts\n    help           Display help\n\n  Options:\n\n    --version         Output the version number\n    --config-path     Path to the JSON file containing the blobpack config\n    --tmp-root        Path to a temporary working directory\n    --config-root     Path to the directory containing the artifact configs\n    --resources-root  Path to the directory all resources are relative to\n    --dist-root       Path to the directory to output artifacts\n\n```\n\n## Development and Testing\n\n### Quickstart\n\n```\n$ git clone https://github.com/razor-x/blobpack.git\n$ cd blobpack\n$ nvm install\n$ npm install\n```\n\nRun the command below in a separate terminal window:\n\n```\n$ npm run test:watch\n```\n\nPrimary development tasks are defined under `scripts` in `package.json`\nand available via `npm run`.\nView them with\n\n```\n$ npm run\n```\n\n### Source code\n\nThe [source code] is hosted on GitHub.\nClone the project with\n\n```\n$ git clone git@github.com:razor-x/blobpack.git\n```\n\n[source code]: https://github.com/razor-x/blobpack\n\n### Requirements\n\nYou will need [Node.js] with [npm] and a [Node.js debugging] client.\n\nBe sure that all commands run under the correct Node version, e.g.,\nif using [nvm], install the correct version with\n\n```\n$ nvm install\n```\n\nSet the active version for each shell session with\n\n```\n$ nvm use\n```\n\nInstall the development dependencies with\n\n```\n$ npm install\n```\n\n[Node.js]: https://nodejs.org/\n[Node.js debugging]: https://nodejs.org/en/docs/guides/debugging-getting-started/\n[npm]: https://www.npmjs.com/\n[nvm]: https://github.com/creationix/nvm\n\n### Publishing\n\nUse the [`npm version`][npm-version] command to release a new version.\nThis will push a new git tag which will trigger a GitHub action.\n\nPublishing may be triggered using a [workflow_dispatch on GitHub Actions].\n\n[npm-version]: https://docs.npmjs.com/cli/version\n[workflow_dispatch on GitHub Actions]: https://github.com/razor-x/blobpack/actions?query=workflow%3Aversion\n\n## GitHub Actions\n\n_GitHub Actions should already be configured: this section is for reference only._\n\nThe following repository secrets must be set on [GitHub Actions]:\n\n- `NPM_TOKEN`: npm token for installing and publishing packages.\n\nThese must be set manually.\n\n### Secrets for Optional GitHub Actions\n\nThe version and format GitHub actions\nrequire a user with write access to the repository.\nSet these additional secrets to enable the action:\n\n- `GH_TOKEN`: A personal access token for the user.\n- `GIT_USER_NAME`: The GitHub user's real name.\n- `GIT_USER_EMAIL`: The GitHub user's email.\n- `GPG_PRIVATE_KEY`: The GitHub user's [GPG private key].\n- `GPG_PASSPHRASE`: The GitHub user's GPG passphrase.\n\nThese must be set manually.\n\n[GitHub Actions]: https://github.com/features/actions\n[GPG private key]: https://github.com/marketplace/actions/import-gpg#prerequisites\n\n## Contributing\n\nPlease submit and comment on bug reports and feature requests.\n\nTo submit a patch:\n\n1. Fork it (https://github.com/razor-x/blobpack/fork).\n2. Create your feature branch (`git checkout -b my-new-feature`).\n3. Make changes.\n4. Commit your changes (`git commit -am 'Add some feature'`).\n5. Push to the branch (`git push origin my-new-feature`).\n6. Create a new Pull Request.\n\n## License\n\nThis npm package is licensed under the MIT license.\n\n## Warranty\n\nThis software is provided by the copyright holders and contributors \"as is\" and\nany express or implied warranties, including, but not limited to, the implied\nwarranties of merchantability and fitness for a particular purpose are\ndisclaimed. In no event shall the copyright holder or contributors be liable for\nany direct, indirect, incidental, special, exemplary, or consequential damages\n(including, but not limited to, procurement of substitute goods or services;\nloss of use, data, or profits; or business interruption) however caused and on\nany theory of liability, whether in contract, strict liability, or tort\n(including negligence or otherwise) arising in any way out of the use of this\nsoftware, even if advised of the possibility of such damage.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frazor-x%2Fblobpack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frazor-x%2Fblobpack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frazor-x%2Fblobpack/lists"}