Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bubblydoo/serverless-externals-plugin
Package only external modules, and let Rollup bundle all the other modules.
https://github.com/bubblydoo/serverless-externals-plugin
rollup rollup-plugin serverless serverless-framework
Last synced: 23 days ago
JSON representation
Package only external modules, and let Rollup bundle all the other modules.
- Host: GitHub
- URL: https://github.com/bubblydoo/serverless-externals-plugin
- Owner: bubblydoo
- License: mit
- Created: 2019-05-12T09:30:08.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2024-04-11T12:49:49.000Z (10 months ago)
- Last Synced: 2024-12-24T21:33:53.761Z (about 1 month ago)
- Topics: rollup, rollup-plugin, serverless, serverless-framework
- Language: TypeScript
- Homepage:
- Size: 2.84 MB
- Stars: 8
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Serverless Externals Plugin
Only include listed `node_modules` and their dependencies in Serverless.
This plugin helps Serverless package only external modules, and Rollup bundle all the other modules.
## Installation
```bash
npm install serverless-externals-plugin
```
or
```bash
yarn add serverless-externals-plugin
```
`serverless.yml`:
```yml
plugins:
- serverless-externals-plugin
package:
individually: true
functions:
handler:
handler: dist/bundle.handler
externals:
report: dist/node-externals-report.json
package:
patterns:
- "!./**"
- ./dist/bundle.js
```
`rollup.config.js`:
```js
import { rollupPlugin as externals } from "serverless-externals-plugin";
export default {
...
output: { file: "dist/bundle.js", format: "cjs" },
treeshake: {
moduleSideEffects: "no-external",
},
plugins: [
externals(__dirname, { modules: ["pkg3"] }),
commonjs(),
nodeResolve({ preferBuiltins: true, exportConditions: ["node"] }),
...
],
}
```
## Example
Externals Plugin interacts with both **Serverless** and with your **bundler** (Rollup).
Let's say you have two modules in your `package.json`, `pkg2` and `pkg3`. `pkg3` is a module with native binaries, so it can't be bundled.
```
root
+-- [email protected]
+-- [email protected]
+-- [email protected]
```
Because `pkg3` can't be bundled, both `./node_modules/pkg3` and `./node_modules/pkg2/node_modules/pkg3` should be included in the bundle.
`pkg2` can just be bundled, but should import `pkg3` as follows: `require('pkg2/node_modules/pkg3')`. It cannot just do `require('pkg3')`
because `pkg3` has a different version than `pkg2/node_modules/pkg3`.
In the Serverless package, only `./node_modules/pkg3/**` and `./node_modules/pkg2/node_modules/pkg3/**` should be included, all the other contents of `node_modules` are already bundled.
Externals Plugin provides a Serverless plugin and a Rollup plugin to support this.
There are other reasons modules can't be bundled. For example, [`readable-stream`](https://github.com/nodejs/readable-stream/issues/348) and [`sshpk`](https://github.com/joyent/node-sshpk/issues/42) cannot be bundled due to circular dependency errors.
## Configuration
In `rollup.config.js`:
```js
output: { file: "dist/bundle.js", format: "cjs" },
plugins: [
externals(__dirname, { modules: ["aws-sdk"], packaging: { exclude: ["aws-sdk"] } }),
...
]
```
This will generate a file called `node-externals-report.json` next to `bundle.js`, with the module paths that should be packaged.
It can then be included in `serverless.yml`:
```yml
custom:
externals:
report: dist/node-externals-report.json
```
### Configuration object
The configuration object has these options:
- `modules: string[]`: a list of module names that should be kept external (default `[]`)
- `report?: boolean | string`: whether to generate a report or report path (default `${distPath}/node-externals-report.json`)
- `packaging.exclude?: string[]`: modules which shouldn't be packaged by Serverless (e.g. `['aws-sdk']`, default `[]`)
- `packaging.forceIncludeModuleRoots?: string[]`: module roots that should always be packaged (e.g. `['node_modules/pg']`, default `[]`)
- `file?: string`: path to a different configuration object
It's also possible to filter on module versions. (e.g. `uuid@<8`). This uses a semver range.
## How it works
Externals Plugin uses [Arborist](https://github.com/npm/arborist) by NPM to analyze the `node_modules` tree (using `loadActual()`).
Using the Externals configuration (a list of modules you want to keep external), the Plugin will then build a list of all dependencies that should be kept external.
This list will contain the modules in the configuration and all the (non-dev) dependencies, recursively.
In the example, the list will contain both `pkg2/node_modules/pkg3` and `pkg3`.
The Rollup Plugin will then generate a report (e.g. `node-externals-report.json`) which contains the modules that are actually imported. This file is then used by the Serverless Plugin to generate a list of include patterns.
## Report
If you mark both `aws-sdk` and `sshpk` as external, but you don't use `sshpk`,
the generated report will look as follows:
`node-externals-report.json`:
```json
{
"isReport": true,
"importedModuleRoots": [
"node_modules/aws-sdk"
],
"config": {
"modules": [
"aws-sdk",
"sshpk"
]
}
}
```
Serverless can therefore ignore `sshpk` and all its dependencies, making the bundle even smaller.
The report is generated by analyzing the files Rollup emits (including dynamic imports).
## Rollup Plugin
A fully configured `rollup.config.js` could look as follows:
```js
import { rollupPlugin as externals } from "serverless-externals-plugin";
import commonjs from "@rollup/plugin-commonjs";
import nodeResolve from "@rollup/plugin-node-resolve";
/** @type {import('rollup').RollupOptions} */
const config = {
input: "index.js",
output: {
file: "bundle.js",
format: "cjs",
exports: "named",
dynamicImportInCjs: false,
},
treeshake: {
moduleSideEffects: "no-external",
},
plugins: [
externals(__dirname, { modules: ["aws-sdk"], packaging: { exclude: ["aws-sdk"] } }),
commonjs({ strictMode: true, ignoreDynamicRequires: true }),
nodeResolve({ preferBuiltins: true, exportConditions: ["node"] }),
],
};
export default config;
```
Make sure `externals` comes **before** `@rollup/plugin-commonjs` and `@rollup/plugin-node-resolve`.
Make sure [`moduleSideEffects: "no-external"`](https://rollupjs.org/guide/en/#treeshake) is set. By default, Rollup includes all external modules that appear in the code because they might contain side effects, even if they can be treeshaken.
By setting this option Rollup will assume external modules have no side effects.
(`"no-external"` is equivalent to `(id, external) => !external`)
Preferably, also set `exportConditions: ["node"]` as an option in the node-resolve plugin. This ensures that Rollup uses Node's resolution algorithm, so that packages like [`uuid` can be bundled](https://github.com/uuidjs/uuid/issues/544).
Next to that, in rare cases setting `strictMode: true` in the commonjs plugin can help bundling some modules that depend on the order of requires, like `aws-sdk` when only importing `aws-sdk/clients/dynamodb`.
### Implementation
The Rollup plugin provides a `resolveId` function to Rollup. For every import (e.g. `require('pkg3')`) in your source code,
Rollup will ask the Externals Plugin whether the import is external, and where to find it.
The Plugin will look for the import in the Arborist graph, and if it's declared as being external
it will return the full path to the module that's being imported (e.g. `pkg2/node_modules/pkg3`).
## `node-externals.json`
It's also possible to add the externals config to a file, called `node-externals.json`.
In `node-externals.json`:
```json
{
"modules": [
"pkg3"
]
}
```
In `rollup.config.js`:
```js
plugins: [
externals(__dirname, { file: 'node-externals.json' }),
...
]
```
## Dynamic requires
If your code or one of your Node modules does the following:
```js
require("p" + "g");
// or
import("p" + "g");
```
Then this plugin will not be able to detect which modules should be packaged. You can force include them by using `packaging.forceIncludeModuleRoots`:
```js
...
packaging: {
forceIncludeModuleRoots: ["node_modules/pg"]
}
```
The plugin will then treat `node_modules/pg` as if it was imported directly in the bundle. It will also include all the dependencies.
For example: in `knex`, an SQL builder, `pg` is a peer dependency. Inside `knex` it is imported as follows:
```js
// knex/lib/index.js
const resolvedClientName = resolveClientNameWithAliases(clientName);
Dialect = require(`./dialects/${resolvedClientName}/index.js`);
```
```js
// knex/lib/dialects/postgres/index.js
require('pg');
```
If you're using `knex`, you'd have to force include `node_modules/pg`.
If you want to bundle `knex`, you would also have to enable `ignoreDynamicRequires` in your `rollup.config.js`:
```js
commonjs({ ignoreDynamicRequires: true }),
```
This will make sure the `require` call is not changed.
Another solution is to add `knex` to the list of externals. In that case the whole `node_modules/knex` folder will be uploaded, and none of its code will be transformed.
Rollup sometimes keeps `await import` expressions in the bundle, which might cause import issues if the module is not commonjs. To fix this, you can add the following to your `rollup.config.js`:
```js
output: {
dynamicImportInCjs: false
}
```
## Usage in monorepos/workspaces
If your serverless project is a workspace within a larger monorepo, this is also supported, although not yet fully tested.
For example, in `apps/lambdas/rollup.config.js`:
```js
const root = path.resolve(__dirname, "../..")
const workspaceName = "main-app"
...
plugins: [
externals([root, workspaceName], { modules: ["undici"] }),
...
]
```
If `undici` is installed the monorepo root, the serverless plugin will generate include patterns as follows:
```
!./node_modules/**
!../../node_modules/**
../../node_modules/undici/**
!../../node_modules/undici/node_modules
../../node_modules/busboy/**
!../../node_modules/busboy/node_modules
../../node_modules/streamsearch/**
!../../node_modules/streamsearch/node_modules
```
Due to the way packaging in serverless works, in the final package,
the included modules from the root node_modules will be merged with the included workspace node_modules,
which is exactly what we want.
## Caveats
### Externals with side effects
It's unlikely, but if you have external modules with side effects (like polyfills), make sure to configure Rollup properly.
**NOTE**: This only applies to external modules. You should probably bundle your polyfills.
```js
import "some-external-module"; // this doesn't work, Rollup will treeshake it away
```
As Rollup will remove external modules with side effects, make sure to add something like this
to the Rollup config:
```js
treeshake: {
moduleSideEffects: (id, external) => !id.test(/some-external-module/) || !external
}
```
### Only one `node_modules` supported
This plugin doesn't have support for analyzing multiple `node_modules` folders. If you have
more `node_modules` folders on your `NODE_PATH` (e.g. from a Lambda layer), you can still use
the [`external` field of Rollup](https://rollupjs.org/guide/en/#external).
### Keeping `aws-sdk` excluded
As the `aws-sdk` node module is included by default in Lambdas, you can add `packaging.exclude: ["aws-sdk"]` to exclude it from the Serverless package. Note that this is not recommended because of possible version differences. (Check the `aws-sdk` version included in runtimes [here](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html))
### All subdependencies marked as external
When listing modules as external, all their subdependencies will also be marked as external.
For example:
```js
// rollup.config.js
plugins: [
externals(__dirname, { modules: ["botkit"] }),
...
]
```
```js
// lambda.js
const express = require('express');
const serverlessHttp = require('serverless-http');
const app = express();
module.exports.handler = serverlessHttp(app);
```
In the resulting bundle, `express` will not be bundled. This is because `botkit` also depends on `express`, and is therefore marked as external. `botkit` itself and the rest of its subdependencies will be filtered out of the modules to be uploaded.
It is therefore recommended to limit the length of the modules array to only the necessary. In that way you can achieve the smallest bundles.
## Todo
- Ensure compatibility with Serverless Jetpack or speedup packaging somehow
- Webpack plugin
- Esbuild plugin
- Layer support
- Look into externalizing single files in a module (e.g. `.node` files), and bundling the rest
- Yarn PnP support
- Pre-calculate actually used top-level externals to solve last caveat
## Motivation
I wanted to include Cheerio/JSDom and AWS SDK in a Typescript project, but neither could be bundled because of obscure errors, so they needed to be external. To reduce package size, I didn't want to make every module external. Manually looking up a module and adding its dependencies to `rollup.config.js` `and serverless.yml` is simply too much work. This plugin makes this much easier.
## Credits
Some Serverless-handling code was taken from [Serverless Jetpack](https://github.com/FormidableLabs/serverless-jetpack). Also inspired by [Serverless Plugin Include Dependencies](https://github.com/dougmoscrop/serverless-plugin-include-dependencies) and [Webpack Node Externals](https://github.com/liady/webpack-node-externals)