Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/little-core-labs/tiny-module-compiler
Compile, archive, unpack, and load compiled modules leveraging v8 cached data.
https://github.com/little-core-labs/tiny-module-compiler
cache compiler module node tiny tinybox v8
Last synced: 1 day ago
JSON representation
Compile, archive, unpack, and load compiled modules leveraging v8 cached data.
- Host: GitHub
- URL: https://github.com/little-core-labs/tiny-module-compiler
- Owner: little-core-labs
- License: mit
- Created: 2019-11-04T02:49:09.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2021-11-11T21:53:16.000Z (about 3 years ago)
- Last Synced: 2024-11-17T02:51:31.298Z (about 2 months ago)
- Topics: cache, compiler, module, node, tiny, tinybox, v8
- Language: JavaScript
- Homepage:
- Size: 195 KB
- Stars: 14
- Watchers: 4
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
**_Compile, archive, unpack, and load compiled modules leveraging v8 cached data._**
* [**Status**](#status)
* [**Installation**](#installation)
* [**Abstract**](#abstract)
* [**Basic Usage**](#usage)
* [**API**](#api)
* [`compile(target[, options], callback)`](#api-compile)
* [`archive(target, objects[, options], callback)`](#api-archive)
* [`load(target[, options], callback)`](#api-load)
* [`unpack(target[, options], callback)`](#api-unpack)
* [Class: `Compiler`](#api-class-compiler)
* [Constructor: `new Compiler([options])`](#api-class-compiler-constructor)
* [Accessor: `compiler.targets`](#api-class-compiler-targets)
* [`compiler.ready(callback)`](#api-class-compiler-ready)
* [`compiler.target(filename[, options], callback)`](#api-class-compiler-target)
* [`compiler.compile([options], callback)`](#api-class-compiler-target)
* [Class: `Archiver`](#api-class-archiver)
* [Constructor: `new Archiver([options])`](#api-class-archiver-constructor)
* [`archiver.ready(callback)`](#api-class-archiver-ready)
* [`archiver.archive(filename, objects[, options], callback)`](#api-class-archiver-archive)
* [Class: `Loader`](#api-class-loader)
* [Constructor: `new Loader([options])`](#api-class-loader-constructor)
* [`loader.ready(callback)`](#api-class-loader-ready)
* [`loader.load(filename[, options], callback)`](#api-class-loader-load)
* [Class: `Target`](#api-class-target)
* [Constructor: `new Target(filename[, options])`](#api-class-target-constructor)
* [Accessor: `target.fd`](#api-class-target-fd)
* [`target.stat(callback)`](#api-class-target-stat)
* [`target.read(offset, size, callback)`](#api-class-target-read)
* [`target.ready(callback)`](#api-class-target-ready)
* [**Command Line Interface**](#cli)
* [**The `tmc(1)` Command**](#cli-tmc)
* [**Compiling Modules**](#cli-compiling)
* [**Archiving Compiled Modules**](#cli-archiving-modules)
* [**Copying Relocated Assets**](#cli-copying-assets)
* [**Archiving Relocated Assets**](#cli-archiving-assets)
* [**Unpacking Archives**](#cli-unpacking)
* [**Loading Modules & Archives**](#cli-loading)
* [**Constraints**](#constraints)
* [**v8 Version**](#constraints-v8-version)
* [`Function.prototype.toString()`](#constraints-function-to-string)
* [**See Also**](#see-also)
* [**Prior Art**](#prior-art)
* [**License**](#license)
> **Stable/Documentation**
> [](https://github.com/little-core-labs/tiny-module-compiler/actions)
```sh
$ npm install tiny-module-compiler
```
The `tiny-module-compiler` module is small toolkit for compiling
JavaScript CommonJs modules into standalone binaries that leverage that
v8 cache data format exposed by the [`vm`](https://nodejs.org/api/vm.html)
API. Compiled module objects can be archived into a single file based on the
[TinyBox][tinybox] file format and then unpacked later to the file
system.
This toolkit allows for the compilation of an entire project into a single
compiled binary object file. Multiple binary object files and various
assets (`*.node`, `*.so`, etc) can be packaged into an archive and
unpacked to file system for later use making it suitable as a delivery
mechanism.
Compiled modules objects can be loaded and executed but must be in a
runtime that uses the same [version of v8](#constraints-v8-version).
```js
const tmc = require('tiny-module-compiler')
const targets = '*.js'
const archiveName = 'modules.a'
// compile module targets into compiled module objects
tmc.compile(targets, (err, objects) => {
// archive objects into file specified by `archiveName`
tmc.archive(archiveName, objects, (err) => {
tmc.load(archiveName, (err, archive) => {
// `archive` contains a mapping of compiled module object
// filenames to loaded exports
})
})
})
```
## API
The `tiny-module-compiler` module exports a public API described in this
section.
### `compile(target[, options], callback)`
Compiles a file specified at `target`. The default behavior is to write
the output to a file of the same name as `target` with `.out` appended
to the end. This behaviour can be configured by specifying an
`options.output` option or `options.storage` as a custom
[random-access-storage][ras] _factory function_.
The value of `options` is optional and can be:
```js
{
// current working directory for compilation
cwd: process.cwd(),
// output filename (single file compilation) or directory (multiple files)
output: target + '.out',
// custom storage factory function to return
// 'random-access-storage' compliant object
storage(filename) {
return require('random-access-file')(filename)
}
}
```
#### Examples
##### Simple Compilation
A simple compilation example that compiles a target input file to an
output file.
```js
const { compile } = require('tiny-module-compiler')
// compile this file
const target = __filename
const output = __filename + '.out'
compile(target, { output }, (err) => {
// `target` compiled to `output`
})
```
##### Simple Compilation to Memory
A simple compilation example that compiles a target input file to an in
memory `random-access-memory` storage.
```js
const { compile } = require('tiny-module-compiler')
const ram = require('random-access-memory')
// compile this file
const target = __filename
const storage = ram()
compile(target, { storage: () => storage }, (err) => {
// `target` compiled and written to `storage`
})
```
### `archive(target, inputs[, options], callback)`
Archives the entries found in a given `inputs` `Map` into `target`. The
default behaviour is to enumerate the entries in `inputs` and write them
to a [TinyBox][tinybox] instance specified at `target` where the keys
and values of the entries are "put" into the TinyBox storage that lives
on the file system. This behaviour can be configured by specifying
`options.storage` as a custom [random-access-storage][ras] _instance_.
The value of `options` is optional and can be:
```js
{
// if `false`, will not truncate archive storage
truncate: true,
// custom 'random-access-storage' compliant object
storage: require('random-access-file')(target)
}
```
#### Examples
##### Simple Archive
A simple example that archives the compiled objects of a compiled file.
```js
const { compile, archive } = require('tiny-module-compiler')
// compile this file
const target = __filename
compile(target, (err, objects) => {
archive(__filename + '.a', objects, (err) => {
// `target` is compiled and then archived
})
})
```
##### Simple Archive in Memory
A simple example that archives the compiled objects of a compiled file
into an in memory `random-access-memory` storage.
```js
const { compile, archive } = require('tiny-module-compiler')
const ram = require('random-access-memory')
// compile this file
const target = __filename
compile(target, { storage: ram }, (err) => {
const storage = ram()
archive(__filename + '.a', objects, { storage}, (err) => {
// `target` is compiled and then archived
// `storage` contains archived objects
})
})
```
### `load(target[, options], callback)`
Loads the exports from a module or the entries in archive specified at
`target`. The default behaviour is to load the contents of the file
specified at `target` as a compiled module and call `callback(err,
exports)` upon success or error with the module exports. If `target` is
a path to an archive represented by a [TinyBox][tinybox], then the
entries are loaded and `callback(err, archive)` is called upon success
or error. The storage of the compiled module or archive can be
explicitly set by specifying `options.storage` as a custom
[random-access-storage][ras] _instance_.
The value of `options` is optional and can be:
```js
{
// custom 'random-access-storage' compliant object
storage: require('random-access-file')(target),
// custom cache `Map` store for loaded modules
cache: new Map(),
// current working directory to load module in
cwd: process.cwd()
}
```
#### Example
##### Simple Compile and Load
A simple compilation and load example that compiles a target input file to an
output file and then loads the exports.
***file.js:***
```js
module.exports = {
hello() {
return 'world'
}
}
```
***compile-and-load.js:***
```js
const { compile, load } = require('tiny-module-compiler')
// compile this file
const target = 'file.js'
const output = target + '.out'
compile(target, { output }, (err) => {
// `target` compiled to `output`
load(output, (err, exports) => {
// `exports` points to `module.exports` in `output`
console.log(exports.hello()) // 'world'
})
})
```
##### Simple Compile and Load in Memory
A simple compilation and load example that compiles a target input file to an
in memory storage and then loads the exports from it.
***file.js:***
```js
module.exports = {
hello() {
return 'world'
}
}
```
***compile-and-load-in-memory.js:***
```js
const { compile, load } = require('tiny-module-compiler')
const ram = require('random-access-memory')
// compile this file
const target = 'file.js'
const storage = ram()
compile(target, { storage: () => storage }, (err) => {
// `target` compiled to `storage`
load('loaded from memory', { storage }, (err, exports) => {
// `exports` points to `module.exports` in `output`
console.log(exports.hello()) // 'world'
})
})
```
### `unpack(target[, options], callback)`
Unpacks `target` archive entries. The default behaviour is to enumerate
the entries in the archive specified at `target` and copy them to the
file system. This behaviour can be configured by specifying a `options.storage`
[random-access-storage][ras] _factory function_ to provide custom
storage for the archive entries. If `target` is a [random-access-storage][ras]
instance, it will be used instead of reading from the file system.
The value of `options` is optional and can be:
```js
{
// output path for default storage
output: process.cwd(),
// custom storage factory function to return
// 'random-access-storage' compliant object
storage(filename) {
return require('random-access-file')(filename)
}
}
```
#### Example
##### Simple Unpack
A simple example to unpack an archive's entries to file system.
```js
const { unpack } = require('tiny-module-compiler')
const archive = '/path/to/archive'
unpack(archive, (err, entries) => {
// `archive` entries unpacked to file system
console.log(entries) // array of unpacked file names
})
```
##### Simple Unpack in Memory
A simple example to unpack an archive's entries to an in memory file
store.
```js
const { unpack } = require('tiny-module-compiler')
const ram = require('random-access-memory'
const archive = '/path/to/archive'
const files = new Map()
unpack(archive, { storage: createStorage }, (err, entries) => {
// `archive` entries unpacked to file system
console.log(entries) // array of unpacked file names
for (const entry of entries) {
console.log(files[entry.filename])
}
})
function createStorage(filename) {
return files.set(filename, ram()).get(filename)
}
```
* Extends: [`nanoresource-pool`][nanoresource-pool]
The `Compiler` class represents a container of compile targets
that can be compiled into a single binary file containing
v8 cache data and header information about the compiled output.
#### Constructor: `new Compiler([options])`
Creates a new [`Compiler`](#api-class-compiler) instance where `options`
can be:
```js
{
// current working directory for compilation
cwd: process.cwd()
}
```
##### Example
```js
const compiler = new Compiler()
```
#### Accessor: `compiler.targets`
* `Array`
All opened targets in the compiler.
##### Example
```js
for (const target of compiler.targets) {
// handle opened `target`
}
```
#### `compiler.ready(callback)`
Waits for compiler to be ready and calls `callback()` upon success.
##### Example
```js
compiler.ready(() => {
// `compiler` is opened and ready
})
```
#### `compiler.target(filename[, options], callback)`
Creates and returns a new compile [target](#api-class-target) that is added
to compiler pool calling `callback(err, target)` when the target resource
is opened or an error occurs. The target will be compiled when
[`compiler.compiler()`](#api-class-compiler-compile) is called.
The value of `options` can be:
```js
{
// the default output for a compilation target
output: target + '.out',
// custom storage factory function to return
// 'random-access-storage' compliant object
storage(filename) {
return require('random-access-file')(filename)
}
}
```
##### Example
```js
compiler.target('/path/to/file.js', (err, target) => {
// `target` is an opened `nanoresource`
})
```
#### `compiler.compile([options], callback)`
Compiles all pending compile [targets](#api-class-compiler-target) calling
`callback(err, objects, assets)` upon success or error. Callback will be
given a `Map` of compiled objects and a `Map` of extracted assets that
should live with the compiled objects on the file system.
The value of `options` can be:
```js
{
// if `true`, will produce a source map stored in the `assets` map
map: false,
// if `false`, `ncc` will produce verbose output
quiet: true,
// if `true`, will use `ncc` cache for faster builds
cache: false,
// if `true`, will produce compiled javascript source debug output
debug: false,
// if `true`, will minify compiled javascript source before creating binary
optimize: false,
// an array of external modules that should _not_ be compiled
externals: [],
}
```
##### Example
```js
compiler.compile({ externals: ['sodium-native'] }, (err, objects, assets) => {
// `objects` is a `Map` mapping file names to compiled module objects
// `assets` is a `Map` mapping file names to assets that should be copied
})
```
* Extends: [`nanoresource`][nanoresource]
The `Archiver` class represents an abstraction for storing
compiled objects into a [TinyBox][tinybox]
#### Constructor: `new Archiver([options])`
Creates a new [`Archiver`](#api-class-archiver) instance where `options`
can be:
```js
{
// default custom storage factory function to return
// 'random-access-storage' compliant object used in
// `archiver.archive()` calls. This storage can be overloaded
// by supplying a storage factory function to `archiver.archive()`
storage(filename) {
return require('random-access-file')(filename)
}
}
```
##### Example
```js
const ram = require('random-access-memory')
// in memory archiver
const archiver = new Archiver({ storage: ram })
```
#### `archiver.ready(callback)`
Waits for archiver to be ready and calling `callback()` when it is.
##### Example
```js
archiver.ready(() => {
// `archiver` is opened and ready
})
```
#### `archiver.archive(filename, inputs[, options], callback)`]
Archives the entries found in a given `inputs` `Map` into `target`. The
underlying storage for `filename` can be given by `options.storage` or a
new one is created by the `storage` factory function given to the
[`Archiver`](#api-class-archiver-constructor) class constructor.
`callback(err)` is called upon success or error.
The value of `options` can be:
```js
{
// if `false`, will not truncate archive storage
truncate: true,
// custom 'random-access-storage' compliant object
// where `inputs` are archived to
storage: require('random-access-file')(filename)
}
```
##### Example
```js
const extend = require('map-extend')
compiler.compile((err, objects, assets) => {
// merge `objects` and `assets` and archive
archiver.archive('/path/to/archive', extend(objects, assets), (err) => {
// inputs should be archived
})
```
* Extends: [`nanoresource-pool`][nanoresource-pool]
The `Loader` class represents an abstraction for loading compiled
module objects and JavaScript sources as node modules.
#### Constructor: `new Loader([options])`
Creates a new [`Loader`](#api-class-loader) instance where `options`
can be:
```js
{
// custom cache `Map` store for loaded modules
cache: new Map()
}
```
Waits for loader to be ready and calling `callback()` when it is.
##### Example
```js
loader.ready(() => {
// `loader` is opened and ready
})
```
#### `loader.load(filename[, options], callback)`
Loads a compiled module object or JavaScript source module
specified at filename calling `callback(err, exports)` upon success
or error. Success loads will cache resulting module for subsequent
requests to load the module.
The value of `options` can be:
```js
{
// custom 'random-access-storage' compliant object
// where module is loaded from
storage: require('random-access-file')(filename)
}
```
##### Example
```js
loader.load('/path/to/compiled/module.js', (err, exports) => {
// `exports` points to `module.exports` of loaded module
})
```
* Extends: [`nanoresource`][nanoresource]
The `Target` class represents a [`nanoresource`][nanoresource] to a target
file backed by a [`random-access-storage`][ras] instance.
#### Constructor: `new Target(filename[, options])`
Creates a new [`Target`](#api-class-target) instance where `filename`
is the name of the target file and `options` can be:
```js
{
// custom 'random-access-storage' compliant object
// where file is loaded from
storage: require('random-access-file')(filename)
}
```
***Note:*** _This class is intended for internal and advanced use. You
will most likely not use this directly._
##### Example
```js
const target = new Target('/path/to/file.js')
```
* `?(Number)`
The active file descriptor for the target resource. Will be `null`
if not opened.
Queries for stats from the underlying target storage calling
`callback(err, stats)` upon success or error.
##### Example
```js
target.stat((err, stats) => {
console.log(stats.size)
})
```
#### `target.read(offset, size, callback)`
Reads data from the underlying target storage at a specified `offset` and
`size` calling `callback(err, buffer)` upon success or error.
##### Example
```js
taret.read(32, 64, (err, buffer) => {
console.log(buffer)
})
```
Waits for loader to be ready and calling `callback()` when it is.
##### Example
```js
target.ready(() => {
// `target` is opened and ready
})
```
The `tiny-module-compiler` exposes a command line interface through the
`tmc` command that is suitable for compiling, archiving, and unpacking
compiled modules and their assets.
This section describes the command line interface and a few workflows
for making the best use of the `tmc` command.
The `tmc` command has a command line signature of the following:
```sh
tmc [-hV] [-acu] [-vCDMO] [options] ...input
```
Where `options` can be:
```sh
-a, --archive If present, will archive input into "tinybox" format
-c, --compile If present, will compile input into header prefixed v8 cached data
, --concurrency An alias for '--jobs'
-C, --copy-assets If present, will copy assets to directory of output
-D, --debug If present, will enable debug output (DEBUG=tiny-module-compiler)
-e, --external Specifies an external dependency that will be linked at runtime
-h, --help If present, will print this message
-l, --load If present, will load inputs
-j, --jobs Specifies the number of concurrent jobs for batch tasks (--load, --archive, --copy-assets)
-M, --source-map If present, a source map will be generated
-o, --output If present, will change the output path. Assumes directory if multiple inputs given
-O, --optimize If present, will optimize output by minifying JavaScript source prior to compilation
-u, --unpack If present, will treat input as an archive and will unpack files to path specified by '--output'
-v, --verbose If present, will emit verbose output to stdout/stderr
-V, --version If present, will print the version number
-x An alias for '--external'
```
The simplest use of the `tmc` command is to compile a single file. By
default, the `-c` (or `--compile`) is assumed if `-a` (or
`--archive`) and `-u` (or `--unpack`) flags are **not present**.
```sh
$ tmc file.js ## will write ./file.js.out
```
The output of the compiled file can be configured using the `-o` (or `--output`)
flag to set the output name.
```sh
$ tmc file.js -o file.compiled.js
```
If multiple inputs were given, then the output
value will be a directory and the original file names are preserved.
```sh
$ tmc *.js -o build/
```
#### Archiving Compiled Modules
After compiling modules it may be useful to archive them into a single
file. This is often beneficial if the compiled module contained static
assets that need to live on the file system alongside the compiled
module file during runtime.
The `tmc` command makes it easy to specify a number of files that
should be added to an archive by making use of the `-a` (or `--archive`)
flag to indicate that inputs should be archived.
```sh
$ tmc -a modules.archive *.compiled.js ## archives all `*.compiled.js` files into `modules.archive`
```
Any asset can be added to an archive as well.
```sh
$ tmc -a modules.archive *.node ## archives all `*.node` files into `modules.archive`
```
The `modules.archive` file contains an index of entries added to it by
file name. The archive can be [unpacked](#cli-unpacking) using the `tmc`
command as well.
Compiled modules and their assets living in an archive can be easily
unpacked to the file system. The `tmc` command makes it easy to unpack
entries in an archive.
```sh
$ tmc -u modules.archive
```
The output of the entries can be specified with the `-o` (or `--output`)
flag.
```sh
$ tmc -u modules.archive -o build/
```
#### Loading Modules & Archives
Compiled modules and packed archives can be loaded at runtime in an
application. The `tmc` command provides a simple way to do this from the
command line from array of inputs by making use of the `-l` (or
`--load`) flag to indicate the inputs should be loaded.
```sh
$ tmc -l module.compiled.js
```
Archives can be loaded as well.
```sh
$ tmc -l modules.archive
```
Regular JavaScript files an be given as input.
```sh
$ tmc -l file.js
```
Multiple inputs can be loaded.
```sh
$ tmc -l modules.archive file.js module.compiled
```
The load concurrency can be set by specifying the `-j` (or `--jobs` or
`--concurrency`) flag to indicate the number of concurrent loads that
can occur when loading the inputs.
```sh
$ tmc -l -j1 modules.archive file.js module.compiled ## one at a time
```
The `tiny-module-compile` module ships with a few constrains due to the
nature of how v8 cache data works. They are described in this section.
The **v8 version** (`process.versions.v8`) in the runtime of `node` used
to compile modules to v8 cache data must match the one used to load. The
module will throw an error if the versions do not match as unexpected
behaviour may occur.
### `Function.prototype.toString()`
For user-defined functions,
[`Function.prototype.toString()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/toString)
may not work as expected or not at all because the string source code
representation of the function is lost after compilation.
## See Also
- [nanoresource-pool][nanoresource-pool]
- [nanoresource][nanoresource]
- [tinybox][tinybox]
- [@zeit/ncc][ncc]
- [glob][glob]
## Prior Art
- https://github.com/OsamaAbbas/bytenode
- https://github.com/zertosh/v8-compile-cache
- https://github.com/v8/v8/blob/master/src/snapshot/code-serializer.h
- https://hackernoon.com/how-to-compile-node-js-code-using-bytenode-11dcba856fa9
## License
MIT
[ras]: https://github.com/random-access-storage/random-access-storage
[ncc]: https://github.com/zeit/ncc
[glob]: https://github.com/isaacs/node-glob
[tinybox]: https://github.com/hyperdivision/tinybox
[nanoresource]: https://github.com/mafintosh/nanoresource
[nanoresource-pool]: https://github.com/little-core-labs/nanoresource-pool