Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pinojs/thread-stream
A streaming way to send data to a Node.js Worker Thread
https://github.com/pinojs/thread-stream
Last synced: about 2 months ago
JSON representation
A streaming way to send data to a Node.js Worker Thread
- Host: GitHub
- URL: https://github.com/pinojs/thread-stream
- Owner: pinojs
- License: mit
- Created: 2021-03-05T10:51:02.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-22T12:23:18.000Z (5 months ago)
- Last Synced: 2024-04-22T12:25:14.778Z (5 months ago)
- Language: JavaScript
- Size: 157 KB
- Stars: 203
- Watchers: 12
- Forks: 21
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# thread-stream
[](https://www.npmjs.com/package/thread-stream)
[](https://github.com/pinojs/thread-stream/actions)
[](https://standardjs.com/)
A streaming way to send data to a Node.js Worker Thread.
## install
```sh
npm i thread-stream
```
## Usage
```js
'use strict'
const ThreadStream = require('thread-stream')
const { join } = require('path')
const stream = new ThreadStream({
filename: join(__dirname, 'worker.js'),
workerData: { dest },
workerOpts: {}, // Other options to be passed to Worker
sync: false, // default
})
stream.write('hello')
// Asynchronous flushing
stream.flush(function () {
stream.write(' ')
stream.write('world')
// Synchronous flushing
stream.flushSync()
stream.end()
})
```
In `worker.js`:
```js
'use strict'
const fs = require('fs')
const { once } = require('events')
async function run (opts) {
const stream = fs.createWriteStream(opts.dest)
await once(stream, 'open')
return stream
}
module.exports = run
```
Make sure that the stream emits `'close'` when the stream completes.
This can usually be achieved by passing the [`autoDestroy: true`](https://nodejs.org/api/stream.html#stream_new_stream_writable_options)
flag your stream classes.
The underlining worker is automatically closed if the stream is garbage collected.
### External modules
You may use this module within compatible external modules, that exports the `worker.js` interface.
```js
const ThreadStream = require('thread-stream')
const modulePath = require.resolve('pino-elasticsearch')
const stream = new ThreadStream({
filename: modulePath,
workerData: { node: 'http://localhost:9200' }
})
stream.write('log to elasticsearch!')
stream.flushSync()
stream.end()
```
This module works with `yarn` in PnP (plug'n play) mode too!
### Emit events
You can emit events on the ThreadStream from your worker using [`worker.parentPort.postMessage()`](https://nodejs.org/api/worker_threads.html#workerparentport).
The message (JSON object) must have the following data structure:
```js
parentPort.postMessage({
code: 'EVENT',
name: 'eventName',
args: ['list', 'of', 'args', 123, new Error('Boom')]
})
```
On your ThreadStream, you can add a listener function for this event name:
```js
const stream = new ThreadStream({
filename: join(__dirname, 'worker.js'),
workerData: {},
})
stream.on('eventName', function (a, b, c, n, err) {
console.log('received:', a, b, c, n, err) // received: list of args 123 Error: Boom
})
```
### Post Messages
You can post messages to the worker by emitting a `message` event on the ThreadStream.
```js
const stream = new ThreadStream({
filename: join(__dirname, 'worker.js'),
workerData: {},
})
stream.emit('message', message)
```
On your worker, you can listen for this message using [`worker.parentPort.on('message', cb)`](https://nodejs.org/api/worker_threads.html#event-message).
```js
const { parentPort } = require('worker_threads')
parentPort.on('message', function (message) {
console.log('received:', message)
})
```
## License
MIT