Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/rpidanny/streamline.js
A JavaScript class that reads and processes a stream line-by-line in order.
https://github.com/rpidanny/streamline.js
big-data data data-processing file-stream javascript stream streams typescript
Last synced: about 6 hours ago
JSON representation
A JavaScript class that reads and processes a stream line-by-line in order.
- Host: GitHub
- URL: https://github.com/rpidanny/streamline.js
- Owner: rpidanny
- Created: 2020-10-28T18:39:15.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2023-02-20T09:06:31.000Z (almost 2 years ago)
- Last Synced: 2024-12-29T14:29:10.822Z (5 days ago)
- Topics: big-data, data, data-processing, file-stream, javascript, stream, streams, typescript
- Language: TypeScript
- Homepage:
- Size: 392 KB
- Stars: 3
- Watchers: 2
- Forks: 3
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# streamline.js
[](https://codecov.io/gh/rpidanny/streamline.js)
A JavaScript package that helps to reads and processes a stream line-by-line in order or in batches in parallel.
## Install
```sh
$ npm install --save @rpidanny/streamline.js
```
## Usage
### Functions
#### processLine(readStream, handler, concurrency)
| Parameters | Default | Description |
| ------------- | ------- | ---------------------------------------------------------------------------------------------- |
| `readStream` | `None` | Any Node.js `ReadStream` object |
| `handler` | `None` | A processing function that is called on every line of the stream with the line as the argument |
| `concurrency` | `1` | The number of concurrent execution of the handler |
#### processJson(readStream, handler, concurrency)
| Parameters | Default | Description |
| ------------- | ------- | -------------------------------------------------------------------------------------------------------------- |
| `readStream` | `None` | Any Node.js `ReadStream` object |
| `handler` | `None` | A processing function that is called on every line of the stream with the parsed `JSON` object as the argument |
| `concurrency` | `1` | The number of concurrent execution of the handler |
#### processCsv(readStream, handler, concurrency)
| Parameters | Default | Description |
| ------------- | ------- | ---------------------------------------------------------------------------------------------------- |
| `readStream` | `None` | Any Node.js `ReadStream` object |
| `handler` | `None` | A processing function that is called on every line of the stream with the parsed CSV as the argument |
| `concurrency` | `1` | The number of concurrent execution of the handler |
#### Example
```js
const readStream = fs.createReadStream(`jsonlFile.json`)
await processJson(
readStream,
async (item: Record) => {
console.log(item)
},
2,
)
```
### Classes
#### Base classes
* `Streamline`
* `StreamlineCsv`
* `StreamlineJson`
These are abstract classes that can be inherited to create complex classes to process read streams.
#### Forked classes
These classes are derived from `Base classes` which is suitable for some common use cases.
* `SimpleCsvProcessor`
* `SimpleJsonProcessor`
* `SimpleCsvProcessor`