https://github.com/equalma/jsonlines
stringify / parse jsonlines for NodeJS using streams
https://github.com/equalma/jsonlines
json jsonlines nodejs parse stream stringify
Last synced: 7 months ago
JSON representation
stringify / parse jsonlines for NodeJS using streams
- Host: GitHub
- URL: https://github.com/equalma/jsonlines
- Owner: EqualMa
- License: mit
- Created: 2020-07-23T10:12:30.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-03-05T10:30:33.000Z (over 2 years ago)
- Last Synced: 2025-03-15T16:07:00.388Z (7 months ago)
- Topics: json, jsonlines, nodejs, parse, stream, stringify
- Language: TypeScript
- Homepage:
- Size: 1.09 MB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# jsonlines
[](http://npm.im/@jsonlines/core)
[]()
[](https://github.com/semantic-release/semantic-release)
a Node.js library to parse, stringify [jsonlines](http://jsonlines.org/) files as streams
## Install
```shell
npm install @jsonlines/core
yarn add @jsonlines/core
```
## Features Guide
Easy to use. parse stream and stringify stream are standard node duplex streams
stringify
```js
require("stream")
.Readable.from([{ v: 1 }, { v: 2 }])
.pipe(require("@jsonlines/core").stringify())
.pipe(require("fs").createWriteStream("mydata.jsonl"));
```
parse
```js
require("fs")
.createReadStream("mydata.jsonl")
.pipe(require("@jsonlines/core").parse())
.on("data", (data) => {
console.log("parsed data: ", data);
});
```
Custom stringify / parse functions. Async function or function that returns a Promise is also supported.
```js
require("stream")
.Readable.from([{ v: 1 }, { v: 2 }])
.pipe(
require("@jsonlines/core").stringify({
stringify: myCustomStringifyFunction,
}),
)
.pipe(require("fs").createWriteStream("mydata.jsonl"));
```
```js
require("fs")
.createReadStream("mydata.jsonl")
.pipe(
require("@jsonlines/core").parse({
parse: myCustomParseFunction,
}),
)
.on("data", (data) => {
console.log("receive data: ", data);
});
```
Gzip / Gunzip
stringify to a `.jsonl.gz`
```js
require("stream")
.Readable.from([{ v: 1 }, { v: 2 }])
.pipe(
require("@jsonlines/core").stringify({
gzip: true,
}),
)
.pipe(require("fs").createWriteStream("mydata.jsonl.gz"));
```
parse from a `.jsonl.gz`
```js
require("fs")
.createReadStream("mydata.jsonl.gz")
.pipe(
require("@jsonlines/core").parse({
gzip: true,
}),
)
.on("data", (data) => {
console.log("receive data: ", data);
});
```
## Usage
### stringify
```js
const { stringify } = require("@jsonlines/core");
// or import from sub-module
const { stringify } = require("@jsonlines/core/stringify");
// or import with es module
import { stringify } from "@jsonlines/core";
import { stringify } from "@jsonlines/core/stringify";
require("stream")
.Readable.from([
// objects
{ v: "object1" },
{ name: "Lady Gaga", records: ["Chromatica"] },
// arrays
[1, 2, 3, 4],
// booleans
true,
false,
// numbers
2020,
-1,
// null
// Note that single null value can't be written to node streams,
// so @jsonlines/core provides a helper value to represent null
require("@jsonlines/core").nullValue,
require("@jsonlines/core/null-value").nullValue,
// note that it not necessary to use this helper value when null is in an array or object
{ value: null },
[null, null],
])
.pipe(
// create a stringify stream, which is a duplex stream
stringify(),
)
.pipe(process.stdout);
```
the output will be:
```jsonlines
{"v":"object1"}
{"name":"Lady Gaga","records":["Chromatica"]}
[1,2,3,4]
true
false
2020
-1
null
null
{"value":null}
[null,null]
```
#### `stringify` API
```ts
// prettier-ignore
function stringify(options?: JsonLinesStringifyOptions): JsonLinesStringifyStream;
```
`stringify` function accepts an optional object as options and returns an instance of `JsonLinesStringifyStream`.
Note that `JsonLinesStringifyStream` extends `Duplex`.
options:
```ts
export interface JsonLinesStringifyOptions {
/**
* specify the encoding to encode string to buffer
*
* NOTE that [the standard jsonlines](http://jsonlines.org/)
* requires `utf8` as file encoding
*
* Defaults to `Buffer.from` default encoding,
* which is `utf8`.
*/
encoding?: BufferEncoding;
/**
* specify a function to stringify values.
* It accepts a value as parameter,
* and should return a string or a Promise.
*
* Defaults to `JSON.stringify`
*/
stringify?: (v: V) => string | Promise;
/**
* specify whether to gzip the output
*
* Omit or use `false` to disable gzip.
* Use `true` to gzip with default options.
* Or use an object as params for `require('zlib').createGzip`
*/
gzip?: JsonLinesGzipOption;
/**
* specify the line ending to be used in the output
*
* NOTE that [the standard jsonlines](http://jsonlines.org/)
* requires `\n` as line separator
*
* Defaults to `\n`
*/
lineSep?: "lf" | "\n" | "crlf" | "\r\n";
}
```
### parse
```js
const { parse } = require("@jsonlines/core");
// or import from sub-module
const { parse } = require("@jsonlines/core/parse");
// or import with es module
import { parse } from "@jsonlines/core";
import { parse } from "@jsonlines/core/parse";
const source = require("stream").Readable.from(`{"v":"object1"}
{"name":"Lady Gaga","records":["Chromatica"]}
[1,2,3,4]
true
false
2020
-1
null
null
{"value":null}
[null,null]
`);
// create a parse stream, which is a duplex stream
const parseStream = parse();
source.pipe(parseStream);
// you can also consume it with for await ... of
parseStream.on("data", (value) => {
if (value === require("@jsonlines/core/null-value").nullValue)
console.log(`--- The following value is nullValue ---`);
console.log(value);
});
```
the output will be:
```
{ v: 'object1' }
{ name: 'Lady Gaga', records: [ 'Chromatica' ] }
[ 1, 2, 3, 4 ]
true
false
2020
-1
--- The following value is nullValue ---
[Object: null prototype] [Null] {}
--- The following value is nullValue ---
[Object: null prototype] [Null] {}
{ value: null }
[ null, null ]
```
#### `parse` API
```ts
function parse(options?: JsonLinesParseOptions): JsonLinesParseStream;
```
`parse` function accepts an optional object as options
and returns an instance of `JsonLinesParseStream`.
Note that `JsonLinesParseStream` extends `Duplex`.
options:
```ts
export interface JsonLinesParseOptions {
/**
* specify the encoding to decode buffer to string
*
* NOTE that [the standard jsonlines](http://jsonlines.org/)
* requires `utf8` as file encoding
*
* Defaults to `utf8`
*/
encoding?: BufferEncoding;
/**
* specify a function to parse json line.
* It accepts a string as parameter,
* and should return a value or a Promise.
*
* Defaults to `JSON.parse`
*/
parse?: (line: string) => V | Promise;
/**
* specify whether to gunzip the source
*
* Omit or use `false` to disable gunzip.
* Use `true` to gunzip with default options.
* Or use an object as params for `require('zlib').createGunzip`
*/
gzip?: JsonLinesGzipOption;
/**
* specify the line ending to be parsed
*
* NOTE that [the standard jsonlines](http://jsonlines.org/)
* requires `\n` as line separator
*
* If set to `auto`, both `\n` and `\r\n` will be accepted
*
* Defaults to `\n`
*/
lineSep?: "lf" | "\n" | "crlf" | "\r\n" | "auto";
}
```