{"id":16298097,"url":"https://github.com/equalma/jsonlines","last_synced_at":"2025-03-20T05:30:39.234Z","repository":{"id":38174385,"uuid":"281918100","full_name":"EqualMa/jsonlines","owner":"EqualMa","description":"stringify / parse jsonlines for NodeJS using streams","archived":false,"fork":false,"pushed_at":"2023-03-05T10:30:33.000Z","size":1140,"stargazers_count":3,"open_issues_count":20,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-15T16:07:00.388Z","etag":null,"topics":["json","jsonlines","nodejs","parse","stream","stringify"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EqualMa.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-07-23T10:12:30.000Z","updated_at":"2023-03-23T09:10:23.000Z","dependencies_parsed_at":"2024-02-08T06:44:47.756Z","dependency_job_id":null,"html_url":"https://github.com/EqualMa/jsonlines","commit_stats":{"total_commits":31,"total_committers":3,"mean_commits":"10.333333333333334","dds":"0.22580645161290325","last_synced_commit":"5e08ee6f7a08b955471d40bc4fc7ed18c46f207c"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EqualMa%2Fjsonlines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EqualMa%2Fjsonlines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EqualMa%2Fjsonlines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EqualMa%2Fjsonlines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EqualMa","download_url":"https://codeload.github.com/EqualMa/jsonlines/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244056345,"owners_count":20390719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","jsonlines","nodejs","parse","stream","stringify"],"created_at":"2024-10-10T20:43:33.165Z","updated_at":"2025-03-20T05:30:38.832Z","avatar_url":"https://github.com/EqualMa.png","language":"TypeScript","readme":"# jsonlines\n\n[![npm package @jsonlines/core](https://img.shields.io/npm/v/@jsonlines/core?style=flat-square)](http://npm.im/@jsonlines/core)\n[![GitHub package.json dependency version (dev dep on branch)](https://img.shields.io/github/package-json/dependency-version/EqualMa/jsonlines/dev/typescript?style=flat-square)]()\n[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg?style=flat-square)](https://github.com/semantic-release/semantic-release)\n\na Node.js library to parse, stringify [jsonlines](http://jsonlines.org/) files as streams\n\n## Install\n\n```shell\nnpm install @jsonlines/core\nyarn add @jsonlines/core\n```\n\n## Features Guide\n\n\u003cdetails\u003e\n\u003csummary\u003e\nEasy to use. parse stream and stringify stream are standard node duplex streams\n\u003c/summary\u003e\n\nstringify\n\n```js\nrequire(\"stream\")\n  .Readable.from([{ v: 1 }, { v: 2 }])\n  .pipe(require(\"@jsonlines/core\").stringify())\n  .pipe(require(\"fs\").createWriteStream(\"mydata.jsonl\"));\n```\n\nparse\n\n```js\nrequire(\"fs\")\n  .createReadStream(\"mydata.jsonl\")\n  .pipe(require(\"@jsonlines/core\").parse())\n  .on(\"data\", (data) =\u003e {\n    console.log(\"parsed data: \", data);\n  });\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\nCustom stringify / parse functions. Async function or function that returns a Promise is also supported.\n\u003c/summary\u003e\n\n```js\nrequire(\"stream\")\n  .Readable.from([{ v: 1 }, { v: 2 }])\n  .pipe(\n    require(\"@jsonlines/core\").stringify({\n      stringify: myCustomStringifyFunction,\n    }),\n  )\n  .pipe(require(\"fs\").createWriteStream(\"mydata.jsonl\"));\n```\n\n```js\nrequire(\"fs\")\n  .createReadStream(\"mydata.jsonl\")\n  .pipe(\n    require(\"@jsonlines/core\").parse({\n      parse: myCustomParseFunction,\n    }),\n  )\n  .on(\"data\", (data) =\u003e {\n    console.log(\"receive data: \", data);\n  });\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\nGzip / Gunzip\n\u003c/summary\u003e\n\nstringify to a `.jsonl.gz`\n\n```js\nrequire(\"stream\")\n  .Readable.from([{ v: 1 }, { v: 2 }])\n  .pipe(\n    require(\"@jsonlines/core\").stringify({\n      gzip: true,\n    }),\n  )\n  .pipe(require(\"fs\").createWriteStream(\"mydata.jsonl.gz\"));\n```\n\nparse from a `.jsonl.gz`\n\n```js\nrequire(\"fs\")\n  .createReadStream(\"mydata.jsonl.gz\")\n  .pipe(\n    require(\"@jsonlines/core\").parse({\n      gzip: true,\n    }),\n  )\n  .on(\"data\", (data) =\u003e {\n    console.log(\"receive data: \", data);\n  });\n```\n\n\u003c/details\u003e\n\n## Usage\n\n### stringify\n\n```js\nconst { stringify } = require(\"@jsonlines/core\");\n// or import from sub-module\nconst { stringify } = require(\"@jsonlines/core/stringify\");\n\n// or import with es module\nimport { stringify } from \"@jsonlines/core\";\nimport { stringify } from \"@jsonlines/core/stringify\";\n\nrequire(\"stream\")\n  .Readable.from([\n    // objects\n    { v: \"object1\" },\n    { name: \"Lady Gaga\", records: [\"Chromatica\"] },\n    // arrays\n    [1, 2, 3, 4],\n    // booleans\n    true,\n    false,\n    // numbers\n    2020,\n    -1,\n    // null\n    // Note that single null value can't be written to node streams,\n    // so @jsonlines/core provides a helper value to represent null\n    require(\"@jsonlines/core\").nullValue,\n    require(\"@jsonlines/core/null-value\").nullValue,\n    // note that it not necessary to use this helper value when null is in an array or object\n    { value: null },\n    [null, null],\n  ])\n  .pipe(\n    // create a stringify stream, which is a duplex stream\n    stringify(),\n  )\n  .pipe(process.stdout);\n```\n\nthe output will be:\n\n```jsonlines\n{\"v\":\"object1\"}\n{\"name\":\"Lady Gaga\",\"records\":[\"Chromatica\"]}\n[1,2,3,4]\ntrue\nfalse\n2020\n-1\nnull\nnull\n{\"value\":null}\n[null,null]\n```\n\n#### `stringify` API\n\n```ts\n// prettier-ignore\nfunction stringify(options?: JsonLinesStringifyOptions): JsonLinesStringifyStream;\n```\n\n`stringify` function accepts an optional object as options and returns an instance of `JsonLinesStringifyStream`.\n\nNote that `JsonLinesStringifyStream` extends `Duplex`.\n\noptions:\n\n```ts\nexport interface JsonLinesStringifyOptions\u003cV\u003e {\n  /**\n   * specify the encoding to encode string to buffer\n   *\n   * NOTE that [the standard jsonlines](http://jsonlines.org/)\n   * requires `utf8` as file encoding\n   *\n   * Defaults to `Buffer.from` default encoding,\n   * which is `utf8`.\n   */\n  encoding?: BufferEncoding;\n\n  /**\n   * specify a function to stringify values.\n   * It accepts a value as parameter,\n   * and should return a string or a Promise\u003cstring\u003e.\n   *\n   * Defaults to `JSON.stringify`\n   */\n  stringify?: (v: V) =\u003e string | Promise\u003cstring\u003e;\n\n  /**\n   * specify whether to gzip the output\n   *\n   * Omit or use `false` to disable gzip.\n   * Use `true` to gzip with default options.\n   * Or use an object as params for `require('zlib').createGzip`\n   */\n  gzip?: JsonLinesGzipOption;\n\n  /**\n   * specify the line ending to be used in the output\n   *\n   * NOTE that [the standard jsonlines](http://jsonlines.org/)\n   * requires `\\n` as line separator\n   *\n   * Defaults to `\\n`\n   */\n  lineSep?: \"lf\" | \"\\n\" | \"crlf\" | \"\\r\\n\";\n}\n```\n\n### parse\n\n```js\nconst { parse } = require(\"@jsonlines/core\");\n// or import from sub-module\nconst { parse } = require(\"@jsonlines/core/parse\");\n\n// or import with es module\nimport { parse } from \"@jsonlines/core\";\nimport { parse } from \"@jsonlines/core/parse\";\n\nconst source = require(\"stream\").Readable.from(`{\"v\":\"object1\"}\n{\"name\":\"Lady Gaga\",\"records\":[\"Chromatica\"]}\n[1,2,3,4]\ntrue\nfalse\n2020\n-1\nnull\nnull\n{\"value\":null}\n[null,null]\n`);\n\n// create a parse stream, which is a duplex stream\nconst parseStream = parse();\n\nsource.pipe(parseStream);\n\n// you can also consume it with for await ... of\nparseStream.on(\"data\", (value) =\u003e {\n  if (value === require(\"@jsonlines/core/null-value\").nullValue)\n    console.log(`--- The following value is nullValue ---`);\n\n  console.log(value);\n});\n```\n\nthe output will be:\n\n```\n{ v: 'object1' }\n{ name: 'Lady Gaga', records: [ 'Chromatica' ] }\n[ 1, 2, 3, 4 ]\ntrue\nfalse\n2020\n-1\n--- The following value is nullValue ---\n[Object: null prototype] [Null] {}\n--- The following value is nullValue ---\n[Object: null prototype] [Null] {}\n{ value: null }\n[ null, null ]\n```\n\n#### `parse` API\n\n```ts\nfunction parse(options?: JsonLinesParseOptions\u003cV\u003e): JsonLinesParseStream;\n```\n\n`parse` function accepts an optional object as options\nand returns an instance of `JsonLinesParseStream`.\n\nNote that `JsonLinesParseStream` extends `Duplex`.\n\noptions:\n\n```ts\nexport interface JsonLinesParseOptions\u003cV\u003e {\n  /**\n   * specify the encoding to decode buffer to string\n   *\n   * NOTE that [the standard jsonlines](http://jsonlines.org/)\n   * requires `utf8` as file encoding\n   *\n   * Defaults to `utf8`\n   */\n  encoding?: BufferEncoding;\n\n  /**\n   * specify a function to parse json line.\n   * It accepts a string as parameter,\n   * and should return a value or a Promise\u003cV\u003e.\n   *\n   * Defaults to `JSON.parse`\n   */\n  parse?: (line: string) =\u003e V | Promise\u003cV\u003e;\n\n  /**\n   * specify whether to gunzip the source\n   *\n   * Omit or use `false` to disable gunzip.\n   * Use `true` to gunzip with default options.\n   * Or use an object as params for `require('zlib').createGunzip`\n   */\n  gzip?: JsonLinesGzipOption;\n\n  /**\n   * specify the line ending to be parsed\n   *\n   * NOTE that [the standard jsonlines](http://jsonlines.org/)\n   * requires `\\n` as line separator\n   *\n   * If set to `auto`, both `\\n` and `\\r\\n` will be accepted\n   *\n   * Defaults to `\\n`\n   */\n  lineSep?: \"lf\" | \"\\n\" | \"crlf\" | \"\\r\\n\" | \"auto\";\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalma%2Fjsonlines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fequalma%2Fjsonlines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalma%2Fjsonlines/lists"}