{"id":25883145,"url":"https://github.com/maxlath/ndjson-apply","last_synced_at":"2025-03-02T16:31:37.744Z","repository":{"id":57309412,"uuid":"234432126","full_name":"maxlath/ndjson-apply","owner":"maxlath","description":"apply a JS function to a stream of newline-delimited JSON","archived":false,"fork":false,"pushed_at":"2024-12-20T15:12:45.000Z","size":290,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-20T16:03:33.025Z","etag":null,"topics":["apply-function","cli","diff","ndjson","transformation"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxlath.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"liberapay":"Association_Inventaire"}},"created_at":"2020-01-16T23:33:08.000Z","updated_at":"2024-12-20T15:12:49.000Z","dependencies_parsed_at":"2024-08-11T18:39:56.636Z","dependency_job_id":null,"html_url":"https://github.com/maxlath/ndjson-apply","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fndjson-apply","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fndjson-apply/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fndjson-apply/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlath%2Fndjson-apply/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxlath","download_url":"https://codeload.github.com/maxlath/ndjson-apply/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241537363,"owners_count":19978521,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apply-function","cli","diff","ndjson","transformation"],"created_at":"2025-03-02T16:31:37.225Z","updated_at":"2025-03-02T16:31:37.733Z","avatar_url":"https://github.com/maxlath.png","language":"JavaScript","funding_links":["https://liberapay.com/Association_Inventaire"],"categories":[],"sub_categories":[],"readme":"# ndjson-apply\nA CLI tool to transform a stream of newline-delimited JSON by applying a JS function to each JSON object.\n\nFeatures:\n* take the JS function to apply from a file\n* the function may return async results\n* preview the transformation results with the `--diff` option\n\n[![NPM](https://nodei.co/npm/ndjson-apply.png?stars\u0026downloads\u0026downloadRank)](https://npmjs.com/package/ndjson-apply/)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n[![Node](https://img.shields.io/badge/node-%3E=%20v7.6.0-brightgreen.svg)](http://nodejs.org)\n\n\n## Summary\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n- [Install](#install)\n- [How To](#how-to)\n  - [Basic](#basic)\n  - [Async](#async)\n  - [Diff mode](#diff-mode)\n  - [Filter mode](#filter-mode)\n  - [Use sub-function](#use-sub-function)\n  - [Pass additional arguments](#pass-additional-arguments)\n  - [`after` hook](#after-hook)\n  - [Typescript support](#typescript-support)\n- [See also](#see-also)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n\n## Install\n```sh\nnpm i -g ndjson-apply\n```\n\n## How To\n\n### Basic\n```sh\ncat some_data.ndjson | ndjson-apply some_transform_function.js \u003e some_data_transformed.ndjson\n# Which can also be written\nndjson-apply some_transform_function.js \u003c cat some_data.ndjson \u003e some_data_transformed.ndjson\n```\nwhere `some_transform_function.js` just needs to export a JS function. This should work both with the ESM export syntax\n```js\n// some_transform_function.js\nexport default function (doc) {\n  doc.total = doc.a + doc.b\n  if (doc.total % 2 === 0) {\n    return doc\n  } else {\n    // returning null or undefined drops the entry\n  }\n}\n```\nor with the CommonJS export syntax\n```js\n// some_transform_function.js\nmodule.exports = function (doc) {\n  doc.total = doc.a + doc.b\n  if (doc.total % 2 === 0) {\n    return doc\n  } else {\n    // returning null or undefined drops the entry\n  }\n}\n```\n\n### Async\nThat function can also be async:\n```js\nimport { getSomeExtraData } from './path/to/get_some_extra_data.js'\n\n// some_async_transform_function.js\nexport default async function (doc) {\n  doc.total = doc.a + doc.b\n  if (doc.total % 2 === 0) {\n    doc.extraData = await getSomeExtraData(doc)\n    return doc\n  } else {\n    // returning null or undefined drops the entry\n  }\n}\n```\n\n### Diff mode\nAs a way to preview the results of your transformation, you can use the diff mode\n```sh\ncat some_data.ndjson | ndjson-apply some_transform_function.js --diff\n```\nwhich will display a colored diff of each line before and after transformation.\n\nFor more readability, each line diff output is indented and on several lines.\n\n### Filter mode\nUse the js function only to filter lines: lines returning `true` will be let through. No transformation will be applied.\n```sh\ncat some_data.ndjson | ndjson-apply some_transform_function.js --filter\n```\n\n### Use sub-function\nGiven a `function_collection.js` file like:\n```js\nexport function foo (obj) {\n  obj.timestamp = Date.now()\n  return obj\n}\n\nexport function bar (obj) {\n  obj.count += obj.count\n  return obj\n}\n```\n\nYou can use those subfunction by passing their key as an additional argument\n```sh\ncat some_data.ndjson | ndjson-apply ./function_collection.js foo\ncat some_data.ndjson | ndjson-apply ./function_collection.js bar\n```\n\nThis should also work with the CommonJS syntax:\n```js\n// function_collection.cjs\nmodule.exports = {\n  foo: (obj) =\u003e {\n    obj.timestamp = Date.now()\n    return obj\n  },\n  bar: (obj) =\u003e {\n    obj.count += obj.count\n    return obj\n  }\n}\n```\n\n### Pass additional arguments\nAny remaining argument will be passed to the function\n```sh\n# Pass '123' as argument to the exported function\ncat some_data.ndjson | ndjson-apply ./function.js 123\n# Pass '123' as argument to the exported sub-function foo\ncat some_data.ndjson | ndjson-apply ./function_collection.js foo 123\n```\n\n### `after` hook\n\nThis allows, for instance, to implement [`reduce` functions](https://en.wikipedia.org/wiki/Fold_(higher-order_function)), or any kind of side effects that needs to be performed once all lines have been processed.\n\nGiven a `aggregate.js` file like:\n```js\nlet sum = 0\n\nexport function addA (doc) {\n  sum += doc.a\n}\n\nexport function outputSum () {\n  return sum\n}\n```\n\n```js\necho '\n{ \"a\": 1, \"b\": 8 }\n{ \"a\": 3, \"b\": 6 }\n' | ndjson-apply ./aggregate.js addA --after outputSum\n// =\u003e 4\n```\n\n### Typescript support\nTo use `ndjson-apply` with `.ts` files, you can execute it with [`tsx`](https://github.com/privatenumber/tsx) as follow:\n```sh\n# Get a tsx executable\nnpm install --global tsx\n# Use ndjson-apply-ts just like you would use ndjson-apply\nndjson-apply-ts ./some_transform_function.ts \u003c ./tests/assets/sample.ndjson\n```\n\n## See also\n* [jq](https://stedolan.github.io/jq/) is great to work with NDJSON: `cat entries_array.json | jq '.[]' -cr \u003e entries.ndjson`\n* [ndjson-cli#map](https://github.com/mbostock/ndjson-cli#map)\n* [json-apply](https://github.com/maxlath/json-apply/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlath%2Fndjson-apply","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxlath%2Fndjson-apply","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlath%2Fndjson-apply/lists"}