https://github.com/maxlath/ndjson-apply
apply a JS function to a stream of newline-delimited JSON
https://github.com/maxlath/ndjson-apply
apply-function cli diff ndjson transformation
Last synced: about 2 months ago
JSON representation
apply a JS function to a stream of newline-delimited JSON
- Host: GitHub
- URL: https://github.com/maxlath/ndjson-apply
- Owner: maxlath
- Created: 2020-01-16T23:33:08.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-12-20T15:12:45.000Z (4 months ago)
- Last Synced: 2025-02-20T16:03:33.025Z (2 months ago)
- Topics: apply-function, cli, diff, ndjson, transformation
- Language: JavaScript
- Homepage:
- Size: 283 KB
- Stars: 2
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
Awesome Lists containing this project
README
# ndjson-apply
A CLI tool to transform a stream of newline-delimited JSON by applying a JS function to each JSON object.
Features:
* take the JS function to apply from a file
* the function may return async results
* preview the transformation results with the `--diff` option
[](https://npmjs.com/package/ndjson-apply/)
[](https://opensource.org/licenses/MIT)
[](http://nodejs.org)
## Summary
- [Install](#install)
- [How To](#how-to)
- [Basic](#basic)
- [Async](#async)
- [Diff mode](#diff-mode)
- [Filter mode](#filter-mode)
- [Use sub-function](#use-sub-function)
- [Pass additional arguments](#pass-additional-arguments)
- [`after` hook](#after-hook)
- [Typescript support](#typescript-support)
- [See also](#see-also)
## Install
```sh
npm i -g ndjson-apply
```
## How To
### Basic
```sh
cat some_data.ndjson | ndjson-apply some_transform_function.js > some_data_transformed.ndjson
# Which can also be written
ndjson-apply some_transform_function.js < cat some_data.ndjson > some_data_transformed.ndjson
```
where `some_transform_function.js` just needs to export a JS function. This should work both with the ESM export syntax
```js
// some_transform_function.js
export default function (doc) {
doc.total = doc.a + doc.b
if (doc.total % 2 === 0) {
return doc
} else {
// returning null or undefined drops the entry
}
}
```
or with the CommonJS export syntax
```js
// some_transform_function.js
module.exports = function (doc) {
doc.total = doc.a + doc.b
if (doc.total % 2 === 0) {
return doc
} else {
// returning null or undefined drops the entry
}
}
```
### Async
That function can also be async:
```js
import { getSomeExtraData } from './path/to/get_some_extra_data.js'
// some_async_transform_function.js
export default async function (doc) {
doc.total = doc.a + doc.b
if (doc.total % 2 === 0) {
doc.extraData = await getSomeExtraData(doc)
return doc
} else {
// returning null or undefined drops the entry
}
}
```
### Diff mode
As a way to preview the results of your transformation, you can use the diff mode
```sh
cat some_data.ndjson | ndjson-apply some_transform_function.js --diff
```
which will display a colored diff of each line before and after transformation.
For more readability, each line diff output is indented and on several lines.
### Filter mode
Use the js function only to filter lines: lines returning `true` will be let through. No transformation will be applied.
```sh
cat some_data.ndjson | ndjson-apply some_transform_function.js --filter
```
### Use sub-function
Given a `function_collection.js` file like:
```js
export function foo (obj) {
obj.timestamp = Date.now()
return obj
}
export function bar (obj) {
obj.count += obj.count
return obj
}
```
You can use those subfunction by passing their key as an additional argument
```sh
cat some_data.ndjson | ndjson-apply ./function_collection.js foo
cat some_data.ndjson | ndjson-apply ./function_collection.js bar
```
This should also work with the CommonJS syntax:
```js
// function_collection.cjs
module.exports = {
foo: (obj) => {
obj.timestamp = Date.now()
return obj
},
bar: (obj) => {
obj.count += obj.count
return obj
}
}
```
### Pass additional arguments
Any remaining argument will be passed to the function
```sh
# Pass '123' as argument to the exported function
cat some_data.ndjson | ndjson-apply ./function.js 123
# Pass '123' as argument to the exported sub-function foo
cat some_data.ndjson | ndjson-apply ./function_collection.js foo 123
```
### `after` hook
This allows, for instance, to implement [`reduce` functions](https://en.wikipedia.org/wiki/Fold_(higher-order_function)), or any kind of side effects that needs to be performed once all lines have been processed.
Given a `aggregate.js` file like:
```js
let sum = 0
export function addA (doc) {
sum += doc.a
}
export function outputSum () {
return sum
}
```
```js
echo '
{ "a": 1, "b": 8 }
{ "a": 3, "b": 6 }
' | ndjson-apply ./aggregate.js addA --after outputSum
// => 4
```
### Typescript support
To use `ndjson-apply` with `.ts` files, you can execute it with [`tsx`](https://github.com/privatenumber/tsx) as follow:
```sh
# Get a tsx executable
npm install --global tsx
# Use ndjson-apply-ts just like you would use ndjson-apply
ndjson-apply-ts ./some_transform_function.ts < ./tests/assets/sample.ndjson
```
## See also
* [jq](https://stedolan.github.io/jq/) is great to work with NDJSON: `cat entries_array.json | jq '.[]' -cr > entries.ndjson`
* [ndjson-cli#map](https://github.com/mbostock/ndjson-cli#map)
* [json-apply](https://github.com/maxlath/json-apply/)