https://github.com/winebarrel/jluniq
Remove duplicate key member lines from sorted NDJSON.
https://github.com/winebarrel/jluniq
json jsonlines ndjson rust
Last synced: about 1 month ago
JSON representation
Remove duplicate key member lines from sorted NDJSON.
- Host: GitHub
- URL: https://github.com/winebarrel/jluniq
- Owner: winebarrel
- License: mit
- Created: 2021-08-29T13:41:00.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-08-30T12:32:07.000Z (over 3 years ago)
- Last Synced: 2024-10-19T04:31:25.478Z (6 months ago)
- Topics: json, jsonlines, ndjson, rust
- Language: Rust
- Homepage:
- Size: 11.7 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# jluniq
Remove duplicate key member lines from sorted [NDJSON](http://ndjson.org/).
**Important: NDJSON file must be sorted by the value of the uniq key member. (e.g. [jlsort](https://github.com/winebarrel/jlsort))**
[](https://github.com/winebarrel/jluniq/actions)
## Installation
```
brew install winebarrel/jl/jluniq
```
## Usage
```
Usage: jluniq [OPTIONS] [FILE]
Options:
-k, --key KEY JSON key to make it unique
-g, --group Group rows with the same value
-c, --count Count lines
-v, --version Print version and exit
-h, --help Print usage and exit
```
```
% cat data.ndjson
{"id": 1, "sub_id": 11, "val":"abc"}
{"id": 2, "sub_id": 21, "val":"def"}
{"id": 2, "sub_id": 22, "val":"ghi"}
{"id": 3, "sub_id": 31, "val":"jkl"}
{"id": 4, "sub_id": 41, "val":"mno"}
{"id": 5, "sub_id": 51, "val":"pqr"}
{"id": 5, "sub_id": 52, "val":"slu"}
{"id": 6, "sub_id": 61, "val":"vwz"}
% jluniq -k id data.ndjson
{"id":1,"sub_id":11,"val":"abc"}
{"id":2,"sub_id":21,"val":"def"}
{"id":3,"sub_id":31,"val":"jkl"}
{"id":4,"sub_id":41,"val":"mno"}
{"id":5,"sub_id":51,"val":"pqr"}
{"id":6,"sub_id":61,"val":"vwz"}
% cat data.ndjson | jluniq -k id
{"id":1,"sub_id":11,"val":"abc"}
{"id":2,"sub_id":21,"val":"def"}
{"id":3,"sub_id":31,"val":"jkl"}
{"id":4,"sub_id":41,"val":"mno"}
{"id":5,"sub_id":51,"val":"pqr"}
{"id":6,"sub_id":61,"val":"vwz"}
% jluniq -k id -g data.ndjson
[1,{"id":1,"sub_id":11,"val":"abc"}]
[2,{"id":2,"sub_id":21,"val":"def"},{"id":2,"sub_id":22,"val":"ghi"}]
[3,{"id":3,"sub_id":31,"val":"jkl"}]
[4,{"id":4,"sub_id":41,"val":"mno"}]
[5,{"id":5,"sub_id":51,"val":"pqr"},{"id":5,"sub_id":52,"val":"slu"}]
[6,{"id":6,"sub_id":61,"val":"vwz"}]
% jluniq -c id data.ndjson
[1,1]
[2,2]
[3,1]
[4,1]
[5,2]
[6,1]
```
## Related Links
* https://github.com/winebarrel/jlsort
* https://github.com/winebarrel/jljoin