https://github.com/lucaswerkmeister/ja2l
JSON array to lines
https://github.com/lucaswerkmeister/ja2l
dgsh json
Last synced: about 2 months ago
JSON representation
JSON array to lines
- Host: GitHub
- URL: https://github.com/lucaswerkmeister/ja2l
- Owner: lucaswerkmeister
- License: agpl-3.0
- Created: 2017-07-24T22:09:04.000Z (almost 9 years ago)
- Default Branch: main
- Last Pushed: 2023-05-31T16:55:51.000Z (about 3 years ago)
- Last Synced: 2025-10-27T02:40:11.087Z (8 months ago)
- Topics: dgsh, json
- Language: C
- Size: 38.1 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ja2l
[](https://travis-ci.org/lucaswerkmeister/ja2l)
JSON array to lines.
Expects a JSON array like this on standard input:
```json
[
{"foo":"bar"},
{"abc":"xyz"},
{"whatever":0}
]
```
and converts it into a stream of JSON values on standard output:
```json
{"foo":"bar"}
{"abc":"xyz"}
{"whatever":0}
```
This stream is suitable for processing with [jq],
and can also be split up for parallelized processing.
(This format is also variously known as
line delimited JSON (ldjson),
[newline delimited JSON (ndjson)][ndjson],
or [JSON lines (jsonl)][jsonl].)
## Project status
Functional, but more tests and various other goodies would be nice.
## Build instructions
`make && sudo make install`. Needs glibc and GCC.
The Makefile follows standard GNU conventions;
for instance, packagers can use something like
`make DESTDIR="$pkgdir" prefix=/usr install`.
(Note that standard `$(bindir)`, `$(man1dir)`, etc. directories
are expected to exist,
because the GNU Make manual says to not use the nonstandard `mkdir -p`
and I don’t know how to create the directories without it.)
## Requirements
* glibc (`error`, `getopt_long`)
* Linux (`/proc/sys/fs/pipe-max-size`, `fcntl(F_SETPIPE_SZ)`)
* GCC (`-fanalyzer`; if you remove that from `CFLAGS`, clang may or may not work)
* optional: [dgsh]
The reason lists in parentheses are probably not exhaustive.
(The most likely reason for that is that
I’ll probably forget to update them as I update the program.)
## dgsh support
ja2l can be built with [dgsh] support.
When used in a dgsh pipeline,
it accepts zero or one inputs
(depending on whether or not a file name was specified on the command line)
and scatters the JSON values across any (nonzero) number of outputs.
To build ja2l with dgsh support,
add `-DUSE_DGSH` to the `CPPFLAGS`
and `-ldgsh` to the `LDLIBS`,
e. g. like this:
```sh
make CPPFLAGS=-DUSE_DGSH LDLIBS=-ldgsh clean all
```
This can be used to speed up processing of the JSON data with [jq],
similar to this script:
```sh
function countElements {
jq -r '
.elements |
.[]
' | awk '
{
a[$0]++
}
END {
for (k in a)
print a[k] "\t" k
}
'
}
function summarizeElements {
awk -F'\t' '
{
a[$2] += $1
}
END {
for (k in a)
if (a[k] >= 1000)
print a[k] "\t" k
}
' |
sort -nr
}
ja2l | {{
countElements &
countElements &
countElements &
countElements &
}} |
cat |
summarizeElements
```
This prints the most common “elements” in the JSON input,
parallelizing the extraction and counting of elements across four `countElements` invocations.
The results of those invocations are then aggregated into a single result list again.
If the processing is CPU-bound,
`jq` is the expensive part,
and you have four processors or processor cores (without counting hyper-threading),
this should speed up processing by about a factor of four.
## Attribution
The `cleanup.h` header file is based on systemd header files,
which are published under the LGPL2.1+.
The install-related variables in the `Makefile` are copied from the GNU Make manual,
which is published under the FDL1.3+.
## License
The content of this repository is released under the AGPL3+
as provided in the LICENSE file that accompanied this code.
By submitting a “pull request” or otherwise contributing to
this repository, you agree to license your contribution under
the license mentioned above.
[dgsh]: https://www.spinellis.gr/sw/dgsh/
[jq]: https://stedolan.github.io/jq/
[ndjson]: http://ndjson.org/
[jsonl]: http://jsonlines.org/