Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mtrunkat/jsonlash
CLI utility for filtering and aggregation of JSONL streams.
https://github.com/mtrunkat/jsonlash
aggregation cli filtering logs stream
Last synced: about 1 month ago
JSON representation
CLI utility for filtering and aggregation of JSONL streams.
- Host: GitHub
- URL: https://github.com/mtrunkat/jsonlash
- Owner: mtrunkat
- Created: 2018-10-01T20:15:37.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2019-02-03T21:18:24.000Z (almost 6 years ago)
- Last Synced: 2024-10-03T07:12:52.338Z (about 1 month ago)
- Topics: aggregation, cli, filtering, logs, stream
- Language: JavaScript
- Homepage: https://www.npmjs.com/package/jsonlash
- Size: 14.6 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
jsonlash
========
[](https://oclif.io)
[](https://npmjs.org/package/jsonlash)
CLI utility for filtering and aggregation of [JSONL](http://jsonlines.org/) streams. No matter which service for logging you use (LogDNA, Papertrail, Loggly, etc.) simply pipe log into **jsonlash** set up filters and aggregators and see aggregated data in realtime.
## Usage
### Installation
Install from [NPM](http://npmjs.com) globally:
```bash
npm install -g jsonlash
```
After installation you can simply run jsonlash from your terminal with `-h` parameter to display help page:
```bash
jsonlash -h
```
### Basic usage with filtering
We currently use [Log DNA](https://logdna.com/) as logging service so I am going to use it in examples but it's going to work with any [JSONL](http://jsonlines.org/) stream. So pipe your log stream to jsonlash:
```bash
logdna tail | jsonlash
```
Now it will simply print out the log as it comes. So let's filter the API logs that are in the form:
```
{
"msg": "API call",
"req": {
"duration": 590,
"method": "GET",
"route": "V2.datasets.items",
...
}
...
}
```
Filtering is done using `-f [FILTER]` parameter:
```bash
logdna tail | jsonlash -f 'msg=API call'
```
We can add more filters to filter out only requests with POST method and duration over 1000ms. And also add parameter `-e` to expand printed JSONs to be more readable:
```bash
logdna tail | jsonlash -f 'msg=API call' -f 'req.method=POST' -f 'req.duration>1000' -e
```
### Aggregations
Let's continue with API logs example. To group log lines by request method and compute average and maximal duration call:
```bash
logdna tail | jsonlash -f 'msg=API call' -a req.method --max req.duration --avg req.duration
```
and output will be a table with data aggregated in realtime:
## Examples
### 1.
Aggregate logs by two fields `req.method` and `req.routeName` and compute average duration and the maximum duration
```bash
... | jsonlash -a req.method -a req.routeName --max req.duration --avg req.duration
```
### 2.
Filter out requests taking more than a 10s, grouped them by `req.routeName` and compute how many users requested each of them:
```bash
... | jsonlash -f 'req.duration>10000' -a req.routeName --uni req.userId
```
## Command reference
```
This is a simple command line tool to filter and aggregate JSONL (json-lines) streams.
USAGE
$ jsonlash
OPTIONS
-a, --aggregate=[FIELD] aggregate JSONL items
-d, --debug debug mode, shows JSON parsing errors
-e, --expand expand outputted JSON
-f, --filter=[CONDITION] filter JSONL items
-h, --help show CLI help
-v, --version show CLI version
--avg=avg aggregate average value over all occurrences of given field
--max=max aggregate maximum value over all occurrences of given field
--min=min aggregate minimum value over all occurrences of given field
--sum=sum aggregate sum over all occurrences of given field
--uni=uni aggregate number of unique occurrences of given field
DESCRIPTION
Simply pipe in any JSONL stream and with filter and/or aggregation flags.
If you use only --filter flag then jsonlash outputs filtered jsonl stream.
If you also use --aggregate flag then it renders a table with aggregated data.
Additionally you may add one or more --min|--max|--sum|---avg|--uni flags to
compute aggregated values of given fields.
```