Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mpkocher/qpyson
Use Python to query/munge JSON files
https://github.com/mpkocher/qpyson
cli configuration-management jq json manipulate-jsons python python3
Last synced: 7 days ago
JSON representation
Use Python to query/munge JSON files
- Host: GitHub
- URL: https://github.com/mpkocher/qpyson
- Owner: mpkocher
- License: mit
- Created: 2020-01-13T23:59:38.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2020-01-21T03:31:34.000Z (almost 5 years ago)
- Last Synced: 2024-12-01T15:16:58.083Z (about 1 month ago)
- Topics: cli, configuration-management, jq, json, manipulate-jsons, python, python3
- Language: Python
- Homepage:
- Size: 21.5 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- License: LICENSE
Awesome Lists containing this project
README
---
title: "qpyson Tool to munge JSON documents using Python"
author: "MK"
date: "1/20/2020"
output: md_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(reticulate)
use_python("/Users/mkocher/miniconda3/envs/core/bin/python")
use_condaenv("core", required = TRUE, conda = "/Users/mkocher/miniconda3/bin/conda")
```
# qpyson: Thin Commandline Tool to Explore, Transform, and Munge JSON using Python
The JSON querying tool, [jq](https://stedolan.github.io/jq/), is a really powerful tool. However, it's sometimes a bit involved and has a learning curve that requires digging into the [jq manual](https://stedolan.github.io/jq/manual/) and familiarizing yourself with a custom language.
`qpyson` is a thin commandline tool to explore, transform, or munge JSON using Python as the processing language.
## Goals
- Process (filter, map, general munging) of JSON files using Python
- Thin layer to process or apply transforms written in Python
- Provide Python function as a string to the commandline or define Python functions in an external file
- Custom functions can be parameterized and configured from the commandline
- Output results are emitted as JSON or in tabular form (using [tabulate](https://pypi.org/project/tabulate/) for quick viewing from the commandline
### Non-Goals
- A replacement for `jq`
- No custom DSL for filtering or querying (use Python directly)
- Does not support streaming (JSON files are loaded into memory)
## Installation
Recommended to install using a [virtualenv](https://docs.python-guide.org/dev/virtualenvs/) or [conda](https://docs.conda.io/en/latest/) env to install.
```{bash, eval=FALSE}
pip install qpyson
```
## Quick Tour
Example data from the Iris dataset.
```{bash, comment=NA}
head examples/iris.json
```
The commandline tool takes a function written as commandline string or referenced in an external file as well as the JSON file to be processed.
```{bash, comment=NA, error=TRUE}
qpyson --help
```
We can define a custom function to process the JSON dataset. By default the function is named `f` and can be customized by `-f` or `--function-name` commandline argument.
```{bash, comment=NA}
qpyson "def f(d): return d[0]" examples/iris.json
```
We can also write custom functions in a Python file.
```{bash, comment=NA}
cat examples/iris_explore.py
```
Executing `--help` will show the output options.
```{bash, comment=NA}
qpyson examples/iris_explore.py examples/iris.json --help
```
Executing function `f`, yields:
```{bash, comment=NA}
qpyson examples/iris_explore.py examples/iris.json
```
The output view can be changed to a table view using `--print-table` or `-t`.
```{bash, comment=NA}
qpyson examples/iris_explore.py examples/iris.json --print-table --table-style github
```
A better example using function `f2` defined in `iris_explore.py`
```{bash, comment=NA}
qpyson examples/iris_explore.py examples/iris.json --function-name f2 --print-table
```
Custom functions can be defined with required or optional values (with defaults) combined with Python 3 type annotations to generate
```{bash, comment=NA}
cat examples/iris.py
```
And calling `--help` will show the custom function specific arguments (e.g., `--max_items` and `--sort_direction`)
```{bash, comment=NA}
qpyson examples/iris.py examples/iris.json --help
```
And calling with custom options yields:
```{bash, comment=NA}
qpyson examples/iris.py examples/iris.json -t --max_items=3 --sort_direction=desc --sort_field sepalLength
```
Another Example calling pandas underneath the hood to get a quick summary of the data.
```{bash, comment=NA}
qpyson examples/iris.py examples/iris.json -t -f g --field=sepalLength
```
It's also possible to create thin JSON munging tools for configuration of systems or tools that take JSON as input.
For example a JSON configuration template with defaults.
```{bash, comment=NA}
cat examples/config_template.json
```
And a processing function, `f`.
```{bash, comment=NA}
cat examples/config_processor.py
```
Running `--help` will show the supported configuration options.
```{bash, comment=NA}
qpyson examples/config_processor.py examples/config_template.json --help
```
Now configuring `alpha`, `beta` and `gamma`.
```{bash, comment=NA}
qpyson examples/config_processor.py examples/config_template.json --alpha 1.23 --beta 2.34 --gamma 3.45
```
# Testing
Testing is currently done using RMarkdown using the make target `doc`.
This should probably be ported to non-R based approach. However, this current approach does keep the docs (e.g., README.md) up to date.
# Related JQ-ish tools
https://github.com/dbohdan/structured-text-tools#json