{"id":23512722,"url":"https://github.com/mpkocher/qpyson","last_synced_at":"2025-10-18T01:31:17.924Z","repository":{"id":57459053,"uuid":"233718456","full_name":"mpkocher/qpyson","owner":"mpkocher","description":"Use Python to query/munge JSON files","archived":false,"fork":false,"pushed_at":"2020-01-21T03:31:34.000Z","size":22,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-13T09:54:48.084Z","etag":null,"topics":["cli","configuration-management","jq","json","manipulate-jsons","python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpkocher.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-13T23:59:38.000Z","updated_at":"2024-05-08T16:57:57.000Z","dependencies_parsed_at":"2022-08-27T21:01:11.003Z","dependency_job_id":null,"html_url":"https://github.com/mpkocher/qpyson","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpkocher%2Fqpyson","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpkocher%2Fqpyson/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpkocher%2Fqpyson/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpkocher%2Fqpyson/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpkocher","download_url":"https://codeload.github.com/mpkocher/qpyson/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249692870,"owners_count":21311419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","configuration-management","jq","json","manipulate-jsons","python","python3"],"created_at":"2024-12-25T13:19:12.754Z","updated_at":"2025-10-18T01:31:17.859Z","avatar_url":"https://github.com/mpkocher.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\ntitle: \"qpyson Tool to munge JSON documents using Python\"\nauthor: \"MK\"\ndate: \"1/20/2020\"\noutput: md_document\n---\n\n```{r setup, include=FALSE}\nknitr::opts_chunk$set(echo = TRUE)\nlibrary(reticulate)\nuse_python(\"/Users/mkocher/miniconda3/envs/core/bin/python\")\nuse_condaenv(\"core\", required = TRUE,  conda = \"/Users/mkocher/miniconda3/bin/conda\")\n```\n\n# qpyson: Thin Commandline Tool to Explore, Transform, and Munge JSON using Python\n\nThe JSON querying tool, [jq](https://stedolan.github.io/jq/), is a really powerful tool. However, it's sometimes a bit involved and has a learning curve that requires digging into the [jq manual](https://stedolan.github.io/jq/manual/) and familiarizing yourself with a custom language.\n\n`qpyson` is a thin commandline tool to explore, transform, or munge JSON using Python as the processing language. \n\n## Goals\n\n- Process (filter, map, general munging) of JSON files using Python\n- Thin layer to process or apply transforms written in Python \n- Provide Python function as a string to the commandline or define Python functions in an external file\n- Custom functions can be parameterized and configured from the commandline\n- Output results are emitted as JSON or in tabular form (using [tabulate](https://pypi.org/project/tabulate/) for quick viewing from the commandline \n\n\n### Non-Goals\n\n- A replacement for `jq`\n- No custom DSL for filtering or querying (use Python directly)\n- Does not support streaming (JSON files are loaded into memory)\n\n## Installation\n\nRecommended to install using a [virtualenv](https://docs.python-guide.org/dev/virtualenvs/) or [conda](https://docs.conda.io/en/latest/) env to install.\n\n```{bash, eval=FALSE}\npip install qpyson\n```\n\n## Quick Tour\n\nExample data from the Iris dataset.\n\n```{bash, comment=NA}\nhead examples/iris.json\n```\n\nThe commandline tool takes a function written as commandline string or referenced in an external file as well as the JSON file to be processed. \n\n```{bash, comment=NA, error=TRUE}\nqpyson --help\n```\n\n\nWe can define a custom function to process the JSON dataset. By default the function is named `f` and can be customized by `-f` or `--function-name` commandline argument.\n\n```{bash, comment=NA}\nqpyson \"def f(d): return d[0]\" examples/iris.json\n```\n\nWe can also write custom functions in a Python file. \n\n```{bash, comment=NA}\ncat examples/iris_explore.py\n```\n\nExecuting `--help` will show the output options.\n\n```{bash, comment=NA}\nqpyson examples/iris_explore.py examples/iris.json --help\n```\n\nExecuting function `f`, yields:\n\n```{bash, comment=NA}\nqpyson examples/iris_explore.py examples/iris.json \n```\n\nThe output view can be changed to a table view using `--print-table` or `-t`.\n\n```{bash, comment=NA}\nqpyson examples/iris_explore.py examples/iris.json --print-table --table-style github\n```\n\nA better example using function `f2` defined in `iris_explore.py`\n\n```{bash, comment=NA}\nqpyson examples/iris_explore.py examples/iris.json  --function-name f2 --print-table\n```\n\nCustom functions can be defined with required or optional values (with defaults) combined with Python 3 type annotations to generate \n\n```{bash, comment=NA}\ncat examples/iris.py\n```\n\nAnd calling `--help` will show the custom function specific arguments (e.g., `--max_items` and `--sort_direction`)\n\n```{bash, comment=NA}\nqpyson examples/iris.py examples/iris.json --help\n```\n\nAnd calling with custom options yields: \n\n```{bash, comment=NA}\nqpyson examples/iris.py examples/iris.json -t --max_items=3 --sort_direction=desc --sort_field sepalLength\n```\n\nAnother Example calling pandas underneath the hood to get a quick summary of the data. \n\n```{bash, comment=NA}\nqpyson examples/iris.py examples/iris.json -t -f g --field=sepalLength\n```\n\nIt's also possible to create thin JSON munging tools for configuration of systems or tools that take JSON as input.\n\nFor example a JSON configuration template with defaults.\n\n```{bash, comment=NA}\ncat examples/config_template.json\n```\nAnd a processing function, `f`.\n\n```{bash, comment=NA}\ncat examples/config_processor.py\n```\nRunning `--help` will show the supported configuration options. \n\n```{bash, comment=NA}\nqpyson examples/config_processor.py examples/config_template.json --help\n```\n\nNow configuring `alpha`, `beta` and `gamma`.\n\n```{bash, comment=NA}\nqpyson examples/config_processor.py examples/config_template.json --alpha 1.23 --beta 2.34 --gamma 3.45\n```\n\n\n# Testing\n\nTesting is currently done using RMarkdown using the make target `doc`. \n\nThis should probably be ported to non-R based approach. However, this current approach does keep the docs (e.g., README.md) up to date. \n\n# Related JQ-ish tools\n\nhttps://github.com/dbohdan/structured-text-tools#json\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpkocher%2Fqpyson","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpkocher%2Fqpyson","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpkocher%2Fqpyson/lists"}