{"id":13587314,"url":"https://github.com/NorthboundTrain/jqg","last_synced_at":"2025-04-07T21:33:35.915Z","repository":{"id":190865526,"uuid":"363696330","full_name":"NorthboundTrain/jqg","owner":"NorthboundTrain","description":"search JSON using JQ, printing full path/flattened results","archived":false,"fork":false,"pushed_at":"2023-08-27T19:13:45.000Z","size":270,"stargazers_count":19,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-02-13T21:47:19.937Z","etag":null,"topics":["bash-script","jq"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NorthboundTrain.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-05-02T16:24:22.000Z","updated_at":"2024-02-13T21:47:24.250Z","dependencies_parsed_at":"2024-02-13T21:57:25.288Z","dependency_job_id":null,"html_url":"https://github.com/NorthboundTrain/jqg","commit_stats":null,"previous_names":["northboundtrain/jqg"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NorthboundTrain%2Fjqg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NorthboundTrain%2Fjqg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NorthboundTrain%2Fjqg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NorthboundTrain%2Fjqg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NorthboundTrain","download_url":"https://codeload.github.com/NorthboundTrain/jqg/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247732869,"owners_count":20986944,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash-script","jq"],"created_at":"2024-08-01T15:06:09.125Z","updated_at":"2025-04-07T21:33:35.194Z","avatar_url":"https://github.com/NorthboundTrain.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# `jqg` - search, flatten, unflatten, and extract JSON using JQ\n\n[![BATS](https://github.com/NorthboundTrain/jqg/actions/workflows/bats.yml/badge.svg)](https://github.com/NorthboundTrain/jqg/actions/workflows/bats.yml)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-yellow.svg)](https://raw.githubusercontent.com/NorthboundTrain/jqg/main/LICENSE)\n[![Latest Release](https://img.shields.io/github/v/release/NorthboundTrain/jqg?sort=semver)](https://github.com/NorthboundTrain/jqg/releases/latest)\n[![Semantic Versioning](https://img.shields.io/badge/semantic_versioning-grey)](https://semver.org/)\n[![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)\n[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)\n\nJSON is an inherently hierarchical structure, which makes searching it for path information difficult. The JQG script flattens the hierarchical structure so that the path for each JSON end node is represented as a single string, thereby enabling easy searching and producing contextually meaningful results. It also produces valid JSON, which can be further processed, as needed.\n\nFor searching, JQG uses the [PCRE](https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions) engine built into JQ, which is much more powerful than `grep` or `egrep` (and it's certainly easier to use). For added flexibility, JQG can read from STDIN instead of from a file, allowing it to be used in pipelines, too. Finally, there are many options to control what is searched and how, as well as the format and content of the output.\n\nAlternately, JQG can unflatten JSON that has been previously flattened (or structured to look that way) and it can also extract a subset of the JSON input, including the extraction terms in the results. Both of these alternatives can be combined with searching in a \"composite mode\". The convenience scripts `jqu` and `jqx` are provided to invoke those composite modes more easily.\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\n###### example JSON used below: `odd-values.json`\n\n\u003c/summary\u003e\n\n```none\n$ jq . odd-values.json\n{\n  \"one\": {\n    \"start-string\": \"foo\",\n    \"null-value\": null,\n    \"integer-number\": 101,\n    \"string-with-pipe\": \"this|that\",\n    \"key|with|pipe\": true,\n    \"string-with-parens\": \"(this and that)\",\n    \"key(with)parens\": true,\n    \"bare-parens()\": true,\n    \"left(paren-only\": true,\n    \"unmatched-left)-paren\": false,\n    \"dollar $ign\": \"both-sides-$now\"\n  },\n  \"two\": [\n    {\n      \"two-a\": {\n        \"non-integer-number\": -101.75,\n        \"number-zero\": 0\n      },\n      \"true-boolean\": true,\n      \"two-b\": {\n        \"false-boolean\": false\n      }\n    },\n    {\n      \"two-c\": {\n        \"alpha-num-1\": \"a1\",\n        \"alpha-num-2\": \"2b\",\n        \"alpha-num-3\": \"a12b\"\n      }\n    }\n  ],\n  \"three\": {\n    \"empty-string\": \"\",\n    \"empty-object\": {},\n    \"empty-array\": []\n  },\n  \"four\": [\n    \"first\",\n    null,\n    {},\n    999,\n    \"fourth\"\n  ],\n  \"end-string\": \"bar\"\n}\n```\n\n\u003c/details\u003e\n\n##### Search Mode\n\n###### the old way: `jq` \u0026 `grep`\n\n```none\n# some not-too-useful jq/grep results\n$ jq . odd-values.json | grep string\n    \"start-string\": \"foo\",\n    \"string-with-pipe\": \"this|that\",\n    \"string-with-parens\": \"(this and that)\",\n    \"empty-string\": \"\",\n  \"end-string\": \"bar\"\n\n$ jq . odd-values.json | grep 0\n    \"integer-number\": 101,\n        \"non-integer-number\": -101.75,\n        \"number-zero\": 0\n\n$ jq . odd-values.json | grep 'int\\|false'\n    \"integer-number\": 101,\n    \"unmatched-left)-paren\": false,\n        \"non-integer-number\": -101.75,\n        \"false-boolean\": false\n```\n\n###### new and improved: `jqg`\n\n```none\n# much more useful jqg results\n$ jqg string odd-values.json\n{\n  \"one.start-string\": \"foo\",\n  \"one.string-with-pipe\": \"this|that\",\n  \"one.string-with-parens\": \"(this and that)\",\n  \"three.empty-string\": \"\",\n  \"end-string\": \"bar\"\n}\n\n$ jqg -v 0 odd-values.json\n{\n  \"one.integer-number\": 101,\n  \"two.0.two-a.non-integer-number\": -101.75,\n  \"two.0.two-a.number-zero\": 0\n}\n\n$ jqg 'int|false' odd-values.json\n{\n  \"one.integer-number\": 101,\n  \"one.unmatched-left)-paren\": false,\n  \"two.0.two-a.non-integer-number\": -101.75,\n  \"two.0.two-b.false-boolean\": false\n}\n\n\n\n# The power of PCRE\n# - search values looking for a 0 without a preceding number\n$ jqg -v '(?\u003c!\\d)0' odd-values.json\n{\n  \"two.0.two-a.number-zero\": 0\n}\n\n# - the same or an empty array\n$ jqg -v '(?\u003c!\\d)0|\\[]' odd-values.json\n{\n  \"two.0.two-a.number-zero\": 0,\n  \"three.empty-array\": []\n}\n\n\n\n# can be used in pipelines, too\n$ curl -s https://raw.githubusercontent.com/NorthboundTrain/jqg/main/test/odd-values.json | jqg -v '(?\u003c!\\d)0|\\[]'\n{\n  \"two.0.two-a.number-zero\": 0,\n  \"three.empty-array\": []\n}\n```\n\n##### Unflatten Mode\n\n```none\n# flatten \u0026 search\n$ jqg 'int|false' odd-values.json\n{\n  \"one.integer-number\": 101,\n  \"one.unmatched-left)-paren\": false,\n  \"two.0.two-a.non-integer-number\": -101.75,\n  \"two.0.two-b.false-boolean\": false\n}\n\n# flatten \u0026 search, then unflatten\n$ jqg 'int|false' odd-values.json | jqg -u\n{\n  \"one\": {\n    \"integer-number\": 101,\n    \"unmatched-left)-paren\": false\n  },\n  \"two\": [\n    {\n      \"two-a\": {\n        \"non-integer-number\": -101.75\n      },\n      \"two-b\": {\n        \"false-boolean\": false\n      }\n    }\n  ]\n}\n```\n\n##### Extract Mode\n\n```none\n# regular JQ command\n$ jq .two odd-values.json\n[\n  {\n    \"two-a\": {\n      \"non-integer-number\": -101.75,\n      \"number-zero\": 0\n    },\n    \"true-boolean\": true,\n    \"two-b\": {\n      \"false-boolean\": false\n    }\n  },\n  {\n    \"two-c\": {\n      \"alpha-num-1\": \"a1\",\n      \"alpha-num-2\": \"2b\",\n      \"alpha-num-3\": \"a12b\"\n    }\n  }\n]\n\n\n# JQG's extract retains the selector given\n$ jqg -x .two odd-values.json\n{\n  \"two\": [\n    {\n      \"two-a\": {\n        \"non-integer-number\": -101.75,\n        \"number-zero\": 0\n      },\n      \"true-boolean\": true,\n      \"two-b\": {\n        \"false-boolean\": false\n      }\n    },\n    {\n      \"two-c\": {\n        \"alpha-num-1\": \"a1\",\n        \"alpha-num-2\": \"2b\",\n        \"alpha-num-3\": \"a12b\"\n      }\n    }\n  ]\n}\n\n# extract deep into a structure\n$ jqg -x '.two[0].\"two-b\"' odd-values.json\n{\n  \"two\": [\n    {\n      \"two-b\": {\n        \"false-boolean\": false\n      }\n    }\n  ]\n}\n\n# composite mode - extract \u0026 search\n$ jqx .two 'int|false' odd-values.json\n{\n  \"two.0.two-a.non-integer-number\": -101.75,\n  \"two.0.two-b.false-boolean\": false\n}\n```\n\nMany more examples are provided in [jqg-examples.md](docs/jqg-examples.md).\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Usage](#usage)\n- [Documentation \u0026 Examples](#documentation--examples)\n- [Version History](#version-history)\n- [Acknowledgements](#acknowledgements)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Installation\n\n### Prerequisites\n\nThe JQG script is entirely self-contained except for the need to have both `jq` and `bash` on the system somewhere; `bash` itself needs to be on your `$PATH`, but `jq` does not -- see the [documentation](docs/jqg.md) for more details on how to work when `jq` is not in your `$PATH`.\n\n### Download the Script\n\nIt's easy to get the latest stable version of the script. First decide where you want to put it, and then grab it using `wget`:\n\n```none\ncd /path/to/script/dir\nwget https://github.com/NorthboundTrain/jqg/raw/main/src/jqg\nwget https://github.com/NorthboundTrain/jqg/raw/main/src/jqu\nwget https://github.com/NorthboundTrain/jqg/raw/main/src/jqx\nchmod +x jqg jqu jqx\n```\n\n### Clone the Repo\n\nAlternately, you can clone the whole repo:\n\n```none\n# HTTP\ncd /path/to/git/parent/dir\ngit clone https://github.com/NorthboundTrain/jqg.git\n\n# SSH\ncd /path/to/git/parent/dir\ngit clone git@github.com:NorthboundTrain/jqg.git\n```\n\nIf you want to run the unit tests, you will also need the [BATS](https://github.com/bats-core) sub-modules; you can clone them at the same time by adding in the `--recurse-submodules` option for `git clone`:\n\n```none\ngit clone --recurse-submodules https://github.com/NorthboundTrain/jqg.git\ngit clone --recurse-submodules git@github.com:NorthboundTrain/jqg.git\n```\n\n## Usage\n\n### Requirements\n\n- Bash 3.0.27+\n- JQ 1.6+\n\n### Basic Usage\n\nExecute JQG against a specific JSON file:\n\n```none\njqg search-string foo.json\njqg --unflatten flat.json\njqg --extract .some_elem foo.json\n```\n\nExecute JQG in a pipeline:\n\n```none\ncurl -s 'https://api.github.com/repos/NorthboundTrain/jqg' | jqg 'name|count'\n```\n\n## Documentation \u0026 Examples\n\n- [jqg.md](docs/jqg.md) - the JQG man page\n- [jqg-examples.md](docs/jqg-examples.md) - an exhaustive look at the different invocation methods as well as each command line option\n- [jqg-filters.md](docs/jqg-filters.md) - all JQG filters fully annotated\n- [runing-tests.md](test/running-tests.md) - information on running the provided unit tests\n\n## Version History\n\nsee [CHANGELOG.md](CHANGELOG.md)\n\n## Acknowledgements\n\nThe filters to convert the hierarchical structure of JSON into a flat structure are largely based on a blog post from [Fabian Keller](https://www.fabian-keller.de/about/) entitled [5 Useful jq Commands to Parse JSON on the CLI](https://www.fabian-keller.de/blog/5-useful-jq-commands-parse-json-cli/).\n\nThe core of the unflatten filter is taken from a [StackOverflow answer](https://stackoverflow.com/a/69650189) given by user [pmf](https://stackoverflow.com/users/2158479/pmf).\n\nThe core of the extract filter is taken from a [StackOverflow answer](https://stackoverflow.com/a/71334337/45978) given again by user [pmf](https://stackoverflow.com/users/2158479/pmf).\n\nThis project uses code from the following projects:\n\n- [pure-getopt](https://github.com/agriffis/pure-getopt) - a drop-in replacement for GNU `getopt`, written in Bash\n- [bats-core](https://github.com/bats-core) - Bash Automated Testing System\n  - [bats-core](https://github.com/bats-core/bats-core), [bats-support](https://github.com/bats-core/bats-support), [bats-assert](https://github.com/bats-core/bats-assert), [bats-file](https://github.com/bats-core/bats-file)\n\n## Contributing\n\n### Bugs / Feature Requests\n\nBugs \u0026 feature requests are tracked as GitHub [issues](https://github.com/NorthboundTrain/jqg/issues).\n\n### Pull Requests\n\n1. Check the open issues or open a new issue to start a discussion around your feature idea or the bug you found\n1. Fork the repository and make your changes\n1. Make sure all unit tests pass (and add new ones, if appropriate)\n1. Update documentation as needed\n1. Open a new pull request\n\n## License\n\n[Apache-2.0](LICENSE)\u003cbr /\u003e\n© 2021 Joseph Casadonte\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNorthboundTrain%2Fjqg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNorthboundTrain%2Fjqg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNorthboundTrain%2Fjqg/lists"}