https://github.com/clux/lq
jq compatible yq/tq implementation in rust
https://github.com/clux/lq
cli jq toml tq yaml yq
Last synced: about 2 months ago
JSON representation
jq compatible yq/tq implementation in rust
- Host: GitHub
- URL: https://github.com/clux/lq
- Owner: clux
- License: apache-2.0
- Created: 2023-08-30T16:36:09.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-11-06T02:24:19.000Z (11 months ago)
- Last Synced: 2024-12-13T20:38:54.254Z (10 months ago)
- Topics: cli, jq, toml, tq, yaml, yq
- Language: Rust
- Homepage:
- Size: 184 KB
- Stars: 22
- Watchers: 3
- Forks: 1
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# lq - low overhead yq/tq/jq cli
[](https://github.com/clux/lq/actions/workflows/release.yml)
[](https://crates.io/crates/lq)
[](https://deps.rs/repo/github/clux/lq)
A lightweight and portable [jq](https://jqlang.github.io/jq/) style cli for doing jq queries/filters/maps/transforms on **YAML**/**TOML**/**JSON** documents by converting to **JSON** and passing data to `jq`. Output is raw `jq` output which can optionally be mapped to TOML or YAML.
## Installation
Via cargo:
```sh
cargo install lq
```
or download a prebuilt from [releases](https://github.com/clux/lq/releases) either manually, or via [binstall](https://github.com/cargo-bins/cargo-binstall):
```sh
cargo binstall lq
```
**Note**: Requires `jq`.
## Why / Why Not
### jq compatibility
- arbitrary `jq` usage on any input format (yaml/toml/json) by going through json and jq
- [same syntax](https://jqlang.github.io/jq/manual/), same filters, types, operators, conditionals, regexes, assignment, modules, etc
- matches `jq`'s cli interface (only some extra input/output format controlling flags)
- supports `jq` output formatters such as `-c`, `-r`, and `-j` (compact, raw, joined output resp)
### Extra Features
- supports __multidoc yaml__ input, handles [yaml merge keys](https://yaml.org/type/merge.html) (expanding tags)
- supports __multidoc__ document splitting into expression based filenames
- supports __in-place edits__ of documents
- maintains __key order__ even while roundtripping between formats
- reads from __stdin xor file__ (file if last arg is a file)
- filetype format inference when passing files
- quick input/output flags: `-y` (YAML out) or `-t` (TOML out), `-T` (TOML in), `-J` (JSON in)
### Portable yq replacement
- ~[1MB](https://github.com/clux/lq/releases/latest) in binary (for small CI images / [binstalled ci actions](https://github.com/cargo-bins/cargo-binstall#faq))
- 99% replacement of [python-yq](https://kislyuk.github.io/yq/) (with `yq` named/linked to `lq`)
### Limitations
- Shells out to `jq` (not standalone - [for now](https://github.com/clux/lq/issues/64))
- Expands [YAML tags](https://yaml.org/spec/1.2-old/spec.html#id2764295) (input is [singleton mapped](https://docs.rs/serde_yaml/latest/serde_yaml/with/singleton_map/index.html) -> [recursively](https://docs.rs/serde_yaml/latest/serde_yaml/with/singleton_map_recursive/index.html), then [merged](https://docs.rs/serde_yaml/latest/serde_yaml/value/enum.Value.html#method.apply_merge)) - so tags are [not preserved](https://github.com/clux/lq/issues/12) in the output
- Does not preserve indentation (unsupported in [serde_yaml](https://github.com/dtolnay/serde-yaml/issues/337))
- Halts on [duplicate keys](https://github.com/clux/lq/issues/14) in the input document
- Formats require a [serde implementation](https://serde.rs/#data-formats).
- Limited format support. No XML/CSV/RON support (or other more exotic formats). [KDL wanted](https://github.com/clux/lq/issues/56).
## Usage
### YAML
Use as [jq](https://jqlang.github.io/jq/tutorial/) either via stdin:
```sh
$ lq '.[3].kind' -r < test/deploy.yaml
Service
$ lq -y '.[3].metadata' < test/deploy.yaml
labels:
app: controller
name: controller
namespace: default
```
or from a file arg (at the end):
```sh
$ lq '.[3].kind' -r test/deploy.yaml
$ lq -y '.[3].metadata' test/deploy.yaml
```
### TOML
Infers input format from extension, or set explicitly via `-T` or `--input=toml`.
```sh
$ lq '.package.categories[]' -r Cargo.toml
command-line-utilities
parsing
```
convert jq output back into toml (`-t`):
```sh
$ lq -t '.package.metadata' Cargo.toml
[binstall]
bin-dir = "lq-{ target }/{ bin }{ format }"
pkg-url = "{ repo }/releases/download/{ version }lq-{ target }{ archive-suffix }"
```
convert jq output to yaml (`-y`) and set explicit toml input when using stdin (`-T`):
```sh
$ lq -Ty '.dependencies.clap' < Cargo.toml
features:
- cargo
- derive
version: 4.4.2
```
jq style compact output:
```sh
$ lq '.profile' -c Cargo.toml
{"release":{"lto":true,"panic":"abort","strip":"symbols"}}
```
To shortcut passing input formats, you can add `alias tq='lq --input=toml'` in your `.bashrc` / `.zshrc` (etc).
### JSON Input
Infers input format from extension, or set explicitly via `-J` or `--input=json`.
```sh
$ lq -Jy '.ingredients | keys' < test/guacamole.json
- avocado
- coriander
- cumin
- garlic
- lime
- onions
- pepper
- salt
- tomatoes
```
Using JSON input is kind of like talking to `jq` directly, with the benefit that you can change output formats, or do inplace edits.
### Formats
Default is going from `yaml` input to `jq` output to allow further pipes into `jq`.
- **Input** flags are **upper case** :: `-J` json input, `-T` toml input (shorthands for `--input=FORMAT`)
- **Output** flags are **lower case** :: `-y` yaml output, `-t` toml output (shorthands for `--output=FORMAT`)
Ex;
- `lq` :: yaml -> jq output
- `lq -t` :: yaml -> toml
- `lq -y` :: yaml -> yaml
- `lq -Tt` :: toml -> toml
- `lq -Jy` :: json -> yaml
- `jq -Ty` :: toml -> yaml
- `jq -Jt` :: json -> toml
Output formatting such as `-y` for YAML or `-t` for TOML will require the output from `jq` to be parseable json.
If you pass on `-r`,`-c` or `-c` for raw/compact output, then this output may not be parseable as json.
### Advanced Features
Two things you cannot do in `jq`:
#### Multidoc Splits
Split a bundle of yaml files into a yaml file per Kubernetes `.metadata.name` key:
```sh
mkdir -p crds
curl -sSL https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.82.1/stripped-down-crds.yaml \
| lq . -y --split '"crds/" + (.metadata.name) + ".yaml"'
```
#### In Place Edits
Patch a json file ([without multiple pipes](https://github.com/jqlang/jq/issues/105)):
```sh
lq -i '.SKIP_HOST_UPDATE=true' ~/.config/discord/settings.json
```
### Advanced jq
Any weird things you can do with `jq` works. Some common (larger) examples:
#### Selects
Select on yaml multidoc:
```sh
$ lq '.[] | select(.kind == "Deployment") | .spec.template.spec.containers[0].ports[0].containerPort' test/deploy.yaml
8000
```
Escaping keys with slashes etc in them:
```sh
lq '.updates[] | select(.["package-ecosystem"] == "cargo") | .groups' .github/dependabot.yml
```
#### Modules
You can import [jq modules](https://jqlang.github.io/jq/manual/#modules) e.g. [k.jq](https://github.com/clux/lq/blob/main/test/modules/k.jq):
```sh
$ lq 'include "k"; .[] | gvk' -r -L$PWD/test/modules < test/deploy.yaml
v1.ServiceAccount
rbac.authorization.k8s.io/v1.ClusterRole
rbac.authorization.k8s.io/v1.ClusterRoleBinding
v1.Service
apps/v1.Deployment
```
### Debug Logs
The project respects `RUST_LOG` when set, and sends these diagnostic logs to stderr:
```sh
$ RUST_LOG=debug lq '.version' test/circle.yml
2023-09-18T23:17:04.533055Z DEBUG lq: args: Args { input: Yaml, output: Jq, yaml_output: false, toml_output: false, in_place: false, jq_query: ".version", file: Some("test/circle.yml"), compact_output: false, raw_output: false, join_output: false, modules: None }
2023-09-18T23:17:04.533531Z DEBUG lq: found 1 documents
2023-09-18T23:17:04.533563Z DEBUG lq: input decoded as json: {"definitions":{"filters":{"on_every_commit":{"tags":{"only":"/.*/"}},"on_tag":{"branches":{"ignore":"/.*/"},"tags":{"only":"/v[0-9]+(\\.[0-9]+)*/"}}},"steps":[{"step":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"step":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}}]},"jobs":{"build":{"docker":[{"image":"clux/muslrust:stable"}],"environment":{"IMAGE_NAME":"lq"},"resource_class":"xlarge","steps":["checkout",{"run":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}},{"run":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"run":"echo versions"}]},"release":{"docker":[{"image":"clux/muslrust:stable"}],"resource_class":"xlarge","steps":["checkout",{"run":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}},{"run":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"upload":{"arch":"x86_64-unknown-linux-musl","binary_name":"${IMAGE_NAME}","source":"target/x86_64-unknown-linux-musl/release/${IMAGE_NAME}","version":"${CIRCLE_TAG}"}}]}},"version":2.1,"workflows":{"my_flow":{"jobs":[{"build":{"filters":{"tags":{"only":"/.*/"}}}},{"release":{"filters":{"branches":{"ignore":"/.*/"},"tags":{"only":"/v[0-9]+(\\.[0-9]+)*/"}}}}]},"version":2}}
2023-09-18T23:17:04.533650Z DEBUG lq: jq args: [".version"]
2023-09-18T23:17:04.538606Z DEBUG lq: jq stdout: 2.1
2.1
```
### lq as yq
Because yaml is the default input language, you __can__ use it as your top level `yq` executable with a symlink or alias:
```sh
# globally make yq be lq
ln -s $(which lq) /usr/local/bin/yq
# alias yq to lq in shell environment only
alias yq=lq
```
It is mostly compatible with `python-yq` (which uses `jq` syntax) but differs from go yq (which invents its own syntax).
(This use-case was the first use-case for this tool, i.e. to get rid of heavy python deps in CI images)