{"id":13760398,"url":"https://github.com/markus-wa/cq","last_synced_at":"2026-01-11T22:53:58.923Z","repository":{"id":43258304,"uuid":"349503674","full_name":"markus-wa/cq","owner":"markus-wa","description":"Clojure Query: A Command-line Data Processor for JSON, YAML, EDN, XML and more","archived":false,"fork":false,"pushed_at":"2024-07-22T10:34:12.000Z","size":204,"stargazers_count":162,"open_issues_count":1,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-08-03T13:04:47.271Z","etag":null,"topics":["cli","clojure","command-line","csv","data-processing","data-transformation","edn","hacktoberfest","json","msgpack","transformation","xml","yaml"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/markus-wa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-19T17:28:04.000Z","updated_at":"2024-08-03T13:04:48.795Z","dependencies_parsed_at":"2024-06-24T12:34:10.094Z","dependency_job_id":"763cb322-ccf4-43cb-85fb-abdeb6f33c8b","html_url":"https://github.com/markus-wa/cq","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markus-wa%2Fcq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markus-wa%2Fcq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markus-wa%2Fcq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markus-wa%2Fcq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/markus-wa","download_url":"https://codeload.github.com/markus-wa/cq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224949795,"owners_count":17397236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","clojure","command-line","csv","data-processing","data-transformation","edn","hacktoberfest","json","msgpack","transformation","xml","yaml"],"created_at":"2024-08-03T13:01:09.476Z","updated_at":"2026-01-11T22:53:58.915Z","avatar_url":"https://github.com/markus-wa.png","language":"Clojure","funding_links":[],"categories":["Clojure"],"sub_categories":[],"readme":"# cq (Clojure Query)\n\nCommand-line Data Processor for EDN, YAML, JSON, XML and other data formats.\n\nThe joy of Clojure's threading macros, but on the command line!\n\nMay or may not have invented the Hash-Pipe `#|` operator 🍁 (citation needed)\n\n[![CI / CD](https://github.com/markus-wa/cq/actions/workflows/cicd.yaml/badge.svg)](https://github.com/markus-wa/cq/actions/workflows/cicd.yaml)\n[![codecov](https://codecov.io/gh/markus-wa/cq/branch/main/graph/badge.svg?token=zGovO2H0bm)](https://codecov.io/gh/markus-wa/cq)\n[![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/markus-wa/cq)](https://github.com/markus-wa/cq/releases)\n[![License](https://img.shields.io/badge/license-EPL--2.0-blue)](LICENSE)\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fmarkus-wa%2Fcq.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fmarkus-wa%2Fcq?ref=badge_shield)\n\n![image](https://user-images.githubusercontent.com/5138316/214540308-414fb004-0b3f-4377-b3a1-239cb8c7d3fb.png)\n\n## Installation\n\n### Homebrew\n\n    brew install markus-wa/brew/cq\n### Windows / Scoop\n    scoop bucket add scoop-clojure\n    scoop install scoop-clojure/cq\n### Manual\n\n1. Download the latest version for your OS from the [releases](https://github.com/markus-wa/cq/releases) page.\n    - note: you should avoid `cq-jvm` if possible as these are not GraalVM native images and will be slow to start.\n2. Rename binary to `cq`\n3. `chmod +x cq`\n4. Move into a location on `$PATH`\n\n## Rationale\n\nWhile there are a few similar, amazing tools out there (such as `jq`, `jet` or `babashka`), `cq` tries to resolve some of their shortcomings such as having to learn custom query languages, lacking powerful data transformation libraries or quick and _easy_ (yes I said the e word) handling of many input and output formats.\n\ncq aims to:\n\n- Not require learning yet another query language - it's just ~~data~~ Clojure!\n- Give the user complete power for writing queries, with as few limitations as possible\n- Provide various input and output formats out of the box\n- Be opinionated, ship useful tools/libraries pre-bundled\n\n## Features\n\n- Supports all elements of Clojure that are supported by [SCI](https://github.com/borkdude/sci)\n- Data Formats:\n  - EDN\n  - YAML\n  - JSON (regular + [line-delimited](https://jsonlines.org/))\n  - XML\n  - MsgPack\n  - CSV\n  - Cognitec's Transit format\n  - Text (raw and line-separated)\n  - HTML (via Hickory)\n- [Various reader macros](#reader-macros) that make writing queries easier\n- [Threading Macro Redirection](#threading-macro-redirection) reduces need for parentheses\n- Coloured output / syntax highlig for EDN output\n- No startup lag thanks to GraalVM native-images\n- Comes batteries-included with the following libraries for transforming nested data structures and utilities\n  - [specter](https://github.com/redplanetlabs/specter) - `(s/...)`\n  - [medley](https://github.com/weavejester/medley) - `(m/...)`\n  - [camel-snake-kebab](https://clj-commons.org/camel-snake-kebab/) - `(csk/...)`\n  - [xml-in](https://github.com/tolitius/xml-in) - `(xml-in/...)`\n  - [data.json](https://github.com/clojure/data.json) - `(json/...)`\n  - [data.csv](https://github.com/clojure/data.csv) - `(csv/...)`\n  - [data.xml](https://github.com/clojure/data.xml) - `(xml/...)`\n\n## Usage\n\n```\n$ cq --help\ncq is a command-line data processor for JSON, YAML, EDN and other data formats that utilises Clojure as it's query language.\n\nUsage: cq [options] [--] QUERY\n\nExamples\n  echo '{a: {b: [1, 2, 3]}}' | cq ':a :b (map inc)'\n\n  printf 'http://example.com/some/path' | cq -i text -- '-\u003e str/upper-case (str/split #\"/\") -\u003e\u003e (map str/reverse)'\n\nOptions:\n  -i, --in FORMAT                         yaml     Input format: csv, edn, json, json-ld, lines, msgpack, text, transit, yaml, html\n  -o, --out FORMAT                        edn      Output format: csv, edn, json, json-ld, lines, msgpack, text, transit, yaml, html\n  -p, --[no-]pretty                                Pretty print output - default is true\n      --color COLOR                       auto     When pretty printing, whether to use colors: auto, off, on - default is auto\n  -c                                               Same as --color=on\n  -C                                               Same as --color=off\n  -k, --key-fn FN                         keyword  Function used to transform keys - currently only supported for JSON and CSV\n      --yaml-unsafe                                Enables unsafe mode in clj-yaml / SnakeYAML\n      --[no-]yaml-keywords                         Turn map keys into keywords in clj-yaml - default is true\n      --yaml-max-aliases-for-collections           Sets max aliases for collections in clj-yaml / SnakeYAML\n      --yaml-allow-recursive-keys                  Allows recursive keys in clj-yaml / SnakeYAML\n      --yaml-allow-duplicate-keys                  Allows duplicate keys in clj-yaml / SnakeYAML\n      --yaml-flow-style STYLE             auto     Sets flow style in SnakeYAML: auto, block, flow\n      --transit-format-in FORMAT          json     Set the reader type for transit: json, json-verbose, msgpack\n      --transit-format-out FORMAT         json     Set the writer type for transit: json, json-verbose, msgpack\n  -h, --help\n\nSee https://github.com/markus-wa/cq for more information.\n```\n\nBy default `cq` uses [thread last (`-\u003e\u003e`)](https://clojure.org/guides/threading_macros#thread-last) semantics.\n\n```bash\n$ echo '{\"a\": {\"b\": [1, 2, 3]}}' | cq ':a :b (map inc)'\n(2 3 4)\n```\n\nYou can change this by using [Threading Macro Redirection](#threading-macro-redirection)\n\n```bash\n$ echo '{\"a\": {\"b\": [3, 4, 5]}}' | cq ':a :b -\u003e (conj 2) -\u003e\u003e (map dec)'\n```\n\nYou can use `clojure.instant` (alias `inst`) to parse timestamps.\n\n```bash\n$ echo '{\"a\": \"2023-03-11T03:01:11.000Z\"}' | cq :a inst/read-instant-timestamp\n#inst \"2023-03-11T03:01:11.000-00:00\"\n```\n\nUsing `#|` you can use the current value as `.`.\n\n```bash\n$ curl -s 'https://api.github.com/repos/markus-wa/cq/commits?per_page=5' | \\\ncq 'second #| {:author (-\u003e . :commit :author :name) :message (-\u003e . :commit :message) :parents (-\u003e\u003e . :parents (map :html_url))}'\n{:author \"Markus Walther\",\n :message \"tests: fix coloured output breaking tests\",\n :parents (\"https://github.com/markus-wa/cq/commit/92ff81edbd6f53f0d20aa5a18ccf6cac53bbe50e\")}\n```\n\nThere's also a destructuring macro `#\u0026` to make using `(let)` easier.\n\n```bash\n$ printf \"http://example.com/some/path\" | \\\n  cq -i text -- '-\u003e (str/split #\"/\") #\u0026 ([protocol _ host] {:protocol protocol :host host})'\n{:protocol \"http:\", :host \"example.com\"}\n```\n\n`#f` can be used to simplify creating an anonymous function that returns a value, rather than calls a function.\u003cbr\u003e\nAlso note how `m/map-kv` is provided by `medley.core`.\n```bash\n$ echo '{a: {b: 2, c: 3}}' | cq ':a  (m/map-kv #f [%2 %1])'\n{2 :b, 3 :c}\n```\n\n### Threading Macro Redirection\n\nWhile things like [`-\u003e-\u003e\u003e\u003c?as-\u003econd-\u003e!`](https://github.com/randomcorp/thread-first-thread-last-backwards-question-mark-as-arrow-cond-arrow-bang) are pretty funny,\nit can be pretty convenient to just redirect a threading macro when you're working on a simple terminal without paredit.\n\nAll threading operators will change the query after that point to their implementation until followed by any other threading operator (no need for parentheses).\n\nNote that threading redirection is currently only supported on the top level, not in nested threading macros.\n\n```bash\n$ printf \"http://example.com/some/path\" | \\\n  cq -i text -- '-\u003e str/upper-case (str/split #\"/\") -\u003e\u003e (map str/reverse)'\n(\":PTTH\" \"\" \"MOC.ELPMAXE\" \"EMOS\" \"HTAP\")\n```\n\nCurrently supported threading operators for redirection:\n\n- `-\u003e` thread first\n- `-\u003e\u003e` thread last\n- `some-\u003e` thread some\n- `some-\u003e\u003e` thread some last\n- `as-\u003e` thread with var name\n\n### Included Libraries \u0026 Namespace Aliases\n\n| Library | Namespace | Alias | Example Query |\n| ------- | --------- | ----- | ------- |\n| `tolitius/xml-in` | `xml-in.core` | `xml-in` | `#\\| (xml-in/find-all . [:universe :system :solar :planet])` |\n| `medley` | `medley.core` | `m` | `(m/mak-kv (fn [k v] [v k]))` |\n| `com.rpl/specter` | `com.rpl.specter` | `s` | `(s/transform [MAP-VALS MAP-VALS] inc)` |\n| `camel-snake-kebab` | `camel-snake-kebab.core` | `csk`  | `csk/-\u003eSCREAMING_SNAKE_CASE` |\n| `clojure.instant` | `clojure.instant` | `inst`  | `inst/read-instant-timestamp` |\n| `clojure.data.csv` | `clojure.data.csv` | `csv`  | `csv/read-csv` |\n| `clojure.data.json` | `clojure.data.json` | `json`  | `json/read-str` |\n| `clojure.data.xml` | `clojure.data.xml` | `xml`  | `xml/parse-str` |\n\n\n### Reader Macros\n\nThis table explains the different reader macros provided by `cq`.\n`\u003cf\u003e` is the form passed in after the reader macro.\n\n| Reader Macro | Description | Interpolates to | Example |\n| ------------ | ----------- | --------------- | ------- |\n| `#\\| \u003cf\u003e` | Hash-Pipe 🍁: Use the current value as `.` | `((fn [.] \u003cf\u003e))` | `#\\| (\u003c 5 . 10)` |\n| `#map \u003cf\u003e` | Map elements of a seq | `(map (fn [.] \u003cf\u003e))` | `#map {:a (:a .) :first-child (-\u003e . :children first)}` |\n| `#\u0026 (\u003cd\u003e \u003cf...\u003e)` | Destructor: Destructure into vars | `((fn [.] (let [\u003cd\u003e .] \u003cf\u003e)` | `#\u0026 ({:keys [a b c]} [a b c]})` |\n| `#f \u003cf\u003e` | Anonymous function, returns value of f, not evaluation of f | `#(do \u003cf\u003e)` | `(map-kv #f [%2 %1])` |\n\n### Tips \u0026 Tricks\n\n#### cq is slow!\n\nPretty printing can be pretty slow with the JSON and EDN libraries we use. one trick is to use `cq` for querying and `jq` for formatting.\n\nE.g. this is pretty fast\n\n    cat data.yaml | cq -o json --no-pretty | jq\n\n## TODO\n\n- maybe Parquet support\n\n## Acknowledgements\n\nThis project takes a lot of inspiration from [`jq`](https://stedolan.github.io/jq/), [`jet`](https://github.com/borkdude/jet) and [`babashka`](https://github.com/babashka/babashka)\n\n\n## License\n\nThis project is licensed under the [EPL-2.0 License](LICENSE).\n\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fmarkus-wa%2Fcq.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fmarkus-wa%2Fcq?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkus-wa%2Fcq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkus-wa%2Fcq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkus-wa%2Fcq/lists"}