{"id":14779853,"url":"https://github.com/inconvergent/lqn","last_synced_at":"2026-04-01T17:18:08.258Z","repository":{"id":210121429,"uuid":"725792062","full_name":"inconvergent/lqn","owner":"inconvergent","description":"query language and terminal utility for querying and transforming Lisp, JSON and other text files. written in Common Lisp","archived":false,"fork":false,"pushed_at":"2026-01-19T23:09:33.000Z","size":495,"stargazers_count":53,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-03-28T00:37:15.732Z","etag":null,"topics":["common-lisp","csv","domain-specific-languages","dsl","json","terminal","txt","txt-files","unix-shell","utility"],"latest_commit_sha":null,"homepage":"","language":"Common Lisp","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inconvergent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-30T22:08:50.000Z","updated_at":"2026-03-28T00:03:21.000Z","dependencies_parsed_at":"2024-01-01T08:13:38.539Z","dependency_job_id":"53d6b379-3ebf-4228-ad53-226885fff096","html_url":"https://github.com/inconvergent/lqn","commit_stats":{"total_commits":119,"total_committers":2,"mean_commits":59.5,"dds":0.01680672268907568,"last_synced_commit":"0ed8e8b47fd181b548547ae58581a7e0812873f4"},"previous_names":["inconvergent/jqn","inconvergent/lqn"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/inconvergent/lqn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inconvergent%2Flqn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inconvergent%2Flqn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inconvergent%2Flqn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inconvergent%2Flqn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inconvergent","download_url":"https://codeload.github.com/inconvergent/lqn/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inconvergent%2Flqn/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290537,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["common-lisp","csv","domain-specific-languages","dsl","json","terminal","txt","txt-files","unix-shell","utility"],"created_at":"2024-09-17T01:01:01.380Z","updated_at":"2026-04-01T17:18:08.236Z","avatar_url":"https://github.com/inconvergent.png","language":"Common Lisp","readme":"# LQN - Lisp Query Notation\n\n## About\n\nLQN is a Common Lisp libary, query language and terminal utility to query and\ntransform text files such as `JSON` and `CSV`, as well as Lisp data (`LDN`), The\nterminal utilities will parse the input data to internal lisp structures\naccording to the input mode. Then the `lqn` query language can be used for queries\nand transformations.\n\n`lqn` started as an experiment and programming exercise. But it has turned into\na little language i find rather useful. Both in the terminal, and\nmore interestingly, as a meta language for writing macros in CL. The main\npurpose of the design is to make something that is intuitive, terse, yet\nflexible enough that you can write generic CL if you need to. I also wanted to\nmake something that requres a relatively simple compiler.\n\nHere is a small tutorial: https://inconvergent.net/2024/lisp-query-notation/.\n\nAn expanded version of the tutorial can be seen in this paper:\nhttps://inconvergent.net/code/lqn.pdf\n\nWhen using `LQN` on the terminal there are three terminal commands, or input\nmodes: `jqn`, `tqn` and `lqn`. For `JSON`, text and lisp data respectively.\n(For installation see below.) You can find some terminal command examples in\n[bin/lqn-sh.lisp](lqn), [bin/jqn-sh.lisp](jqn), and [bin/tqn-sh.lisp](tqn).\n\nSymbol documentation can be seen in [docs/lqn.md](docs/lqn.md).\n\n## Object Representation\n\nInternally `JSON` arrays/lists are represented as `vectors`. and `JSON`\nobjects/dicts are represented as `hash-tables` (`ht`). Thus a text file is a\n`vector` of `strings`.We use `object` in the context of Operators and other\n`LQN` utilities to refer to either a `vector` or a `ht`. Lisp data is read\ndirectly.\n\n## Operators\n\nThe following operators have special behaviour. You can also write generic CL\ncode in almost all contexts, as we demonstrate soon. In operators we use `_` to\nrefer to the current value.\n\nIn the following sections `[d]` represents an optional default value. E.g. if\nkey/index is missing, or if a functon would otherwise return `nil`.\n`k` is an initial counter\nvalue. Whereas `..` means that there can be arbitrary arguments/`expr`.\n`expr` denotes any expression or operator; like `(+ 1 _)` or `#[:id]`.\n\n### Strings and :keywords\n\nIn operators, and several functions, `:keywords` can be used to represent\nlowercase `strings`. This is useful in the terminal to avoid escaping strings.\nParticularly when using `Selector` operators. You can use `\"Strings\"` instead,\nif you need case or whitespace.\n\n### Pipe Operator - `||`\n\n`(|| expr ..)` pipes the results from the first `expr` to the second, and so\non.  Returns the result of the last `expr`. The Pipe operator surrounds all\nqueries by default. So it is usually not neccessary to use it explicitly.\n\nFor convenience the pipe has the following default translations:\n  - `fx`: to `(?map (fx _))`: map `fx` across all items.\n  - `:word`: to `[(isub? _ \"word\")]` to filter by `\"word\"`.\n  - `\"Word\"`: to `[(sub? _ \"Word\")]` to filter by `Word`, case sensitive.\n  - `(..)`: to itself. That is, expressions are not translated.\nso this is the default transalation for top level expressions in any query.\n\n### Get Operator - `@`\n\nSelect `:keys`, indexes or paths from nested structure:\n - `(@ k)`: get this key/index/path from current value.\n - `(@ k [d])`: get this key/index/path from current value.\n - `(@ o k [d])`: get this key/index/path from `o`.\n\nPaths support wildcards (`*`) and numerical indices for nested structures. E.g. this\nis a valid path: `:*/0/things`.\n\n### Map Operator - `#()`\n\nMap operations over `vector`; or over the values of a `ht`:\n  - `#(fx)`: map `(fx _)` across all items.\n  - `#(expr ..)`: evaluate these expressions sequentially on all items in `sequence`.\n\n### Selector Operators - `{}`, `#{}`, `#[]`\n\nSelect from one structure into a new data structure. using selectors:\n  - ` {s1 sel ..}`: from `ht` into new `ht`.\n  - `#{s1 sel ..}`: from `vector` of `hts` into new `vector` of `hts`.\n  - `#[s1 sel ..]`: from `vector` of `hts` into new `vector`.\n\nA selector is a triple `(mode key expr)`. Only key is required. If `expr` is\nnot provided the `expr` is `_`, that is: the value of the `key`. The modes are\nas follows:\n  - `+`: always include this `expr`. [default]\n  - `?`: include `expr` if the key is present and not `nil`.\n  - `%`: include Selector if `expr` is not `nil`.\n  - `-`: drop this key in `#{}` and `{}` operators; ignore Selector entirely in\n    `#[]` E.g. `{_ -@key3}` to select all keys except `key3`. `expr` is\n    ignored.\n\nSelectors can either be written out in full, or they can be be written in short\nform depending on what you want to achieve. The `@` in the following examples\nis used to append a mode to a key without having to wrap the Selector in\nparenthesis. If you need eg. case or spaces you can use `\"strings\"`. Here are\nsome examples using `{}`. It behaves the  same for the other Selector\noperators:\n```lisp\n{_}               ; select all keys.\n{_ :-@key1}       ; select all keys except \"key1\".\n{:key1 \"Key2\"}    ; select \"key1\" and \"Key2\".\n{:+@key}          ; same as :key [+ mode is default].\n{\"+@Key\"}         ; select \"Key\".\n{:?@key }         ; select \"key\" if the value is not nil.\n{(:%@key expr)}   ; select \"key\" if expr is not nil.\n{(\"?@Key\" expr)}  ; select \"Key\" if the value is not nil.\n{(\"%@Key\" expr)}  ; select \"Key\" if expr is not nil.\n{(:+ \"Key\" expr)} ; same as (\"+@Key\" expr).\n\n; Use `_` in `expr` to refer to the value of the selected key:\n{(:key1 sup))          ; convert value of \"key1\" to uppercase\n (:key3 (or _ \"That\")) ; select the value of \"key3\", or literally \"That\".\n (:key2 (+ 33 _))}     ; add 33 to value of \"key2\"\n\n; override and drop keys:\n{_                ; select all keys, then override these:\n (:key2 (sdwn _)) ; lowercase the value of \"key2\"\n  :-@key3}        ; drop \"key3\"\n```\nWe use `{}` in the examples but all Selector operators have the same behaviour.\n\n### Filter Operator - `[]` TODO TODO TODO\n\nFilter `vector`; or the values of a `ht`:\n  - ` [expr1 .. exprn]` to keep any object or value that satisfies the expressions.\n\nThe filter operator behaves somewhat similar to the Selector operators. They are used\nwith `[]`, `?srch`, `?xpr`, `?txpr`, `?mxpr` operators. The modes behave\nlike this:\n  - `+`: if there are multiple expressions with `+` mode, require ALL\n    of them to be satisfies.\n  - `?`: if there are any clauses with `?` mode, it will select items where\n    either of these clauses is satisfied\n  - `-`: items that match any clause with `-` mode will ALWAYS be dropped.\n\nIf this is not what you need, you can compose boolean expressions with regular\nCL boolen operators. Here are some examples:\n\n```lisp\n[:hello]               ; strings containing \"hello\".\n[:hi \"Hello\"]          ; strings containing either \"Hello\" OR \"hi\".\n[:+@hi :+@hello]       ; strings containing \"hi\" AND \"hello\".\n[:+@hi :+@hello \"OH\"]  ; strings containing (\"hi\" AND \"hello\") OR \"OH\".\n[int!?]                ; items that can be parsed as int.\n[(\u003e _ 3)]              ; numbers larger than 3.\n[_ :-@hi]              ; strings except those that contain \"hi\".\n[(+@pref? _ \"start\")   ; strings that start with \"start\" and end with \"end\".\n (+@post? _ \"end\")]\n[(fx1 _)]              ; items where this expression is not nil.\n[(or (fx1 _) (fx2 _))] ; ...\n```\n\n### Fold Operator - `?fld`\n\nReduce `vector`; or the values of a `ht`:\n  - `(?fld init fx)`: fold `(fx acc _)` with `init` as the first `acc` value.\n    `acc` is inserted as the first argument to `fx`.\n  - `(?fld init (fx .. _ ..))`: fold `(fx acc .. _ ..)`. The accumulator is\n    inserted as the first argument to `fx`.\n  - `(?fld init acc (fx .. acc .. nxt))`: fold `(fx .. acc .. nxt)`. Use this\n    if you need to name the accumulator explicity.\n\n### Group by Operator - `?grp`\n\nGroup input into a new `ht`:\n  - `(?grp expr [tx-expr])`: keys are given by `expr`, and values are given by\n    `tx-expr` (or `_`).\n\n### Recursion Operator - `?rec`\n\nRepeat the same expression while something is true:\n - `(?rec test-expr expr)`: repeat `expr` while `test-expr`. `_` refers to the\n   input value, then to the most recent evaluation of `expr`. Use `(cnt)` to\n   get the number of the current iteration. `(par)` always refers to the input\n   value.\n\n### Search Operator - `?srch`\n\nIterate a datastructure (as if with `?txpr`) and collect the matches in a new\n`vector`:\n  - `(?srch sel)`: collect `_` whenever the `Selector` matches.\n  - `(?srch sel .. expr)`: collect `expr` whenever the `Selector` matches.\n\n### Transformer Operators - `?xpr`, `?txpr`, `?mxpr`\n\nPerform operation when pattern or condition is satisfied:\n  - `(?xpr sel)`: match current value against `EXPR Selector`. Return the\n    result if not `nil`.\n  - `(?xpr sel hit-expr)`: match current value against `EXPR Selector`.\n    Evaluates `hit-expr` if not nil. `_` is the matching item.\n  - `(?xpr sel .. hit-expr miss-expr)`: match current value against `expr\n    selectors`.  Evaluate `hit-expr` if not `nil`; else evaluate `miss-expr`.\n    `_` is the matching item.\n\nRecursively traverse a nested structure of `sequences` and `hts` and return a\nnew value for each match:\n  - `(?txpr sel .. tx-expr)`: recursively traverse current value and replace\n    matches with `tx-expr`. `tx-expr` can be a function name or expression.\n    Also traverses vectors and `ht` values.\n  - `(?mxpr (sel .. tx-expr) .. (sel .. tx-expr))`: one or more matches and\n    transforms.  Performs the transform of the first match only.\n\n## Query Utilities\n\nThe internal representation of in `lqn` means you can use the regular CL\nutilities such as `gethash`, `aref`, `subseq`, `length` etc.  But for\nconvenience there are some utility functions/macros in defined in `lqn`. Some\nof them are described below. There are more in the documentation.\n\n### Global Query Context Fxs\n\nDefined in the query scope:\n - `(fi [k=0])`: counts files from `k`.\n - `(fn)`: name of the current file; or `\":internal:\"`, `\"pipe\"`.\n - `(hld k v)`: hold this value at this key in a key value store.\n - `(ghv k [d])`: get the value of this key; or `d`.\n - `(nope [d])`: stop execution, return `d`.\n - `(err [msg])`: raise `error` with `msg`.\n - `(wrn [msg])`: raise `warn` with `msg`.\n\n### Operator Context Fxs\n\nDefined in all operators:\n - `_`: the current value.\n - `(cnt)`: counts from `0` in the enclosing `Selector`.\n - `(key)`: the current `key` if the current value is a `ht`. Otherwise `(cnt)`.\n - `(itr)`: the current object in the iteration of the enclosing `Selector`.\n - `(par)`: the object containing `(itr)`.\n - `(psize)`: number of items in `(par)`.\n - `(isize)`: number of items in `(itr)`.\n\n### Generic Utilities\n\nGeneral utilities:\n - `(?? a expr [res=expr])`: execute `expr` only if `a` is not `nil`. if `expr`\n   is not nil it returns `expr` or `res`; otherwise `nil`.\n - `(fmt f ..)`: format `f` as `string` with these (`format`) args.\n - `(fmt s)`: get printed representation of `s`.\n - `(out f ..)`: format `f` to `*standard-output*` with these (`format`) args.\n   returns `nil`.\n - `(out s)`: output printed representation of `s` to `*standard-output*`.\n   returns `nil`.\n - `(msym? a b)`: compare `symbol` `a` to `b`; if `b` is a `keyword` or `symbol`\n   a perfect match is required; if `b` is a `string` it performs a substring\n   match; if `b` is an expression, `a` is compared to the evaluated value of\n   `b`.\n - `(noop ..)`: do nothing, return `nil`.\n\n### Hash-table / Strings / Vectors / Sequences\n\nFor all `sequences` and `hts`:\n - `(@* o d i ..)`: pick these indices/keys from `sequence`/`ht` into new\n   `vector`.\n - `(size? o [d])`: length of `sequence` or number of keys in `ht`.\n - `(all? o [empty])`: are all items in `sequence` something? or `empty`.\n - `(some? o [empty])`: are some items in `sequence` something? or `emtpy`.\n - `(empty? o [d])`: is `sequence` or `ht` empty?.\n - `(compct o)`: Remove `nil`, empty `vectors`, empty `hts` and keys with empty `hts`.\n\nMake or join `hts`:\n - `(cat$ ..)`: add all keys from these `hts` to a new `ht`. left to right.\n - `(new$ :k1 expr1 ..)`: new `ht` with these keys and expressions.\n\nPrimarily for `sequences` (`string`, `vector`, `list`):\n - `(new* ..)`: new `vector` with these elements.\n - `(ind* s i)`: get this index from `sequence`.\n - `(sel ..)`: get new `vector` with these `ind*s` or `seqs` from `sequence`.\n - `(seq v i [j])`: get range `i ..` or `i .. (1- j)` from `sequence`.\n - `(head s [n=10])`: first `n` items of `sequence`.\n - `(tail s [n=10])`: last `n` items of `sequence`.\n - `(cat* s ..)`: concatenate these `sequences` to a `vector`.\n - `(flatn* s [n=1] [str=nil])`: flatten `sequence` `n` times into a `vector`.\n   If `str=t` strings are flattened into individual chars as well.\n - `(flatall* s [str=nil])`: flatten all `sequences` (except `strings`) into\n   new `vector`. Use `t` as the second argument to flatten `strings` to\n   individual chars as well.\n - `(flatn$ s n)`: flatten `ht` into vector `(new* k0 v0 k1 v1 ..)`\n\nPrimarily for `string` searching. `[i]` means case insensitive:\n - `([i]pref? s pref [d])`: `s` if `pref` is a prefix of `s`; or `d`.\n - `([i]sub? s sub [d])`: `s` if `sub` is a substring of `s`; or `d`.\n - `([i]subx? s sub)`: index where `sub` starts in `s`.\n - `([i]suf? s suf [d])`: `s` if `suf` is a suffix of `s`; or `d`.\n - `(repl s from to)`: replace `from` with `to` in `s`.\n\nString maniuplation:\n - `(sup s ..)`: `str!` and upcase.\n - `(sdwn s ..)`: `str!` and downcase.\n - `(trim s)`: trim leading and trailing whitespace from `string`.\n - `(splt s x [trim=t] [prune=nil])`: split `s` at all `x` into `vector` of\n   `strings`. `trim` removes whitespace. `prune` drops empty strings.\n - `(join s x ..)`: join sequence with `x` (`strings` or `chars`), returns\n   `string`.\n - `(strcat s ..)`: concatenate these `strings`, or all `strings` in one or\n   more `sequences` of `strings`.\n\n### Type Coercion and Tests\n\n`(is? o [d])` returns `o` if not `nil`, empty `sequence`, or empty `ht`; or `d`.\n\nThese functions return the argument if the argument is the corresponding type:\n`flt?`, `int?`, `ht?`, `lst?`, `num?`, `str?`, `vec?`, `seq?`.\n\nThese functions return the argument parsed as the corresponding type if\npossible; otherwise they return the optional second argument: `int!?`, `flt!?`,\n`num!?`, `str!?`, `vec!?`, `seq!?`.\n\nThe following functions will coerce the argument, or fail if the coercion is\nnot supported: `str!`, `int!`, `flt!`, `lst!` `sym!`,\n\n## Install\n\n`lqn` requires [SBCL](https://www.sbcl.org/). And is pretty easy to install via\n`quicklisp`. SBCL is available in most package managers. And you can get\nquicklisp at https://www.quicklisp.org/beta/. Make sure `lqn` is available in\nyour `quicklisp` `local-projects` folder. Mine is at\n`~/quicklisp/local-projects/`.\n\nThen create an alias for SBCL to execute shell wrappers e.g:\n```bash\nalias jqn=\"sbcl --script ~/path/to/lqn/bin/jqn-sh.lisp\"\nalias tqn=\"sbcl --script ~/path/to/lqn/bin/tqn-sh.lisp\"\nalias lqn=\"sbcl --script ~/path/to/lqn/bin/lqn-sh.lisp\"\n```\nUnfortunately this will tend to have a high startup time. To make it run faster\nyou can create an SBCL image/core that has `lqn` preloaded and dump it using\n`sb-ext:save-lisp-and-die`. Then use the core in the alias instead of SBCL.\n\n is an example script for creating your own core. You can also preload\nyour own libraries which will be available to `lqn`.\n\nYou can see an example bash script for making your own core here[bin/core.sh](here)\n","funding_links":[],"categories":["Recently Updated","Interfaces to other package managers"],"sub_categories":["[Sep 15, 2024](/content/2024/09/15/README.md)","Third-party APIs"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finconvergent%2Flqn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finconvergent%2Flqn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finconvergent%2Flqn/lists"}