{"id":20673734,"url":"https://github.com/cpg314/sqlsonnet","last_synced_at":"2025-04-13T17:13:39.516Z","repository":{"id":245760119,"uuid":"819132394","full_name":"cpg314/sqlsonnet","owner":"cpg314","description":"Express SQL queries with a simple Jsonnet representation, which can be easily templated using the Jsonnet configuration language.","archived":false,"fork":false,"pushed_at":"2025-02-26T22:20:30.000Z","size":861,"stargazers_count":1,"open_issues_count":6,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T17:13:31.064Z","etag":null,"topics":["jsonnet","rust","sql"],"latest_commit_sha":null,"homepage":"https://cpg314.github.io/sqlsonnet/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpg314.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-23T21:43:26.000Z","updated_at":"2025-02-26T22:20:34.000Z","dependencies_parsed_at":"2024-10-23T04:52:07.575Z","dependency_job_id":"be0ea70f-d84d-428e-bff6-fccf6e803650","html_url":"https://github.com/cpg314/sqlsonnet","commit_stats":null,"previous_names":["cpg314/sqlsonnet"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsqlsonnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsqlsonnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsqlsonnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsqlsonnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpg314","download_url":"https://codeload.github.com/cpg314/sqlsonnet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248750127,"owners_count":21155687,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jsonnet","rust","sql"],"created_at":"2024-11-16T20:42:23.391Z","updated_at":"2025-04-13T17:13:39.479Z","avatar_url":"https://github.com/cpg314.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"![sqlsonnet](logo.png)\n\n\u003e [!WARNING]  \n\u003e Work in progress.\n\nExpress SQL queries with a simple [Jsonnet](https://jsonnet.org/) representation, which can be easily templated using the [Jsonnet configuration language](https://jsonnet.org/learning/tutorial.html).\n\nFor example, the following Jsonnet input\n\n```jsonnet\nselect: {\n  fields: ['bwv', u.count()],\n  from: 'cantatas',\n} + { groupBy: ['year'], limit: 10 }\n```\n\nproduces the following SQL query\n\n```sql\nSELECT bwv, count(*) AS c\nFROM cantatas\nGROUP BY year\nLIMIT 10;\n```\n\nThis becomes particularly useful when working with sets of complex queries, which can be created and maintained in a composable way, like source code, benefiting from the Jsonnet ecosystem (language server, formatter, linter, editor integrations).\n\nIn-browser playground/demo with [WebAssembly](https://en.wikipedia.org/wiki/WebAssembly): see https://cpg314.github.io/sqlsonnet/\n\n### Features\n\n- Command-line interface to convert sqlsonnet to SQL, and vice-versa.\n  - Good error reporting thanks to [`miette`](https://docs.rs/miette/latest/miette/index.html).\n- Compatible with existing Jsonnet tools (formatter, LSP).\n- Proxy server for Clickhouse (HTTP interface), supporting incoming sqlsonnet or SQL.\n  - Shared library and prelude.\n  - Response caching (similarly to [`chproxy`](https://www.chproxy.org/)).\n  - Interactive playground.\n- Bindings for [WebAssembly](https://en.wikipedia.org/wiki/WebAssembly).\n- In-browser [interactive playground](https://cpg314.github.io/sqlsonnet/), using the WebAssembly bindings.\n\n## Installation\n\nThe [releases page](https://github.com/cpg314/sqlsonnet/releases) contains binaries (simple `tar` archive, Debian/Ubuntu `.deb` package, ArchLinux `.pkg`).\n\nA [Docker image](https://github.com/cpg314/sqlsonnet/pkgs/container/sqlsonnet) with `sqlsonnet` and `sqlsonnet_clickhouse_proxy` is also available at `ghcr.io/cpg314/sqlsonnet:0.1.1`.\n\nAlternatively, see below for building from source.\n\n### Recommended tools\n\n- The [jsonnet language server](https://github.com/grafana/jsonnet-language-server).\n- The [jsonnetfmt formatter](https://github.com/google/go-jsonnet/tree/master).\n- The jsonnet integration for your favourite editor.\n\nSee also the official [Jsonnet Tools page](https://jsonnet.org/learning/tools.html).\n\n## Command line interface\n\nThe `sqlsonnet` command line interface converts Jsonnet statements to (and to a lesser extent from) SQL.\n\n```\nUsage: sqlsonnet [OPTIONS] \u003cINPUT\u003e\n\nArguments:\n  \u003cINPUT\u003e  Input file (path or - for stdin)\n\nOptions:\n      --theme \u003cTHEME\u003e\n          Color theme for syntax highlighting [env: SQLSONNET_THEME=Nord] [possible values: 1337, Coldark-Cold, Coldark-Dark, DarkNeon, Dracula, GitHub, \"Monokai Extended\", \"Monokai Extended Bright\", \"Monokai Extended Light\", \"Monokai Extended Origin\", Nord, OneHalfDark, OneHalfLight, \"Solarized (dark)\", \"Solarized (light)\", \"Sublime Snazzy\", TwoDark, \"Visual Studio Dark+\", ansi, base16, base16-256, gruvbox-dark, gruvbox-light, zenburn]\n  -c, --compact\n          Compact SQL representation\n  -f, --from-sql\n          Convert an SQL file into Jsonnet\n      --diff\n          With --from-sql: Convert back to SQL and print the differences with the original, if any\n      --display-format \u003cDISPLAY_FORMAT\u003e\n          [possible values: sql, jsonnet, json]\n      --clickhouse-url \u003cCLICKHOUSE_URL\u003e\n          Clickhouse HTTP URL, to execute queries [env: SQLSONNET_CLICKHOUSE=]\n  -w, --watch\n          Watch for file changes\n  -J, --jpath \u003cJPATH\u003e\n          Library path [env: JSONNET_PATH=]\n  -e, --execute\n          Send query to Clickhouse proxy (--proxy-url) for execution\n  -h, --help\n          Print help\n  -V, --version\n          Print version\n```\n\n#### Jsonnet to SQL\n\n```console\n$ sqlsonnet test.jsonnet\n$ # stdin input is also supported\n$ cat test.jsonnet | sqlsonnet -\n$ # Piping into clickhouse client\n$ sqlsonnet test.jsonnet | clickhouse client -f PrettyMonoBlock --multiquery --host ... --user ...\n```\n\nThe input should represent a list of queries, e.g.\n\n```jsonnet\n[ { select: { ... } } ]\n```\n\nThe [embedded utility functions](sqlsonnet/sqlsonnet.libsonnet) are automatically imported as\n\n```jsonnet\nlocal u = import \"sqlsonnet.libsonnet\";\n```\n\n#### SQL to Jsonnet (`from-sql`)\n\n```console\n$ sqlsonnet --from-sql test.sql\n$ cat test.sql | sqlsonnet --from-sql -\n```\n\nThis mode is useful to discover the sqlsonnet syntax from SQL queries.\n\nThe parser is far from perfect. Expressions are parsed as long as subqueries are encountered; then they are simply represented as strings. The results do not use the [embedded utility functions](sqlsonnet/sqlsonnet.libsonnet), which can significantly simplify expressions.\n\n## As a Rust library\n\n```rust\nuse sqlsonnet::{Query, sqlsonnet_query, jsonnet::Options};\n\n// This performs compile-time syntax checking\nlet query: Query = sqlsonnet_query!({ select: { fields: [\"name\", \"age\"], from: \"contacts\" } }).unwrap();\n// Convert to SQL\nassert_eq!(query.to_sql(true), \"SELECT name, age FROM contacts\");\n```\n\n## Syntax\n\n```jsonnet\n[\n  {\n    select: {\n      // List of expressions\n      fields: [\n        // Primitive types\n        1,\n        1.0,\n        true,\n        '\"string\"',\n        // Column reference\n        'col',\n        // Aliased expression\n        u.as('col', 'alias'),\n        // Operator, equivalent to [1, \"+\", 2]\n        u.op('+', [1, 2]),\n        // Equivalent to u.op(\"=\", [1, 2])\n        u.eq(1, 2),\n        // Function, equivalent to {fn: \"count\", params: [\"*\"]}\n        u.fn('count', ['*']),\n        // Like operator\n        u.like('text', u.string('%t%')),\n      ],\n      // From expression (optional)\n      from: 'a',\n      // List of expressions (optional)\n      groupBy: [],\n      // List of joins (optional)\n      joins: [\n        // From expression and ON (list of boolean expressions)\n        { from: 'b', on: ['f1=f2'] },\n        // From expression and USING (list of column identifiers)\n        { from: 'c', using: ['f'] },\n        // An empty `on` or `using` parameter results in a CROSS JOIN\n        { from: 'd', using: [] },\n        // Other types of joints\n        { from: 'e', using: ['f'], kind: 'left-outer' },\n      ],\n      // Expression (optional). Use u.and, u.or to combine.\n      having: true,\n      // Expression (optional). Use u.and, u.or to combine.\n      where: true,\n      // List of identifiers or { expr: identifier, order: \"desc\" } or { expr: identifier, order: \"asc\" }\n      orderBy: ['col1', { expr: 'col2', order: 'desc' }, { expr: 'col3', order: 'asc' }],\n      // Integer (optional)\n      limit: 100,\n      // List of expressions (optional)\n      settings: ['join_algorithm=\"parallel_hash\"'],\n    },\n  },\n]\n```\n\nA `From` expression can be either:\n\n```jsonnet\n// Table name\nfrom: 'a',\n// Aliased table name\nfrom: { table: 'a', as: 'b' },\n// Subquery with optional alias\nfrom: { fields: ['*'], from: 'b', as: 'c' },\n```\n\nExpressions (used in `fields`, `groupBy`, `on`, `having`, `orderBy`) are defined recursively with primitive types, column references, aliases, operators, and functions.\n\n### Combining expressions\n\nUse the `+:` operator to add fields or JOINs to an existing query; using the `+` operator would overwrite the existing values.\n\n```jsonnet\nu.select(\n  {\n    fields: [0],\n    from: 'a',\n    joins: [{ from: 'b', using: ['col1'] }],\n  } + {\n    fields+: [1],\n    joins+: [{ from: 'c', using: ['col2'] }],\n  }\n),\n```\n\nSimilarly the `u.where_and` (resp. `u.having_and`) utilities to add `WHERE` (resp `HAVING`) conditions. These essentially change the `where` field to `where: u.and([super.where, expr])`.\n\n```jsonnet\nu.select(\n  {\n    fields: [0],\n    from: 'a',\n    where: u.eq(1, 1),\n  } + u.where_and([u.ge(2, 1)]),\n),\n```\n\n## Database proxies\n\nThe database proxies convert Jsonnet requests into SQL, before sending them to the database server and returning the response.\n\n```mermaid\nsequenceDiagram\n participant Client\n participant Proxy\n participant Database\n Client-\u003e\u003eProxy: Jsonnet\n Proxy-\u003e\u003eDatabase: SQL\n Database-\u003e\u003eProxy: Response\n Proxy-\u003e\u003eClient:Response\n```\n\nThey also support:\n\n- Caching responses to previous requests.\n- Custom import paths.\n- Prepending all queries with a prelude (e.g. to make available libraries in the import path).\n- Serving an interactive playground where users can enter Jsonnet, and see the generated SQL as well as the database response.\n\n### `sqlsonnet_clickhouse_proxy`\n\n\u003e [!WARNING]  \n\u003e The current implementation assumes a fully trusted environment.\n\n```text\nReverse proxies a Clickhouse HTTP server, transforming Jsonnet or JSON queries into SQL\n\nUsage: sqlsonnet_clickhouse_proxy [OPTIONS] --url \u003cURL\u003e --username \u003cUSERNAME\u003e --port \u003cPORT\u003e\n\nOptions:\n      --url \u003cURL\u003e            [env: CLICKHOUSE_URL=]\n      --username \u003cUSERNAME\u003e  Clickhouse username [env: CLICKHOUSE_USERNAME=]\n      --password \u003cPASSWORD\u003e  [env: CLICKHOUSE_PASSWORD=]\n      --cache \u003cCACHE\u003e\n      --library \u003cLIBRARY\u003e    Folder with Jsonnet library files\n      --shares \u003cSHARES\u003e      Folder with shared snippets\n      --prelude \u003cPRELUDE\u003e    Prepended to all requests\n      --port \u003cPORT\u003e\n  -h, --help                 Print help (see more with '--help')\n  -V, --version              Print version\n```\n\nFeatures:\n\n- Caching\n- Library shared across clients\n- Prelude\n- End-to-end compression: when the `Accept-Encoding` header is set by the client, the server directly forwards the encoded data.\n\n## Implementation details\n\n### Jsonnet to SQL\n\nInput Jsonnet is interpreted as JSON using the [jrsonnet crate](https://github.com/CertainLach/jrsonnet), and then mapped via [serde](https://serde.rs/) into an internal representation of queries, which can finally be pretty-printed as SQL.\n\n```mermaid\nflowchart LR\n Jsonnet --jrsonnet--\u003e JSON --serde--\u003e I[Internal representation] --\u003e SQL\n```\n\n### SQL to Jsonnet\n\nInput SQL is parsed into the internal representation using the [pest PEG parser](https://pest.rs/), which can then be converted to JSON using serde, and finally printed as Jsonnet.\n\n```mermaid\nflowchart LR\n SQL --pest--\u003e I[Internal representation] --serde--\u003e JSON --\u003e Jsonnet\n```\n\n## Development\n\nInstall [cargo make](https://github.com/sagiegurari/cargo-make).\n\n### Building from source\n\n```\n$ cargo make packages\n$ # To also build a docker image:\n$ cargo make docker\n$ # Build wasm bindings\n$ cargo make wasm\n$ # Build playground\n$ cargo make playground-wasm\n```\n\n### Running checks and tests\n\nInstall [checkalot](https://github.com/cpg314/checkalot) and run:\n\n```\n$ cargo checkalot\n```\n\nThis runs in particular\n\n```\n$ cargo make docker-compose\n$ cargo nextest run --workspace -r\n```\n\n### `jrsonnet` version\n\nVia the feature flags `jrsonnet-95` and `jrsonnet-96` (default), the crate supports both:\n\n- the latest published version of `jrsonnet` on crates.io, 0.5.0-pre95\n- the [soon-to-be-released](https://github.com/CertainLach/jrsonnet/issues/70) 0.5.0-pre96 version. This is not available when the crate is retrieved from crates.io.\n\n## TODO\n\n- Proxy:\n  - Cache control: expire cache entries, etc.\n  - Support Postgres\n- Prevent representation of invalid SQL.\n- Support more query types than `SELECT`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fsqlsonnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpg314%2Fsqlsonnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fsqlsonnet/lists"}