{"id":19788887,"url":"https://github.com/niklaskorz/diesl","last_synced_at":"2026-06-08T13:32:04.933Z","repository":{"id":147567812,"uuid":"465443717","full_name":"niklaskorz/diesl","owner":"niklaskorz","description":"A Nim DSL for SQL","archived":false,"fork":false,"pushed_at":"2022-03-02T19:33:20.000Z","size":6790,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-02-28T14:44:13.783Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/niklaskorz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-02T19:33:03.000Z","updated_at":"2023-01-09T18:45:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"ab78b786-25f2-4d66-9797-b5a14b58756c","html_url":"https://github.com/niklaskorz/diesl","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/niklaskorz/diesl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niklaskorz%2Fdiesl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niklaskorz%2Fdiesl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niklaskorz%2Fdiesl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niklaskorz%2Fdiesl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/niklaskorz","download_url":"https://codeload.github.com/niklaskorz/diesl/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niklaskorz%2Fdiesl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34065349,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T06:29:14.157Z","updated_at":"2026-06-08T13:32:04.917Z","avatar_url":"https://github.com/niklaskorz.png","language":"Nim","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The DieSL Language\n\n## Documentation\n\nThe documentation of the DieSL API can be found [here](https://pvs-hd.gitlab.io/ot/diesl/documentation/).\nIn particular, see the [documentation for string operations](https://pvs-hd.gitlab.io/ot/diesl/documentation/diesl/operations/strings.html).\n\n## Running the demo\n\nFirst, download and extract the [latest build of the demo](https://gitlab.com/pvs-hd/ot/diesl/-/jobs/artifacts/develop/download?job=build%20demo).\nThen, you can run the demo binary with one of the example scripts from the `example/` folder or with any other DieSL script file you have written.\nYou may also modify the demo data in `example/data.csv`.\nWhile you may change all rows and also add or remove rows, you currently can't change the structure of the demo data without updating the code of the demo in `src/diesl.nim`.\n\nThe demo binary supports two modes: `direct` and `views`.\nIn `direct` mode, the DieSL operations are translated into SQL `UPDATE` statements and the demo table is queried and printed directly.\nIn `views` mode, the operations are translated into SQL `CREATE VIEW` statements and the last created view is queried and printed.\n\nExample:\n\n```\n# Linux\n./diesl.out direct examples/basic_nim.diesl\n# Windows\n.\\diesl.exe direct examples\\basic_nim.diesl\n```\n\nIf you want to build the project from source instead of running the prebuilt binaries (e.g., if you want to run the DieSL demo on macOS), make sure you have `nim` and `nimble` installed. Then run the following from the root directory of the repository:\n\n```\nnimble update\n# nimble run -- \u003cmode\u003e \u003cfile\u003e\nnimble run -- direct examples/nim.diesl\n```\n\n## Writing DieSL Scripts\n\nDieSL is based on NimScript and thus supports most of Nim's language features.\nTo express data manipulations, DieSL offers two APIs: a programmatic API that uses Nim's function calls to stay as close to the language as possible, and a natural API that is based on a macro block called `change`.\n\n### The Nim Way\n\nIn the programmatic API, tables can be accessed as `db.tableName` and columns as `db.tableName.columnName`. To assign values to a column, use the assignment operator `db.tableName.columnName = newValue`.\nYou can also assign to multiple columns at once, for example when you want to assign the same value to multiple columns (`db.tableName[columnA, columnB] = \"same value!\"`) or use an operation that returns multiple values (`db.tableName[columnA, columnB] = db.tableName.columnC.split(\",\")`).\n\nHere is an overview of supported operations (executable example in `examples/all_nim.diesl`):\n\n```nim\n# Store a value in table \"students\" column \"name\"\ndb.students.name = \"Hello world\"\n# Load a value from column \"firstName\" and store in \"name\"\ndb.students.name = db.students.firstName\n# Concat two columns with a space inbetween and store in name\ndb.students.name = db.students.firstName \u0026 \" \" \u0026 db.students.secondName\n# Trim whitespace character on left and right of name column\ndb.students.name = db.students.name.trim() # or .trim(both)\n# Trim whitespace character on left of name column\ndb.students.name = db.students.name.trim(left)\n# Trim whitespace character on right of name column\ndb.students.name = db.students.name.trim(right)\n# Take only characters from zero-based index 2 to 5 from name\ndb.students.name = db.students.name[2..5]\n# Replace firstName in name with \"Mr. \"\ndb.students.name = db.students.name.replace(db.students.firstName, \"Mr. \")\n# Replace pairs in column name\ndb.students.name = db.students.name.replaceAll(@{\n  db.students.firstName: \"Mr. \",\n  db.students.secondName: \"Hello\"\n})\n# Replace value in column with empty string\ndb.students.name = db.students.name.remove(\"some swear word\")\n# Lower case the whole string\ndb.students.name = db.students.name.toLower()\n# Upper case the whole string\ndb.students.name = db.students.name.toUpper()\n# Extract first occurence of pattern\ndb.students.name = db.students.name.extractOne(\"{hashtag}\")\n# Extract groups with pattern\ndb.students[firstName, secondName] = db.students.name.extractAll(\"([a-z]+) ([a-z]+)\")\n# Replace pattern in name with \"Mr. \"\ndb.students.name = db.students.name.patternReplace(\"{email}\", \"Mr. \")\n# Replace pattern pairs in column \"name\"\ndb.students.name = db.students.name.patternReplaceAll(@{\n  \"{email}\": \"Mr. \",\n  \"[a-z]+\": \"Hello\",\n  \"{hashtag}\": \"there\",\n})\n# Split column\ndb.students[firstName, secondName] = db.students.name.split(\" \")\n```\n\n### The Natural Way\n\nTo make it easier to get started with DieSL, our naturally looking macro-based API can be used instead.\nThis allows using DieSL even if you are unfamiliar with programming.\nThe base of this API is the `change` block that operates on a table, for example:\n\n```nim\n# Modify table \"students\"\nchange db.students:\n  # Remove whitespace from end of column \"firstName\"\n  trim ending of firstName\n  # Remove any occurence of \"bad word\" in column \"secondName\"\n  remove \"bad word\" from secondName\n```\n\nIf you only want to apply operations to a single column, you can also use this shorter variant:\n\n```nim\n# Modify column \"firstName\" of table \"students\"\nchange firstName of db.students:\n  # Remove whitespace from end of column\n  trim ending\n  # Remove any occurence of \"bad word\" in column\n  remove \"bad word\"\n```\n\nHere is an overview of supported operations (executable example in `examples/all_natural.diesl`):\n\n```nim\nchange db.students:\n  # Trim whitespace character on left and right of name column\n  trim name\n  # Trim whitespace character on left of name column\n  trim beginning of name\n  # Trim whitespace character on right of name column\n  trim ending of name\n  # Take only characters from 1-based index 3 to 6 (zero-based index 2 to 5) from name\n  take 3 to 6 from name\n  # Replace \"Señor \" in name with \"Mr. \"\n  replace \"Señor \" with \"Mr. \" in name\n  # Replace pairs in column name\n  replace in name:\n    \"Señor \" with \"Mr. \"\n    \"Hello\" with \"there\"\n  # Replace value in column with empty string\n  remove \"some swear word\" from name\n  # Extract first occurence of pattern\n  extract \"{hashtag}\" from name\n  # or:\n  extract hashtag from name\n  # Extract groups with pattern\n  extract \"([a-z]+) ([a-z]+)\" from name into firstName and secondName\n  # Replace pattern in name with \"Mr. \"\n  replace pattern \"{email}\" with \"Mr. \" in name\n  # Replace pattern pairs in column \"name\"\n  replace patterns in name:\n    \"{email}\" with \"Mr. \"\n    \"[a-z]+\" with \"\u003csecondName\u003e\"\n    \"{hashtag}\" with \"there\"\n  # Split column\n  split name on \" \" into firstName, secondName\n```\n\n## Running DieSL Scripts\n\nTo run DieSL scripts inside your own application, use this repository as a package through nimble:\n\n```nim\nrequires \"https://gitlab.com/pvs-hd/ot/diesl.git \u003e= 0.4.0\"\n\nThen, use the DieSL modules:\n\n```nim\nimport db_sqlite\nimport diesl\n# needed for functions like pattern extraction\nimport diesl/extensions/sqlite\n\n# Open your sqlite database\nlet db = open(...)\n\n# Install DieSL sqlite extensions\ndb.installCommands()\n\n# Define your database schema\nlet schema = newDatabaseSchema({ ... })\n\n# Read your user's script\nlet scriptSource = ...\n\n# Run script and receive result\nlet exportedOperations = runScript(scriptSource, schema)\n\n# Option A: Translate operations to Sqlite UPDATE statements\nlet queries = exportedOperations.toSqlite()\n\n# Option B: Translate operations to Sqlite CREATE VIEW statements\nlet (queries, tableAccessMap, views) = exportedOperations.toSqliteViews(schema)\n# tableAccessMap.getTableAccessName(tableName) returns the last view name created for a certain table\n\n# Run generated queries\nfor query in queries:\n  db.exec(query)\n\n# Close your sqlite database\ndb.close()\n```\n\n## How it works\n\nDieSL code consists of one or more change macros (which is defined in [transpilation.nim](src/diesl/syntax/transpilation.nim)). When a script is passed into the [runScript](src/diesl/script.nim#L127) function it is executed in a NimVM where the macro is expanded. The [change macro](src/diesl/syntax/transpilation.nim#L58) looks for all Nim statements that match a pattern of the DieSL commands and translates them to their Nim counterpart - all Nim statements inbetween stay the same. After that the code is evaluated by the Nim interpreter. __Important:__ This does not execute any changes on the database it just creates an [object](src/diesl/operations/base.nim#L16) representing what operations should take place. This object is then retrieved from the VM and translated to the target language (currently only SQLite).\n\n## Repository Structure\n\nTests are located in the tests folder (duh). All library logic is implemented under `src/diesl`.\n\n- backends: Anything related to SQL generation goes here\n- compat: Anything related to compability with other modules goes here (things like conversion functions)\n- extensions: Everything here will be compiled to C and registered to SQLite and can be used in queries\n- transpilation.nim: Contains everything for the parsing of the natural syntax\n- operations: Defines the very important operation datatype and its functions. A operation represents anything that can be done in DieSL.\n- script.nim: Takes care of executing DieSL script in the NimVM and retrieving the operations from there\n\n## Extending DieSL with New Operations\n\nDieSL's operations are defined by the [DieslOperation](src/diesl/operations/types.nim#L49) object.\nSpecifically, this type is an enum that is controlled by the [DieslOperationType](src/diesl/operations/types.nim#L6); each DieslOperationType indicates a unique semantic, and as such, each DieslOperation variant can store different data according thereto.\n\n\n### Defining a new DieslOperation\n\nFor example, if a new operation for trimming strings is to be introduced, a new DieslOperationType called `dotTrim` can be defined\n\n```nim\ntype DieslOperationType* = enum\n  dotStore\n  dotLoad\n  ...\n  dotTrim\n```\n\nwhich enables the DieslOperation object to be accordingly extended:\n```nim\ntype DieslOperation* = ref object\n  case kind*: DieslOperationType\n    of dotStore:\n      ...\n    of dotLoad:\n      ...\n    of dotTrim:\n      # string data that is to be trimmed\n      trimValue*: DieslOperation\n      # from which side should the string be trimmed\n      trimDirection*: TextDirection\n```\n\nDieSL defines multiple visitors on DieslOperations that have to be extended when a new variant is introduced, namely [withAccessIndex](src/diesl/operations/accessindex.nim#L5), [collectTableAccesses](src/diesl/operations/boundaries.nim#L5), [collectLoads](src/diesl/operations/optimizations.nim#L8) and [toDataType](src/diesl/operations/types.nim#L122).\n\n\n### Adding Syntax for the new Operation\n\nThe natural syntax of Diesl is defined in [transpilation.nim](src/diesl/syntax/transpilation.nim). To add a new one it needs to be added in [transpileCommand](src/diesl/syntax/transpilation.nim#15). For parsing we use [pattern matching](https://nim-lang.github.io/fusion/src/fusion/matching.html) on the AST. If the command cannot be parsed it is important that the old command is returned instead per default. At the moment you need to account for both variants of a operation (one where the column is given directly and one where the column is given in the first line of the change block), you can see this in the other operations as well. We tried our hand at smarter solutions and failed to problems with how Nim's type system handles generics.\n\nCurrently, the project's sole backend targets SQLite.\nTo this end, the functionality for each DieslOperation can be implemented directly in SQLite's SQL dialect, or compiled into a native binary, loaded into SQLite at startup and accordingly referenced.\n\n### Simple Implementation: SQLite's SQL dialect\n\nContinuing the running example of trimming strings, the functionality can be implemented by expanding the definition of the [toSqlite](src/diesl/backends/sqlite.nim#L27) procedure.\nIn this case, SQLite's SQL dialect offers built-in scalar functions from [trimming from the left](https://www.sqlite.org/lang_corefunc.html#ltrim), [from the right](https://www.sqlite.org/lang_corefunc.html#rtrim) and from [both sides](https://www.sqlite.org/lang_corefunc.html#trim).\nAs such, the implementation for trimming can simply return an SQL string containing the function call to the correct trimmer:\n\n```nim\nproc toSqlite*(op: DieslOperation): string =\n  case op.kind:\n    of dotStore:\n      ...\n    of dotLoad:\n      ...\n    of dotTrim:\n      # Derive which kind of trimming is desired\n      let trimFunction = case op.trimDirection:\n        of TextDirection.left:\n          \"LTRIM\"\n        of TextDirection.right:\n          \"RTRIM\"\n        of TextDirection.both:\n          \"TRIM\"\n\n      # This is an interpolated string, substituting expressions\n      # into their corresponding placeholders within the string.\n      fmt\"{trimFunction}({op.trimValue.toSqlite})\"\n\n      # The `op.trimValue.toSqlite` expression recurses into the\n      # `trimValue` member for its own SQL generation\n```\n\n\n### Advanced Implementation: exportToSqlite3\n\nMore complex actions are tricky to implement or cannot be solely implemented in SQL.\nTo this end, [extension methods](src/diesl/extensions/sqlite.nim) can be implemented.\nThis approach was deemed viable for string padding, where Nim provides corresponding methods in the standard library and SQLite does not.\n\nFirst, the scalar [padding](src/diesl/extensions/sqlite.nim#L37) procedure is defined in Nim.\nIt is important that the procedure have the `exportToSqlite3` pragma applied, so that [installCommands](src/diesl/extensions/sqlite.nim#L8) can register the procedures during initialisation.\nFinally, the SQLite codegen shall generate SQL that calls the padding procedure as if it were a builtin SQLite function:\n\n```nim\nproc toSqlite*(op: DieslOperation): string {.gcSafe.} =\n  case op.kind:\n    of dotPadString:\n      # Derive direction and character padding variables from the operation\n      let direction = ...\n      let padWith = ...\n\n      # Generate the corresponding SQL call\n      fmt\"padding({op.padStringValue.toSqlite}, {direction}, {op.padStringCount}, {padWith})\"\n```\n\n\n## Side Projects\n\n- [Making Nim functions available to SQLite](https://github.com/niklaskorz/nim-exporttosqlite3/)\n- [Tracking TODOs in the project](https://github.com/preslavmihaylov/todocheck/pull/160)\n\n\n\n## Accounting\n\n* Niklas Korz: 40% + aforementioned exportage library, in particular:\n  * SQL generation and optimization\n* Benjamin Sparks: 30%, in particular:\n  * String and Regex operations + Documentation\n* Samuel Melm: 30%, in particular:\n  * Natural Syntax Parsing (macros)\n\nAll aspects of requirements, main implementation, testing etc., were shared amongst the team and assigned during meetings in the Issue Tracker.\nAccordingly, this means that the responsibility of main and test code was shared equally.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniklaskorz%2Fdiesl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fniklaskorz%2Fdiesl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniklaskorz%2Fdiesl/lists"}