{"id":19491716,"url":"https://github.com/rudderlabs/rudder-json-template-engine","last_synced_at":"2025-04-08T03:16:44.133Z","repository":{"id":88157795,"uuid":"563273262","full_name":"rudderlabs/rudder-json-template-engine","owner":"rudderlabs","description":"JSON path based template language for data transformations","archived":false,"fork":false,"pushed_at":"2025-03-07T06:19:11.000Z","size":1478,"stargazers_count":24,"open_issues_count":0,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-03T02:37:54.355Z","etag":null,"topics":["json","json-filters","json-path","jsonpath","jspath","language","template-engine","templates","typescript"],"latest_commit_sha":null,"homepage":"https://transformers-workflow-engine.rudderstack.com/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rudderlabs.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-08T09:15:48.000Z","updated_at":"2025-03-07T06:19:14.000Z","dependencies_parsed_at":"2024-05-30T11:53:06.804Z","dependency_job_id":"93cb993b-eb34-414b-a590-7070cbf1e607","html_url":"https://github.com/rudderlabs/rudder-json-template-engine","commit_stats":{"total_commits":266,"total_committers":13,"mean_commits":20.46153846153846,"dds":"0.16917293233082709","last_synced_commit":"19d94c2254cdc33468decc8805b3dacb2c263325"},"previous_names":[],"tags_count":69,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-json-template-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-json-template-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-json-template-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-json-template-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rudderlabs","download_url":"https://codeload.github.com/rudderlabs/rudder-json-template-engine/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247767236,"owners_count":20992548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","json-filters","json-path","jsonpath","jspath","language","template-engine","templates","typescript"],"created_at":"2024-11-10T21:17:53.868Z","updated_at":"2025-04-08T03:16:44.114Z","avatar_url":"https://github.com/rudderlabs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://rudderstack.com/\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/59817155/121357083-1c571300-c94f-11eb-8cc7-ce6df13855c9.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cb\u003eThe Customer Data Platform for Developers\u003c/b\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003e\n    \u003ca href=\"https://rudderstack.com\"\u003eWebsite\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/rudderlabs/rudder-json-template-engine/blob/main/docs/syntax.md\"\u003eDocumentation\u003c/a\u003e\n    ·\n    \u003ca href=\"https://rudderstack.com/join-rudderstack-slack-community\"\u003eCommunity Slack\u003c/a\u003e\n  \u003c/b\u003e\n\u003c/p\u003e\n\n---\n\n# JSON Template Engine\n\n## Overview\n\nWelcome to our JSON Template Engine! This powerful tool simplifies transforming JSON data from one format to another, making managing and maintaining complex integrations easier.\n\n### Why JSON Template Engine?\n\nAs an integration platform supporting over 200 integrations, we understand the challenges of maintaining and optimizing these connections. Traditionally, we used native JavaScript code for data transformation, which required significant effort and maintenance. While JSONata offered a more efficient way to manipulate JSON data, we still encountered performance bottlenecks due to its parsing and interpretation overhead.\n\n### Our Solution\n\nTo address these challenges, we've developed our own JSON Transformation Engine. This engine generates optimized JavaScript code from transformation templates, reducing runtime overhead and significantly improving performance.\n\n## Key Features\n\n- **Efficiency**: Our engine generates JavaScript code that minimizes parsing and interpretation overhead, ensuring faster execution.\n\n- **Extensibility**: Easily add new transformation templates to meet your specific integration needs.\n\n- **Simplicity**: Write concise transformation templates that are easy to understand and maintain.\n\n## Implementation\n\nThis library generates a javascript function code from the template and then uses the function to evaluate the JSON data. It outputs the javascript code in the following stages:\n\n1. [Lexing](src/lexer.ts) (Tokenization)\n1. [Parsing](src/parser.ts) (AST Creation)\n1. [Translation](src/translator.ts) (Code generation)\n\n```mermaid\n  flowchart TD;\n      A[Code] --\u003e B[Convert code to tokens];\n      B --\u003e C[Parse tokens to create Expressions];\n      C --\u003e D[Combine expressions to create statements];\n      D --\u003e E[Combine statements to create AST];\n      E --\u003e F[Translate AST to JS code]\n```\n\n[Engine](src/engine.ts) class abstracts the above steps and provides a convenient way to use the json templates to evaluate the inputs.\n\n## Getting started\n\n### Use npm package\n\n`npm install @rudderstack/json-template-engine`\n\n```ts\nconst { JsonTemplateEngine } = require('@rudderstack/json-template-engine');\nconst engine = JsonTemplateEngine.create(`'Hello ' + .name`);\nengine.evaluate({ name: 'World' }); // =\u003e 'Hello World'\n```\n\n### Use CDN URL directly in the browser\nLatest URL: https://cdn.jsdelivr.net/npm/@rudderstack/json-template-engine/build/json-template.min.js\n\u003c!-- x-release-please-start-version --\u003e\nVersioned URL: https://cdn.jsdelivr.net/npm/@rudderstack/json-template-engine@0.19.5/build/json-template.min.js \n\n```html\n\u003cscript type=\"module\"\u003e\n      import { JsonTemplateEngine } from 'https://cdn.jsdelivr.net/npm/@rudderstack/json-template-engine@0.19.5/build/json-template.min.js';\n      const engine = JsonTemplateEngine.createAsSync(`'Hello ' + .name`);\n      engine.evaluate({ name: 'World' });\n\u003c/script\u003e\n```\n\u003c!-- x-release-please-end --\u003e\n\nRefer this [example](/index.html) for more details.\n\n[Demo](https://rudderlabs.github.io/rudder-json-template-engine/)\n\n### Playground\nGive the JSON template engine a try in our [playground](https://transformers-workflow-engine.rudderstack.com/#/json-template) without needing to install anything.\n\n## Features\n\nThe template consists of multiple statements, with the output being the result of the final statement.\n\n\n### Variables\n\n```js\nconst a = 1;\nlet b = a + 2;\na + b;\n```\n\nRefer this [example](test/scenarios/assignments/template.jt) for more details.\n\n#### Template Strings\n\n```js\nlet a = `Input a=${.a}`; \nlet b = `Input b=${.b}`;\n`${a}, ${b}`;\n```\nRefer this [example](test/scenarios/template_strings/template.jt) for more details.\n\n### Basic Expressions\n\n#### Conditions\n\n```js\na \u003e b ? a : c;\n```\n\nRefer this [example](test/scenarios/conditions/template.jt) for more details.\n\n#### Comparisons\n\n```js\na === b || c \u003e d;\n```\n\nRefer this [example](test/scenarios/comparisons/template.jt) for more details.\n\n#### Math Operations\n\n```js\n10 - 2 + 2 * 10;\n```\n\nRefer this [example](test/scenarios/math/template.jt) for more details.\n\n#### Logical operations\n\n```js\nfalse || true;\n```\n\nRefer this [example](test/scenarios/logics/template.jt) for more details.\n\n### Input and Bindings\n\nInput refers to the JSON document we would like to process using a template. Bindings refer to additional data or functions we would provide to process the data efficiently.\n\nExample:\n\n- Template: `\"Hello \" + (.name ?? $.defaultName)`\n- Evaluation: `engine.evaluate({name: 'World'}, {defaultName: 'World'});`\n- `{name: 'World'}` is input.\n  - `^.name` refers to \"name\" property of the input. We can also use `.name` to refer the same. `^` always refers to the root of the input and `.` refers to current context. Refer this [example](test/scenarios/selectors/context_variables.jt) for more details.\n- `{defaultName: 'World'}` is bindings.\n  - `$.defaultName` refers to \"defaultName\" property of the bindings. Refer this [example](test/scenarios/bindings/template.jt) for more details.\n\n### Arrays\n\n```js\nlet arr = [1, 2, 3, 4]\nlet a = arr[1, 2] // [2, 3]\nlet b = arr[0:2] // [1, 2]\nlet c = arr[-2:] // [3, 4]\n```\n\nRefer this [example](test/scenarios/arrays/template.jt) for more details.\n\n### Objects\n\n```js\nlet key = \"some key\"\n// { \"a\": 1, \"b\": 2, \"c\": 3, \"some key\": 4 }\nlet obj = {a: 1, b: 2, c: 3, [key]: 4 }\nlet a = obj[\"a\"] // 1\nlet b = obj.a // 1\nlet c = obj{[\"a\", \"b\"]} // { \"a\": 1, \"b\": 2}\nlet d = obj{~[\"a\", \"b\"]} // { \"c\": 3, \"some key\": 4}\n```\n\nRefer this [example](test/scenarios/objects/template.jt) for more details.\n\n#### Object Context Props\n```js\nlet obj = {a: 1, b: 2, c: 3 };\nobj.({\n  @e [e.key]: e.value * e.value, // @e refers to each key, value pairs,\n  d: 16 // we can have other props also\n})  // { a: 1, b: 4, c: 9, d: 16}\n```\nRefer this [example](test/scenarios/objects/context_props.jt) for more details.\n\n### Functions\n\n#### Normal functions\n\n```js\nlet fn = function (arg1, arg2) {\n  arg1 + arg2;\n};\n```\n\nThe result of the last statement of function will be returned as result of the function. We can also use rest params (`...args`).\n\n#### Lambda/Short functions\n\n```js\nlet fn = array.map(lambda 2 * ?0);\n```\n\nThis function gets converted to:\n\n```js\nlet fn = array.map(function (args) {\n  2 * args[0];\n});\n```\n\nLambda functions are short to express the intention and it is convenient sometimes.\n\n#### Async functions\n\n```js\nlet fn = async function (arg1, arg2) {\n  const result = await doSomethingAsync(arg1, arg2);\n  doSomethingSync(result);\n};\n```\n\n**Note:** When we want to use async functions then we need to create template engine using `JsonTemplateEngine.create`. If you create a template this way then it will be created as an async function so we can `await` anywhere in the template.\n\n```js\nlet result = await doSomething(.a, .b)\n```\n\nRefer this [example](test/scenarios/functions/template.jt) for more details.\n\n### Paths\n\nPaths are used to access properties in `input`, `bindings` and `variables`.\n\n#### Simple Paths\n\nSimple paths support limited path features and get translated as direct property access statements in the generate javascript code.\n`a.b.c` gets translated to `a?.b?.c` so they are very fast compared to [Rich paths](#rich-paths). Simple paths are ideal when we know the object structure.\n\n**Supported features:**\n\n- [Simple Selectors](#simple-selectors)\n- [Single Index Filters](#single-index-or-property-filters)\n  Refer this [example](test/scenarios/paths/simple_path.jt) for more details.\n\n#### Rich Paths\n\nRich paths gets converted complex code to support different variations in the data.\n\nIf we use this rich path`~r a.b.c` then it automatically handles following variations.\n\n- `[{\"a\": { \"b\": [{\"c\": 2}]}}]`\n- `{\"a\": { \"b\": [{\"c\": 2}]}}`\n- `{\"a\": [{ \"b\": [{\"c\": 2}]}]}`\n  Refer this [example](test/scenarios/paths/rich_path.jt) for more details.\n\n#### Json Paths\nWe support some features of [JSON Path](https://goessner.net/articles/JsonPath/index.html#) syntax using path option (`~j`).\nNote: This is an experimental feature and may not support all the features of JSON Paths. \n\nRefer this [example](test/scenarios/paths/json_path.jt) for more details.\n\n#### Simple selectors\n\n```js\nlet x = a.b.c;\nlet y = a.\"some key\".c\n```\n\nRefer this [example](test/scenarios/selectors/template.jt) for more details.\n\n#### Wildcard selectors\n\n```js\na.*.c // selects c from any direct property of a\n```\n\nRefer this [example](test/scenarios/selectors/wild_cards.jt) for more details.\n\n#### Descendent selectors\n\n```js\n// selects c from any child property of a\n// a.b.c, a.b1.b2.c or a.b1.b2.b3.c\nlet x = a..c;\nlet y = a..\"some key\";\n```\n\nRefer this [example](test/scenarios/selectors/template.jt) for more details.\n\n#### Single Index or Property Filters\n\n```js\nlet x = a[0].c;\nlet y = a[-1].c; // selects last element from array\nlet z = a['some key'].c;\n```\n\nRefer this [example](test/scenarios/filters/array_filters.jt) for more details.\n\n#### Multi Indexes or Properties Filters\n\n```js\nlet x = a[(0, 2, 5)].c;\nlet y = a[('some key1', 'some key2')].c;\n```\n\nRefer this [example](test/scenarios/filters/array_filters.jt) for more details.\n\n#### Range filters\n\n```js\nlet x = a[2:5].c;\nlet y = a[:-2].c;\nlet z = a[2:].c;\n```\n\n#### Object Property Filters\n\n```js\nlet x = obj{[\"a\", \"b\"]};  // selects a and b\nlet y = obj{~[\"a\", \"b\"]}; // selects all properties except a and b\n```\n\nRefer this [example](test/scenarios/filters/object_indexes.jt) for more details.\n\n#### Conditional or Object Filters\n\n```js\nlet x = obj{.a \u003e 1};\n```\n\nRefer this [example](test/scenarios/filters/object_filters.jt) for more details.\n\n#### Block expressions\n\n```js\nlet x = obj.({\n  a: .a + 1,\n  b: .b + 2\n});\nlet x = obj.([.a+1, .b+2]);\n```\n\nRefer this [example](test/scenarios/paths/block.jt) for more details.\n\n#### Context Variables\n\n```js\n.orders@order#idx.products.({\n    name: .name,\n    price: .price,\n    orderNum: idx,\n    orderId: order.id\n})\n```\n\nUse context variables: `@order` and `#idx`, we can combine properties of orders and products together. Refer this [example](test/scenarios/context_variables/template.jt) for more details.\n\n#### Path Options\n\nWe can mention defaultPathType while creating engine instance.\n\n```js\n// For using simple path as default path type\n// a.b.c will be treated as simple path\nJsonTemplateEngine.create(`a.b.c`, { defaultPathType: PathType.SIMPLE });\n// For using rich path as default path type\n// a.b.c will be treated as rich path\nJsonTemplateEngine.create(`a.b.c`, { defaultPathType: PathType.RICH });\n```\n\nWe can override the default path option using tags.\n\n```js\n// Use ~s to treat a.b.c as simple path\n~s a.b.c\n// Use ~r to treat a.b.c as rich path\n~r a.b.c\n// Use ~j for using json paths\n~j items[?(@.a\u003e1)]\n```\n\n**Note:** Rich paths are slower compare to the simple paths.\nRefer this [example](test/scenarios/paths/options.jt) for more details.\n\n### Compile time expressions\n\nCompile time expressions are evaluated during compilation phase using compileTimeBindings option.\n\n```js\n// {{$.a.b.c}} gets translated to 1 and\n// final translated code will be \"let a = 1;\"\nJsonTemplateEngine.create(`let a = {{$.a.b.c}};`, {\n  compileTimeBindings: {\n    a: {\n      b: {\n        c: 1,\n      },\n    },\n  },\n});\n```\n\nWe can use compile time expressions to generate a template and then recompile it as expression. Refer these examples [simple compilation](test/scenarios/compile_time_expressions/template.jt) and [complex compilation](test/scenarios/compile_time_expressions/two_level_path_processing.jt) for more details.\n\n### Mappings\nIf you are familiar with [JSON Paths](https://goessner.net/articles/JsonPath/index.html#), you can easily begin working with JSON templates by leveraging your existing knowledge through the mappings feature.\n\n**Example:**\n* Let's say we want to transform the following data.\n* Input:\n```json\n{\n  \"a\": {\n    \"foo\": 1,\n    \"bar\": 2\n  },\n  \"b\": [\n    {\n      \"firstName\": \"foo\",\n      \"lastName\": \"bar\"\n    },\n    {\n      \"firstName\": \"fizz\",\n      \"lastName\": \"buzz\"\n    }\n  ]\n}\n```\n* Output:\n```json\n{\n  \"foo\": 1,\n  \"bar\": 2,\n  \"items\":[\n    {\n      \"name\": \"foo bar\"\n    },\n    {\n      \"name\": \"fizz buzz\" \n    }\n  ] \n}\n```\n* Mappings:\n```json\n[\n  {\n    \"description\": \"Copies properties of a to root level in the output\",\n    \"input\": \"$.a\",\n    \"output\": \"$\"\n  },\n  {\n    \"description\": \"Combines first and last name in the output\",\n    \"input\": \"$.b[*].(@.firstName + ' ' + @.lastName)\",\n    \"output\": \"$.items[*].name\"\n  }\n]\n```\n* Try this example in our [playground](https://transformers-workflow-engine.rudderstack.com/#/mappings?gist=e25a6ac769ee5719e928720f5c439169).\n\nFor more examples, refer [Mappings](test/scenarios/mappings/)\n\n### Comments\n\nSupports both c style single line (`//`) and block comments (`/* .. */`).\nRefer this [example](test/scenarios/comments/template.jt) for more details.\n\nFor more examples, refer [Scenarios](test/scenarios)\n\n## Testing\n\n`npm test`\n\n## Contribute\n\nWe would love to see you contribute to RudderStack. Get more information on how to contribute [**here**](CONTRIBUTING.md).\n\n## License\n\nThe RudderStack `rudder-json-template-engine` is released under the [**MIT License**](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frudderlabs%2Frudder-json-template-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frudderlabs%2Frudder-json-template-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frudderlabs%2Frudder-json-template-engine/lists"}