{"id":15297260,"url":"https://github.com/mehdizonjy/json-transqlify","last_synced_at":"2025-04-13T23:16:03.091Z","repository":{"id":31039781,"uuid":"126701455","full_name":"MehdiZonjy/json-transqlify","owner":"MehdiZonjy","description":"Transforms and loads JSON to a MySQL db","archived":false,"fork":false,"pushed_at":"2023-01-04T01:12:16.000Z","size":324,"stargazers_count":6,"open_issues_count":11,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-13T23:15:53.474Z","etag":null,"topics":["javascript","load","mysql","node-js","nodejs","sql","transform"],"latest_commit_sha":null,"homepage":"https://mzmuse.com/blog/introducing-json-transqlify","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MehdiZonjy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-25T13:42:11.000Z","updated_at":"2024-09-13T22:40:01.000Z","dependencies_parsed_at":"2023-01-14T18:13:10.671Z","dependency_job_id":null,"html_url":"https://github.com/MehdiZonjy/json-transqlify","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MehdiZonjy%2Fjson-transqlify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MehdiZonjy%2Fjson-transqlify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MehdiZonjy%2Fjson-transqlify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MehdiZonjy%2Fjson-transqlify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MehdiZonjy","download_url":"https://codeload.github.com/MehdiZonjy/json-transqlify/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248794569,"owners_count":21162615,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["javascript","load","mysql","node-js","nodejs","sql","transform"],"created_at":"2024-09-30T19:16:06.351Z","updated_at":"2025-04-13T23:16:03.062Z","avatar_url":"https://github.com/MehdiZonjy.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JSON TRANS SQLIFY \nThis library aims to abstract away the following common functionalities involved when transforming and loading JSON into a MySQL database:\n\n- Validation.\n- Transformation.\n- Insert and Update.\n\n## Contents\n- [How It Works](#howItWorks)\n- [API](#api)\n  - [Version](#version)\n  - [Validator](#validator)\n    - [Schema](#schemaValidator)\n    - [Custom Function Validator](#funcValidator) \n  - [Transformers](#transformers)\n    - [Columns Transformer](#columnsTransformer)\n    - [Custom Function Transformer](#funcTransformer)\n  - [Loaders](#loaders)\n    - [Insert Loader](#insertLoader)\n    - [Update Loader](#updateLoader)\n    - [Upsert Loader](#upsertLoader)\n    - [Batch Insert Loader](#batchInsertLoader)\n    - [Batch Upsert Loader](#batchUpsertLoader)\n  - [Preconditions](#preconditions)\n    - [Expression Precondition](#expPrecondition)\n    - [Db Precondition](#dbPrecondition)\n    - [Custom Function Precondition](#funcPrecondition)\n  - [$history](#$history)\n  - [tableName](#$tableName)\n\n## \u003ca name=\"howItworks\"\u003e\u003c/a\u003eHow It Works\n```\nnpm install json-transqlify\n```\n\nYou define how your entire TL (Transform Load) pipeline should look like with a `yaml` definition file.\nEach definition file consists of two main sections:\n - Validator: which uses `json-schema` to validate the entity you are trying to TL.\n - Loaders: which does the actually Insert, Update to the specific tables\n\nAssuming you have a bunch of user objects that you would like to insert to your database and each user contains the following fields:\n```json\n{\n  \"name\": \"FIRST_NAME LAST_NAME\",\n  \"age\": \"NUMBER\",\n  \"address\":{\n    \"country\": \"STRING\",\n    \"city\": \"CITY\"\n  }\n}\n```\n\nThe user Table schema is:\n```sql\nCREATE TABLE `users` (\n  `id` int(11) NOT NULL AUTO_INCREMENT,\n  `fname` varchar(45) NOT NULL,\n  `lname` varchar(45) NOT NULL,\n  `age` int(11) NOT NULL,\n  `country` varchar(45) NOT NULL,\n  `city` varchar(45) NOT NULL,\n  PRIMARY KEY (`id`)\n)\n```\n\nA Simple definition file that TL user objects might look like this:\n```yaml\nversion: 1.0\nvalidator:\n  schema:  # validate user object schema  \n    default: user-schema.json\nloaders: # notice loaders is an array\n  - insert:\n      label: InsertUser # name of this operation (can be anything)\n      tableName: users # table to which the json will be inserted\n      transform: \n        columns: # map each column to appropreiate field on json\n          - column: fname # insert into a column name fname\n            value: $entity.name.split(' ')[0] # $entity refers to the user object we are inserting.\n          - column: lname\n            value: $entity.name.split(' ')[1] # grap last name\n          - column: country\n            value: $entity.address.country\n          - column: city\n            value: $entity.address.city\n          - column: age\n            value: $entity.age\n            \n``` \n\nThe `user-schema.json` uses `json-schema` rules to validate each user object to be inserted:\n``` json\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"name\": {\n      \"type\": \"string\"\n    },\n    \"age\": {\n      \"type\": \"number\"\n    },\n    \"address\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"country\": {\n          \"type\": \"string\"\n        },\n        \"city\": {\n          \"type\": \"string\"\n        }\n      },\n      \"required\": [\n        \"country\",\n        \"city\"\n      ]  \n  },\n  \"required\": [\n    \"name\",\n    \"age\"\n  ]\n}\n```\n\nAll that is left is to construct a JSON transqlifier object:\n```javascript\nconst createFactory = require('json-transqlify').createFactory;\n\nconst db = {\n  host: 'localhost',\n  user: 'root',\n  password: '',\n  database: 'json_transqlify_demos',\n  connectionLimit: 2\n}\n\nconst factory = createFactory(db)\nconst transqlifier = factory.createTransqlifier('./insert-user.yaml');\n\nconst obj = { name: \"Harry Potter\", age: 10, address: { city: 'UK', country: 'Little Whinging' } };\n\ntransqlifier(obj);\n```\n**Please refer to examples folder**\n\n## \u003ca name=\"api\"\u003e\u003c/a\u003eAPI\nThe definition file consists of the following sections:\n\n### \u003ca name=\"version\"\u003e\u003c/a\u003eVersion\nShould be 1.0 for now:\n```yaml\nversion: 1.0\n```\n\n### \u003ca name=\"validator\"\u003e\u003c/a\u003eValidator\nThe Validator filters out entities before they get handed to the Loaders. There are two kind of validators:\n\n#### \u003ca name=\"schemaValidator\"\u003e\u003c/a\u003e1. Schema\nA schema validator can be defined using JSON files to describe how the entity schema should look like. Underneath the hood JSON Transqlifier uses [AJV](https://github.com/epoberezkin/ajv) implementation of [Json Schema](http://json-schema.org/)\n\nThe schema file for the entity should go under the `default` section (refer to the example below). While any `$ref` definitions can be used to load any additional definitions that the default schema might refer to.\nFor example, to write a validator for the following user object:\n```json\n{\n  \"name\": \"User Name\",\n  \"age\": 28,\n  \"address\": {\n    \"country\": \"some country\",\n    \"city\": \"some city\"\n  } \n}\n```\n\nWe might break the validator into two schema definitions into User and Address:\n\n1. `user-definition.json`:\n```json\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"name\": { \"type\": \"string\" },\n    \"age\": { \"type\": \"number\" },\n    \"address\": {\n      \"$ref\": \"Address\"\n    }\n  },\n  \"required\": [\"name\", \"age\", \"address\"]\n}\n```\n\n2. `address-definition.json`:\n```json\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"country\": { \"type\": \"string\" },\n    \"city\": { \"type\": \"string\" }\n  },\n  \"required\": [\"country\", \"city\"]\n}\n```\n\nThen we can reference both schemas like this:\n```yaml\nversion: 1.0\nvalidator:\n  schema:\n    default: user-schema.json #a json file containing the JSON-SCHEMA definition for root entity\n    refs:\n      id: Address\n      file: address-schema.json\n```\n\n#### \u003ca name=\"funcValidator\"\u003e\u003c/a\u003e2. Custom Function Validator\nWhen a `schema` validator is not enough you can have more control by providing a custom function validator.\nThe function should be defined in a separate file and exposed as a default export:\n```javascript\n// is-odd.js\nconst isOdd = num =\u003e num % 2\nmodule.exports = isOdd\n```\n\nThen reference the custom validator in the Transqlifier definition file:\n```yaml\nversion: 1.0\nvalidator:\n  func: is-odd.js\n```\n\n### \u003ca name=\"transformers\"\u003e\u003c/a\u003eTransformers\nTransformers are defined as part lof `loaders`. They map the given `$entity` to table columns.\n\n#### \u003ca name=\"columnsTransformer\"\u003e\u003c/a\u003e1. Columns Transformer\nThe columns transformer allows you to map `$entity` to table columns by defining custom expressions.\n\nFor example, given the `User` object mentioned earlier and `users` table with (fname, lname, age, country, city) columns:\n```yaml\ntransformer:\n  columns:\n    - column: fname\n      value: _.head($entity.name.split(' ')) # $entity refers to the object we are trying to transform. You can define here any expression you like and it will be evaluated at run time. You have access to Lodash by using (_)\n    - column: lname\n      value: _.tail($entity.name.split(' ')).join(' ')\n    - column: age\n      value: $entity.age\n    - column: country\n      value: $entity.address.country\n    - column: city\n      value: $entity.address.city\n```\n\n*$history:* if the transformer was part of multiple loaders pipeline, the `$history` can be used to access values transformed via a previous loader (more on this later)\n\n#### \u003ca name=\"funcTransformer\"\u003e\u003c/a\u003e2. Custom Function Transformer\nA Custom function trasformer can be used by providing a file with a default exported function that returns a Promise:\n```js\n//custom-transformer.js\nconst func = ({$entity, $history, $conn}) =\u003e {\n   // the transformer will be provided the following\n   // $entity: the entity we are currently processing\n   // $history: in case the transformer is part of multiple pipeline loaders, $history will containg previously transformer values\n   // $conn a connection to the db\n  return Promise.resolve({\n    col1: 'val1',\n    col2: 'val2'\n  })\nmodule.exports = func\n}\n```\n\n```yaml\ntransformer:\n  func: customer-transformer.js\n```\n\n### \u003ca name=\"loaders\"\u003e\u003c/a\u003eLoaders\nLoaders handle massaging the JSON (entity) and Inserting / Updating the DB.\n\nThe `loaders` section is an array, so you can insert the JSON into multiple tables by defining multiple loaders.\n\n#### \u003ca name=\"insertLoader\"\u003e\u003c/a\u003e1.Insert\nThe insert loaders inserts entity to a given table. It requires a `transformer` to be defined:\n```yaml\nloaders:\n  - insert:\n      tableName: users # table to insert entity into\n      label: insertUser # a custom name to the loader.\n      trasformer: # refer to transformers doc\n        columns:\n          - column: fname\n            value: $entity.name \n\n```\n\n#### \u003ca name=\"updateLoader\"\u003e\u003c/a\u003e2. Update\nThe Update loader is used to update an existing row in database. It requires a `transformer` and update condition:\n```yaml\nloaders:\n  - update:\n      tableName: users\n      transformer:\n        columns:\n          - column: fname\n            value: $entity.name\n      where:\n        query: id = ?\n        params:\n          - $entity.id\n```\n\n#### \u003ca name=\"upsertLoader\"\u003e\u003c/a\u003e3. Upsert\nThe Upsert loader is used to insert or update (on duplicate key error) existing record. It requires a `transformer`, `tableName`, and `label`:\n```yaml\nloaders:\n  - upsert:\n      tableName: courses\n      primaryKey: id \n      transformer:\n        columns:\n          - column: title # when the title column has a unique index constraint, the existing record will get updated  \n            value: $entity.title\n          - column: difficulty\n            value: $entity.difficulty\n```\n\n`primaryKey` is an optional field. It points to an auto incremented column (if any) in database. In case of update, it will be needed to retrieve the `id` of the affected row.\nSee `examples/upsert-example` for a working demo.\n\n#### \u003ca name=\"batchInsertLoader\"\u003e\u003c/a\u003e3. Batch Insert Loader\nIn case you want to bulk insert data in one go, Batch Insert Loader offers a great performance gain over multiple `Insert Loader`. It requires you to define `transformer`, `tableName`, `label` and `source`:\n```yaml\n  - batchInsert:\n      tableName: users # table to insert entity into\n      source: $entity\n      label: insertUser # a custom name to the loader.\n      trasformer: # refer to transformers doc\n        columns:\n          - column: fname\n            value: $entity.name \n```\n\nSource is an expression that should return an array of items that will be inserted. For example if `$entity` is:\n```javascript\n{\n  items: ['item1', 'item1']\n}\n```\n\nThen `source` should be defined as \n```yaml\nsource: $entity.source\n```\n\nIn case $entity is the array of items you wish to insert, then define `source` as:\n```yaml\nsource: $entity\n```\n\n#### \u003ca name=\"batchUpsertLoader\"\u003e\u003c/a\u003e3. Batch Upsert Loader\nIn cases where you want to insert a bulk of data in one go. Batch Upsert Loader will insert and updated existing record in one transaction. It requires you to define `transformer`, `tableName`, `label` and `source`. \n\n```yaml\n  - batchUpsert:\n      source: $entity\n      tableName: courses\n      primaryKey: id \n      transformer:\n        columns:\n          - column: title # when the title column has a unique index constraint, the existing record will get updated  \n            value: $entity.title\n          - column: difficulty\n            value: $entity.difficulty\n```\n\n### \u003ca name=\"preconditions\"\u003e\u003c/a\u003ePreconditions\nPreconditions validate `$enitity` before executing the loader, and if it returns false, the loader does not get executed.\n\n#### \u003ca name=\"expPrecondition\"\u003e\u003c/a\u003e1. Expression Precondition (exp)\nEvalutes a given expression at runtime that can access `$entity` and `$history` objects. It can also use `_` lodash:\n```yaml\nloaders:\n  - insert:\n    transformer:\n      columns:\n        - column: name\n          value: $entity.name\n    on: # pre conditions are defined here \n      - exp: $entity.age \u003c 30 # only insert uses who are below 30\n```\n\n#### \u003ca name=\"dbPrecondition\"\u003e\u003c/a\u003e2. Database Query (db)\nRuns a query against the database and allows you to assert the returned result.\nForexample, we want to insert a `course` but avoide duplicate titles\n```json\n{\n  \"title\": \"Course Title\"\n}\n```\n\n```yaml\nloaders:\n  - insert:\n    transformer:\n      tableName: courses\n      columns:\n        - column: title\n          value: $entity.title\n    on: # pre conditions are defined here \n      - db:\n          query: SELECT 1 from courses WHERE title = ?\n          params: \n            - $entity.title\n          expect: $rows.length === 0 # $rows refers to the result of query \n```\n\n#### \u003ca name=\"funcPrecondition\"\u003e\u003c/a\u003e3: Custom Precondition function (func)\nExecuted a custom precondition function. The function is expected to return a promise that resolves to `true` or `false\n```js\n// custom-precondition.js\nconst func = ({$entity, $history, $conn}) =\u003e {\n  // the precondition function will be provided the following\n   // $entity: the entity we are currently processing\n   // $history: in case the transformer is part of multiple pipeline loaders, $history will containg previously transformer values\n   // $conn a connection to the db\n  Promise.resolve(true)\n}\nmodule.exports = func\n```\n```yaml\nloaders:\n  - insert:\n    transformer:\n      tableName: courses\n      columns:\n        - column: title\n          value: $entity.title\n    on: # pre conditions are defined here \n      - func: custom-precondition.js\n```\n\n*Note:* preconditions are defined inside an array object. Meaning, you can provide multiple preconditions that should all resolve to `true` for the loader to execute.\n\n### \u003ca name=\"$history\"\u003e\u003c/a\u003e$history\nThe $history object can be accessed inside `transformers` and `preconditions`. It contains the result of previous loaders. For example:\n```yaml\nloaders:\n  - insert:\n      tableName: table1\n      label: InsertTable1\n      transformer:\n        columns:\n          - column: title\n            value: $entity.title\n  - insert:\n      tableName: table2\n      label: InsertTable2\n      transformer:\n        columns:\n          - column: table1_id\n            value: $history.InsertTable1.$insertedId # the tranformed value of each previous loader is added to $history inside \u003cLABEL\u003e of the loader. For Insert loaders the inserted Id for AutoIncreament columns is added to $insertedId.\n          - column: cap_title\n            value: $history.InsertTable.title.toUperCase() # $history has access to the transformed $entity fields \n```\n\n*note:* In case there is a precondition defined in the First Loader, and that precondition happened to evalute to false, the loader result won't be added to `$history` object\n\n### \u003ca name=\"$tableName\"\u003e\u003c/a\u003e$tableName\nThe table name field defined in `loader` can either be a string referring to the table name or it can be an expression evaluated at runtime.\nThe expression has access to the following variables:\n\n- `_`, `R` (ramda).\n- `$entity`.\n- `$source` (in case of batchInsert and batchUpsert loaders).\n\n```yaml\nloaders:\n  - insert:\n      tableName:\n        exp: $entity.tableName\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehdizonjy%2Fjson-transqlify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmehdizonjy%2Fjson-transqlify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehdizonjy%2Fjson-transqlify/lists"}