{"id":13818182,"url":"https://github.com/chartshq/datamodel","last_synced_at":"2025-10-04T01:31:07.883Z","repository":{"id":66269518,"uuid":"146284203","full_name":"chartshq/datamodel","owner":"chartshq","description":"Relational algebra compliant in memory tabular data store.","archived":true,"fork":false,"pushed_at":"2020-07-31T11:19:03.000Z","size":3195,"stargazers_count":61,"open_issues_count":4,"forks_count":7,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-04-26T01:20:17.538Z","etag":null,"topics":["data","datagrid","datamodel","datatable","datatables","javascript","relational-algebra","rust","rust-language","schema","tabular-data","wasm","webassembly"],"latest_commit_sha":null,"homepage":"https://muzejs.org/docs/wa/latest/concepts/datamodel/introducing-datamodel","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chartshq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-08-27T10:45:55.000Z","updated_at":"2023-08-05T06:01:14.000Z","dependencies_parsed_at":"2023-05-03T09:01:01.563Z","dependency_job_id":null,"html_url":"https://github.com/chartshq/datamodel","commit_stats":{"total_commits":550,"total_committers":19,"mean_commits":28.94736842105263,"dds":0.6654545454545455,"last_synced_commit":"c03366dd81bebc83b21cc5bbaaad40b61b68a58c"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chartshq%2Fdatamodel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chartshq%2Fdatamodel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chartshq%2Fdatamodel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chartshq%2Fdatamodel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chartshq","download_url":"https://codeload.github.com/chartshq/datamodel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235208944,"owners_count":18953003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","datagrid","datamodel","datatable","datatables","javascript","relational-algebra","rust","rust-language","schema","tabular-data","wasm","webassembly"],"created_at":"2024-08-04T07:00:35.219Z","updated_at":"2025-10-04T01:31:07.461Z","avatar_url":"https://github.com/chartshq.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"[![Free](https://img.shields.io/badge/cost-Free-brightgreen)](http://muzejs.org/muze-wa/eula)\n[![License](https://img.shields.io/badge/license-Custom-brightgreen)](http://muzejs.org/muze-wa/eula)\n[![NPM version](https://img.shields.io/npm/v/@chartshq/datamodel.svg)](https://www.npmjs.com/package/@chartshq/datamodel)\n[![Contributors](https://img.shields.io/github/contributors/chartshq/datamodel.svg)](https://github.com/chartshq/datamodel/graphs/contributors)\n\n## What is DataModel?\n\nDataModel is an in-browser representation of tabular data. It uses WebAssembly for high performance and works seamlessly with any JavaScript library. It supports [Relational Algebra](https://en.wikipedia.org/wiki/Relational_algebra) operators which enable you to run `select`, `group`, `sort` (and many more) operations on the data.\n\nThe current version performs all the data operations like `filtering`, `aggregation`, etc. on **[WebAssembly](https://webassembly.org/)** which gives a **10x performance** boost compared to the [old JavaScript version](https://github.com/chartshq/datamodel-deprecated).\n\nIt is written in [Rust Language](https://www.rust-lang.org/) to handle computation intensive data operations, which is then compiled to **[WebAssembly](https://webassembly.org/)**, thereby providing a native-like performance for data operations.\n\nDataModel can be used if you need an in-browser tabular data store for data analysis, visualization or just general use of data.\n\n## Features\n\n * 🎉  Supports [**Relational Algebra**](https://en.wikipedia.org/wiki/Relational_algebra) operators e.g. `selection`, `projection`, `group`, `calculateVariable`, `sort` etc out-of-the-box.\n\n* 💎  Every operation creates **Immutable** DataModel instance and builds a Directed Acyclic Graph (DAG) which establishes auto interactivity.\n\n* 🚀  Uses **[WebAssembly](https://webassembly.org/)** for handling huge datasets and for **better performance**.\n\n* ⛺  Also works in **Nodejs** environment out-of-the-box.\n\n## Installation\n\n### CDN\n\nInsert the DataModel build into the `\u003chead\u003e`:\n\n```html\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/@chartshq/datamodel@3.0.0/dist/browser/datamodel.js\" type=\"text/javascript\"\u003e\u003c/script\u003e\n```\n\n### NPM\n\nInstall DataModel from NPM:\n\n```bash\n$ npm install --save @chartshq/datamodel\n```\n\nAs we're using [Worker](https://developer.mozilla.org/en-US/docs/Web/API/Worker) internally, so the [worker-loader](https://www.npmjs.com/package/worker-loader) needs to be installed as follows:\n\n```bash\n$ npm install worker-loader --save-dev\n```\n\nAnd then within your webpack configuration object, you'll need to add the `worker-loader` to the list of module rules, like so:\n\n```js\nmodule.exports = {\n  module: {\n    rules: [\n      // Add the following object to your module `rules` list.\n      {\n        test: /\\.worker/,\n        include: /datamodel/,\n        loader: 'worker-loader',\n        options: {\n          inline: false, // If you want to make it inline, set to true.\n          fallback: true\n        },\n      },\n    ],\n  }\n};\n```\n\nYou also can checkout our [datamodel-app-template](https://github.com/chartshq/datamodel-app-template) to try out the `DataModel` quickly through a boilerplate app.\n\n## Getting Started\n\nOnce the installation is done, please follow the steps below:\n\n1. Prepare the data and the corresponding schema:\n\n```js\n// Prepare the schema for data.\nconst schema = [\n  {\n    name: 'Name',\n    type: 'dimension'\n  },\n  {\n    name: 'Maker',\n    type: 'dimension'\n  },\n  {\n    name: 'Horsepower',\n    type: 'measure',\n    defAggFn: 'avg'\n  },\n  {\n    name: 'Origin',\n    type: 'dimension'\n  }\n]\n\n// Prepare the data.\nconst data = [\n   {\n    \"Name\": \"chevrolet chevelle malibu\",\n    \"Maker\": \"chevrolet\",\n    \"Horsepower\": 130,\n    \"Origin\": \"USA\"\n  },\n  {\n    \"Name\": \"buick skylark 320\",\n    \"Maker\": \"buick\",\n    \"Horsepower\": 165,\n    \"Origin\": \"USA\"\n  },\n  {\n    \"Name\": \"datsun pl510\",\n    \"Maker\": \"datsun\",\n    \"Horsepower\": 88,\n    \"Origin\": \"Japan\"\n  }\n]\n```\n\n2. Import DataModel as follows:\n\nIf you are using the npm package, import the package as below:\n```js\nimport Engine from '@chartshq/datamodel';\n```\n\nIf you are using it in NodeJS, then require it as below:\n```js\nconst Engine = require('@chartshq/datamodel').default;\n```\n\nIf you are using CDN, then use it as follows:\n```js\nconst Engine = window.DataModel;\n```\n\n3. Load the DataModel engine and pass the data and schema to `DataModel` constructor and create a new `DataModel` instance:\n\n```js\n// As the DataModel are asynchronous, so we need to\n// use async-await syntax.\nasync function myAsyncFn() {\n  // Load the DataModel module.\n  const DataModel = await Engine.onReady();\n\n  // Converts the raw data into a format\n  // which DataModel can consume.\n  const formattedData = await DataModel.loadData(data, schema);\n\n  // Create a new DataModel instance with\n  // the formatted data.\n  const dm = new DataModel(formattedData);\n\n  console.log(dm.getData().data);\n  // Output:\n  //  [\n  //     [\"chevrolet chevelle malibu\", \"chevrolet\", 130, \"USA\"],\n  //     [\"buick skylark 320\", \"buick\", 165, \"USA\"],\n  //     [\"datsun pl510\", \"datsun\", 88, \"Japan\"]\n  //  ]\n\n  // Perform the selection operation.\n  const selectDm = dm.select({ field: 'Origin', value: 'USA', operator: DataModel.ComparisonOperators.EQUAL });\n  console.log(selectDm.getData().data);\n  // Output:\n  //  [\n  //     [\"chevrolet chevelle malibu\", \"chevrolet\", 130, \"USA],\n  //     [\"buick skylark 320\", \"buick\", 165, \"USA]\n  //  ]\n\n  // Perform the projection operation.\n  const projectDm = dm.project([\"Origin\", \"Maker\"]);\n  console.log(projectDm.getData().data);\n  // Output:\n  //  [\n  //     [\"USA\", \"chevrolet\"],\n  //     [\"USA\", \"buick\"],\n  //     [\"Japan\", \"datsun\"]\n  //  ]\n\n  console.log(projectDm.getData().schema);\n  // Output:\n  //  [\n  //     {\"name\": \"Origin\",\"type\": \"dimension\"},\n  //     {\"name\": \"Maker\",\"type\": \"dimension\"}\n  //  ]\n}\n\nmyAsyncFn()\n  .catch(console.error.bind(console));\n\n```\n\n4. Now dispose the DataModel instance if it's not needed:\n\n```js\n// This also disposes all the datamodels which are created from it.\ndm.dispose();\n```\n\n## Documentation\n\nFind detailed documentation and API reference from [here](https://muzejs.org/docs/wa/latest/concepts/datamodel/introducing-datamodel).\n\n## What has changed?\n\nDataModel 3.0.0 now has the core written in [Rust language](https://www.rust-lang.org/)  and has been ported to **[WebAssembly](https://webassembly.org/)** bringing in a huge performance difference w.r.t to [previous version](https://github.com/chartshq/datamodel-deprecated), in terms of both data size and computing speed. While the JavaScript version is deprecated and no active development will take place there but critical bugs if raised would be taken and released in GitHub only.\n\nYou can visit the JavaScript (deprecated) version here [https://github.com/chartshq/datamodel-deprecated](https://github.com/chartshq/datamodel-deprecated)\n\n## Migrating from previous versions of DataModel\n\nNow the DataModel became asynchronous as opposed to being synchronous in the previous JavaScript version.\n\n```js\nimport Engine from '@chartshq/datamodel';\n\n(async () =\u003e {\n  // Load the DataModel module.\n  const DataModel = await Engine.onReady();\n\n  // Converts the raw data into a format\n  // which DataModel can consume.\n  const formattedData = await DataModel.loadData(data, schema);\n\n  // Create a new DataModel instance with\n  // the formatted data.\n  const dm = new DataModel(formattedData);\n})();\n```\n\n### **Changed APIs**\n\n- **select**\n\n    DataModel deprecated version:\n\n    ```js\n    dm.select((fields) =\u003e {\n      return fields.Origin.value === 'USA';\n    });\n    ```\n\n    Latest version:\n\n    ```js\n    dm.select({\n      field: 'Origin',\n      operator: DataModel.ComparisonOperators.EQUAL,\n      value: 'USA'\n    });\n    ```\n\n- **groupBy**\n\n    DataModel deprecated version:\n\n    ```js\n    dm.groupBy(['Origin'], {\n      Acceleration: 'avg'\n    });\n    ```\n\n    Latest version:\n\n    ```js\n    dm.groupBy(['Origin'], [{\n      aggn: DataModel.AggregationFunctions.AVG,\n      field: 'Acceleration'\n    }]);\n    ```\n\nSupported data operations:\n\n- select\n- project\n- calculateVariable\n- sort\n- groupBy\n\nUpcoming data operations:\n\n- join\n- bin\n- compose\n- union\n- difference\n- ... many more ...\n\nFor more details on APIs visit our [docs](https://muzejs.org/docs/wa/latest/concepts/datamodel/introducing-datamodel).\n\n## License\n\n[Custom License](https://muzejs.org/muze-wa/eula) (Free to use)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchartshq%2Fdatamodel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchartshq%2Fdatamodel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchartshq%2Fdatamodel/lists"}