{"id":15002515,"url":"https://github.com/dalenewman/transformalize","last_synced_at":"2025-05-16T08:06:27.891Z","repository":{"id":8724741,"uuid":"10396877","full_name":"dalenewman/Transformalize","owner":"dalenewman","description":"Configurable Extract, Transform, and Load","archived":false,"fork":false,"pushed_at":"2025-05-15T23:21:16.000Z","size":178147,"stargazers_count":161,"open_issues_count":12,"forks_count":31,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-05-16T00:25:25.807Z","etag":null,"topics":["data-warehouse","denormalize","elasticsearch","etl","etl-framework","excel","files","mysql","postgresql","solr","sql-server","sqlce","sqlite","ssas"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dalenewman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2013-05-31T03:07:43.000Z","updated_at":"2025-05-15T23:21:20.000Z","dependencies_parsed_at":"2025-05-16T00:33:01.136Z","dependency_job_id":null,"html_url":"https://github.com/dalenewman/Transformalize","commit_stats":{"total_commits":1023,"total_committers":3,"mean_commits":341.0,"dds":"0.030303030303030276","last_synced_commit":"2467493c86553a815c7a18ad0aa44b077ceb5f6c"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dalenewman%2FTransformalize","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dalenewman%2FTransformalize/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dalenewman%2FTransformalize/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dalenewman%2FTransformalize/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dalenewman","download_url":"https://codeload.github.com/dalenewman/Transformalize/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493378,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-warehouse","denormalize","elasticsearch","etl","etl-framework","excel","files","mysql","postgresql","solr","sql-server","sqlce","sqlite","ssas"],"created_at":"2024-09-24T18:50:55.711Z","updated_at":"2025-05-16T08:06:22.881Z","avatar_url":"https://github.com/dalenewman.png","language":"C#","readme":"# Transformalize\n\nTransformalize is a configuration based ETL tool that automates \nincremental denormalization of relational data.  It may be \nused for other types of [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) \nas well.  \n\nThe supported inputs and outputs are below. \n\n\u003cdiv class=\"table-responsive\" style=\"font-size:smaller;\"\u003e\n\u003ctable class=\"table table-condensed\"\u003e\n    \u003cthead\u003e\n      \u003c!--\u003ctr\u003e\n         \u003cth colspan=\"3\" style=\"text-align:center\"\u003eCross Platform\u003c/th\u003e\n      \u003c/tr\u003e--\u003e\n        \u003ctr\u003e\n            \u003cth\u003eRelational\u003c/th\u003e\n            \u003cth colspan=\"2\" style=\"text-align:center\"\u003eNon-Relational\u003c/th\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n        \u003ctr\u003e\n            \u003ctd style=\"vertical-align:top\"\u003e\n                \u003ctable class=\"table table-condensed\"\u003e\n                    \u003cthead\u003e\n                        \u003ctr\u003e\n                            \u003cth\u003eProvider\u003c/th\u003e\n                            \u003cth\u003eSupport\u003c/th\u003e\n                        \u003c/tr\u003e\n                    \u003c/thead\u003e\n                    \u003ctbody\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.SqlServer\"\u003eSQL Server\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd \u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.MySql\"\u003eMySql\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd \u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.PostgreSql\"\u003ePostgreSql\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd \u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Sqlite\"\u003eSQLite\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd \u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                    \u003c/tbody\u003e\n                \u003c/table\u003e\n            \u003c/td\u003e\n            \u003ctd style=\"vertical-align:top\"\u003e\n                \u003ctable class=\"table table-condensed\"\u003e\n                    \u003cthead\u003e\n                        \u003ctr\u003e\n                            \u003cth\u003eProvider\u003c/th\u003e\n                            \u003cth\u003eSupport\u003c/th\u003e\n                        \u003c/tr\u003e\n                    \u003c/thead\u003e\n                    \u003ctbody\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Elasticsearch\"\u003eElasticsearch\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Excel\"\u003eExcel\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.CsvHelper\"\u003eFiles\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Mail\"\u003eMail\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eOut\u003c/td\u003e\n                        \u003c/tr\u003e                        \n                    \u003c/tbody\u003e\n                \u003c/table\u003e\n            \u003c/td\u003e\n            \u003ctd style=\"vertical-align:top\"\u003e\n                \u003ctable class=\"table table-condensed\"\u003e\n                    \u003cthead\u003e\n                        \u003ctr\u003e\n                            \u003cth\u003eProvider\u003c/th\u003e\n                            \u003cth\u003eSupport\u003c/th\u003e\n                        \u003c/tr\u003e\n                    \u003c/thead\u003e\n                    \u003ctbody\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003eConsole\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Razor\"\u003eRazor\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eOut\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd title=\"Forms in Orchard CMS Module\"\u003e\u003ca href=\"https://github.com/dalenewman/OrchardCore.Transformalize\"\u003eForms\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Bogus\"\u003eBogus\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn\u003c/td\u003e\n                        \u003c/tr\u003e\n                    \u003c/tbody\u003e\n                \u003c/table\u003e\n            \u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\u003c!--\u003cdiv class=\"table-responsive\" style=\"font-size:smaller;\"\u003e\n\u003ctable class=\"table table-condensed\"\u003e\n    \u003cthead\u003e\n      \u003ctr\u003e\n         \u003cth colspan=\"3\" style=\"text-align:center\"\u003eWindows Only\u003cspan style=\"font-size:smaller\"\u003e\u003c/span\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n         \u003cth\u003eRelational\u003c/th\u003e\n         \u003cth colspan=\"2\" style=\"text-align:center\"\u003eNon-Relational\u003c/th\u003e\n      \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n        \u003ctr\u003e\n            \u003ctd style=\"vertical-align:top\"\u003e\n                \u003ctable class=\"table table-condensed\"\u003e\n                    \u003cthead\u003e\n                        \u003ctr\u003e\n                            \u003cth\u003eProvider\u003c/th\u003e\n                            \u003cth\u003eSupport\u003c/th\u003e\n                        \u003c/tr\u003e\n                    \u003c/thead\u003e\n                    \u003ctbody\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.SqlCe\"\u003eSqlCe\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Access\"\u003eAccess\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e \u0026nbsp;\u003c/td\u003e\n                            \u003ctd\u003e \u0026nbsp;\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e \u0026nbsp;\u003c/td\u003e\n                            \u003ctd\u003e \u0026nbsp;\u003c/td\u003e\n                        \u003c/tr\u003e\n                    \u003c/tbody\u003e\n                \u003c/table\u003e\n            \u003c/td\u003e\n            \u003ctd style=\"vertical-align:top\"\u003e\n                \u003ctable class=\"table table-condensed\"\u003e\n                    \u003cthead\u003e\n                        \u003ctr\u003e\n                            \u003cth\u003eProvider\u003c/th\u003e\n                            \u003cth\u003eSupport\u003c/th\u003e\n                        \u003c/tr\u003e\n                    \u003c/thead\u003e\n                    \u003ctbody\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Lucene\"\u003eLucene\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Solr\"\u003eSOLR\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Excel\"\u003eExcel\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eOut\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.FileHelpers\"\u003eFiles\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn/Out\u003c/td\u003e\n                        \u003c/tr\u003e\n                    \u003c/tbody\u003e\n                \u003c/table\u003e\n            \u003c/td\u003e\n            \u003ctd style=\"vertical-align:top\"\u003e\n                \u003ctable class=\"table table-condensed\"\u003e\n                    \u003cthead\u003e\n                        \u003ctr\u003e\n                            \u003cth\u003eProvider\u003c/th\u003e\n                            \u003cth\u003eSupport\u003c/th\u003e\n                        \u003c/tr\u003e\n                    \u003c/thead\u003e\n                    \u003ctbody\u003e\n                        \u003ctr\u003e\n                            \u003ctd title=\"SQL Server Analysis Services\"\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.Ssas\"\u003eSSAS\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eOut\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.RethinkDb\"\u003eRethinkDB\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eOut\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e\u003ca href=\"https://github.com/dalenewman/Transformalize.Provider.ActiveDirectory\"\u003eActive Directory\u003c/a\u003e\u003c/td\u003e\n                            \u003ctd\u003eIn\u003c/td\u003e\n                        \u003c/tr\u003e\n                        \u003ctr\u003e\n                            \u003ctd\u003e \u0026nbsp;\u003c/td\u003e\n                            \u003ctd\u003e \u0026nbsp;\u003c/td\u003e\n                        \u003c/tr\u003e                        \n                    \u003c/tbody\u003e\n                \u003c/table\u003e\n            \u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e--\u003e\n\n---\n\n### Getting Started\n\nThis *readme* demonstrates how to denormalize a relational database and load it into Elasticsearch.\n \nTo follow along, see the [prerequisites](https://github.com/dalenewman/Transformalize/wiki/README-Prerequisites).\n\nWhen you start denormalizing a database, it's good to have a diagram. \n\n\u003cimg src=\"./Files/northwind-diagram.png\" class=\"img-responsive img-thumbnail\" alt=\"Northwind Schema\" /\u003e\n\nThis shows eight [normalized](https://en.wikipedia.org/wiki/Database_normalization) \ntables related to *Order Details*. \n\n\u003e This section introduces `\u003cconnections/\u003e`, `\u003centities/\u003e`, and the `page` and `size` attributes.\n\nTransformalize arrangements are written in [XML](https://en.wikipedia.org/wiki/XML) or [JSON](https://en.wikipedia.org/wiki/JSON). \nThey are validated before execution.\n\nTo get started, open VS Code and paste this in:\n\n```xml\n\u003ccfg name=\"NorthWind\" read-only=\"true\"\u003e\n  \u003cconnections\u003e\n    \u003cadd name=\"input\" provider=\"sqlserver\" user=\"sa\" password=\"Secret1!\" database=\"Northwind\" /\u003e\n  \u003c/connections\u003e\n  \u003centities\u003e\n    \u003cadd name=\"Order Details\" page=\"1\" size=\"5\" /\u003e\n  \u003c/entities\u003e\n\u003c/cfg\u003e\n```\n\nThis defines an *input* as the `Order Details` table inside the Northwind database. \nSave it as *NorthWind.xml* and press CTRL-P to find and execute the `tfl:run` command. \n\n![Step01](./Files/Demo/step01-cp.gif \"Step 1\")\n\nTransformalize returned the first page of rows from the `Order Details` table. \nTo transform and save this data, the `\u003cfields/\u003e` must be defined.\n\n\u003e Introducing `\u003cfields/\u003e`.\n\nTo see the schema, press CTRL-P to find and execute the `tfl:schema` command:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003e tfl -a NorthWind.xml \u003cstrong\u003e-m schema\u003c/strong\u003e\n...\n\u0026lt;fields\u0026gt;\n  \u0026lt;add name=\"OrderID\" type=\"int\" primary-key=\"true\" /\u0026gt;\n  \u0026lt;add name=\"ProductID\" type=\"int\" primary-key=\"true\" /\u0026gt;\n  \u0026lt;add name=\"UnitPrice\" type=\"decimal\" precision=\"19\" scale=\"4\" /\u0026gt;\n  \u0026lt;add name=\"Quantity\" type=\"short\" /\u0026gt;\n  \u0026lt;add name=\"Discount\" type=\"single\" /\u0026gt;\n\u0026lt;/fields\u003e\n...\n\u003c/pre\u003e\n\nCopy the `\u003cfields/\u003e` from the output into your arrangement like this:\n\n```xml\n\u003ccfg name=\"NorthWind\"\u003e\n  \u003cconnections\u003e\n    \u003cadd name=\"input\" provider=\"sqlserver\" user=\"sa\" password=\"Secret1!\" database=\"Northwind\" /\u003e\n  \u003c/connections\u003e\n  \u003centities\u003e\n    \u003cadd name=\"Order Details\" page=\"1\" size=\"5\"\u003e\n      \u003c!-- copy/paste the fields here --\u003e\n      \u003cfields\u003e\n        \u003cadd name=\"OrderID\" type=\"int\" primary-key=\"true\" /\u003e\n        \u003cadd name=\"ProductID\" type=\"int\" primary-key=\"true\" /\u003e\n        \u003cadd name=\"UnitPrice\" type=\"decimal\" precision=\"19\" scale=\"4\" /\u003e\n        \u003cadd name=\"Quantity\" type=\"short\" /\u003e\n        \u003cadd name=\"Discount\" type=\"single\" /\u003e\n      \u003c/fields\u003e\n    \u003c/add\u003e\n  \u003c/entities\u003e\n\u003c/cfg\u003e\n```\n\n\u003e Introducing **`\u003ccalculated-fields/\u003e`**, the **`t`** attribute, and the **`js`** and **`round`** transformations\n\nAdd **`\u003ccalculated-fields/\u003e`** right after **`\u003cfields/\u003e`** like this:\n\n```xml\n\u003ccalculated-fields\u003e\n  \u003cadd name=\"Revenue\" \n       type=\"decimal\" \n       t=\"js(Quantity * ((1-Discount) * UnitPrice)).round(2)\" /\u003e\n\u003c/calculated-fields\u003e\n```\nExecute `tfl:run` and it should produce this output:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003e\u003e tfl -a NorthWind.xml\u003c/strong\u003e\nOrderID,ProductID,UnitPrice,Quantity,Discount,\u003cstrong\u003eRevenue\u003c/strong\u003e\n10248,11,14.0000,12,0,\u003cstrong\u003e168\u003c/strong\u003e\n10248,42,9.8000,10,0,\u003cstrong\u003e98\u003c/strong\u003e\n10248,72,34.8000,5,0,\u003cstrong\u003e174\u003c/strong\u003e\n10249,14,18.6000,9,0,\u003cstrong\u003e167.4\u003c/strong\u003e\n10249,51,42.4000,40,0,\u003cstrong\u003e1696\u003c/strong\u003e\n...\n\u003c/pre\u003e\n\n*Revenue* is created by the **js** (JavaScript) and **round** [transformations](./Containers/Autofac/Transformalize.Container.Autofac.Shared/TransformBuilder.cs).  You may chain transformations as long as the output of one is compatible with the input of another.\n\n![Step02](./Files/Demo/step02-cp.gif \"Step 2\")\n\n### Output\n\n\u003e Introducing **`init`** mode\n\nLet's save *all* the `Order Details` into an output. To do this:\n\n1. Remove the `read-only` attribute.\n1. Remove the `page` and `size` attributes.\n1. Define the output as a PostgreSql database named TflNorthwind in `\u003cconnections/\u003e`.\n\n```xml\n\u003cconnections\u003e\n    \u003cadd name=\"input\" provider=\"sqlserver\" user=\"sa\" password=\"Secret1!\" database=\"Northwind\" /\u003e\n    \u003c!-- define output here --\u003e\n    \u003cadd name=\"output\" provider=\"postgres\" user=\"postgres\" password=\"Secret1!\" database=\"TflNorthwind\" /\u003e\n\u003c/connections\u003e\n```\n\nPress CTRL-P to find and run the `tfl:init` command.\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003e tfl -a NorthWind.xml \u003cstrong\u003e-m init\u003c/strong\u003e\n\u003cstrong style=\"color:#FF7F50;\"\u003ewarn  | NorthWind | Order Details | Initializing\u003c/strong\u003e\ninfo  | NorthWind | Order Details | 2155 from input\ninfo  | NorthWind | Order Details | 2155 inserts into output\ninfo  | NorthWind | Order Details | Ending 00:00:03.89\n\u003c/pre\u003e\n\n### Initialization\n\nInitialization is required initially and any time the structure of the output is changed.\n\nIt does three things:\n\n1. destroys pre-existing output structures\n2. creates output structures\n3. bulk inserts data.\n\n![Step03](./Files/Demo/step03-cp.gif \"Step 3\")\n\nNote that writing *Order Details* into PostgreSQL frees up the standard output for logging.\n\n#### Mapping\n\nTransformalize doesn't *map* input to pre-existing output. Instead, it \ncreates a control table and a consistent output structure \nfor handling incremental updates.  \n\nYou decide:\n\n* what new fields to calculate\n* the order of fields\n* the name of fields (using `alias`)\n* the transformation and/or validation of fields\n* and the output of field (using `output=\"true|false\"`)\n\n### Incrementals (by Default)\n\n\u003e Introducing the **`version`** attribute for an **`entity`**\n\nAn *initialization* is a full rebuild and may be time-consuming. So, by default, \nTransformalize performs incrementals. To determine if an update or insert \nis necessary, it compares input with output.\n\nWhile keys and hashes are used, comparison is unnecessary \nwhen an input's provider is queryable and has a row version. \nA row version increments when the row changes.  Many tables \nhave these by design, but if not, you can add them like this:\n\n```sql\n/* SQL Server and SQL CE */\nALTER TABLE [Order Details] ADD [RowVersion] ROWVERSION;\n\n/* MySQL */\nALTER TABLE `Order Details` ADD COLUMN RowVersion TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP;\n\n/* PostgreSql, use the system field xmin */\n```\n\nTo demonstrate incrementals, add a row version to each of the eight tables \nin the SQL Server *Northwind* database.  I have provided a [script](./Files/Demo/add-row-versions-sql-server.sql) for \nconvenience.\n\nOnce added, we have add the `RowVersion` to our arrangement like this:\n\n```xml\n\u003centities\u003e\n                            \u003c!-- Mark it here --\u003e\n  \u003cadd name=\"Order Details\" version=\"RowVersion\" \u003e\n    \u003cfields\u003e\n      \u003cadd name=\"OrderID\" type=\"int\" primary-key=\"true\" /\u003e\n      \u003cadd name=\"ProductID\" type=\"int\" primary-key=\"true\" /\u003e\n      \u003cadd name=\"Discount\" type=\"single\" /\u003e\n      \u003cadd name=\"Quantity\" type=\"short\" /\u003e\n      \u003cadd name=\"UnitPrice\" type=\"decimal\" precision=\"19\" scale=\"4\"/\u003e\n\n      \u003c!-- It's a field, so define it here --\u003e\n      \u003cadd name=\"RowVersion\" type=\"byte[]\" length=\"8\" /\u003e\n    \u003c/fields\u003e\n  \u003c/add\u003e\n\u003c/entities\u003e\n```\n\nAdding a field changes output structure, so re-initialize like so:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003etfl -a NorthWind.xml -m init\u003c/strong\u003e\nwarn  | NorthWind | Order Details | Initializing\ninfo  | NorthWind | Order Details | 2155 from input\ninfo  | NorthWind | Order Details | 2155 inserts into output\ninfo  | NorthWind |               | Time elapsed: 00:00:03.09\n\n\u003cstrong\u003etfl -a NorthWind.xml\u003c/strong\u003e\n\u003cstrong\u003einfo  | NorthWind | Order Details | Change Detected: No.\u003c/strong\u003e\ninfo  | NorthWind |               | Time elapsed: 00:00:00.71\n\u003c/pre\u003e\n\nWith a `version` in place, the second run doesn't read and compare un-changed data.\n\n![Step04](./Files/Demo/step04-cp.gif \"Step 4\")\n\n### Denormalization\n\nRelational data is [normalized](https://en.wikipedia.org/wiki/Database_normalization) and \nstored in many tables. It's optimized for efficient storage and integrity. It may be \nqueried, but not without an overhead of joining busy tables. This makes retrieval slower.\n\n[De-normalization](https://en.wikipedia.org/wiki/Denormalization) is the process of \njoining related data back together. The data is pre-joined (and duplicated) to \navoid joining tables at run-time. Retrieval of de-normalized data is faster.\n\nThe output of *Order Details* (above) is numeric. Some numbers \nare [foreign keys](https://en.wikipedia.org/wiki/Foreign_key) (e.g. `ProductID`, `OrderID`). \nThese refer to more descriptive information in related entities. Others are \n[measures](https://en.wikipedia.org/wiki/Measure_(data_warehouse)) (i.e. `Quantity`, `UnitPrice`).\n\nTo denormalize *Order Details*, we need to use the foreign keys `OrderID` and `ProductID` to \nretrieve the related information from *Orders* and *Products* (see diagram).  This means we have \nto add the *Orders* and *Products* entities to our arrangement.\n\n### Adding an Entity\n\nIt's time to add another entity to our arrangement.  Here is what the `Orders` entity should look like:\n\n```xml\n\u003cadd name=\"Orders\" version=\"RowVersion\"\u003e\n  \u003cfields\u003e\n    \u003cadd name=\"OrderID\" type=\"int\" primary-key=\"true\" /\u003e\n    \u003cadd name=\"CustomerID\" length=\"5\" /\u003e\n    \u003cadd name=\"EmployeeID\" type=\"int\" /\u003e\n    \u003cadd name=\"OrderDate\" type=\"datetime\" /\u003e\n    \u003cadd name=\"RequiredDate\" type=\"datetime\" /\u003e\n    \u003cadd name=\"ShippedDate\" type=\"datetime\" /\u003e\n    \u003cadd name=\"ShipVia\" type=\"int\" /\u003e\n    \u003cadd name=\"Freight\" type=\"decimal\" precision=\"19\" scale=\"4\" /\u003e\n    \u003cadd name=\"ShipName\" length=\"40\" /\u003e\n    \u003cadd name=\"ShipAddress\" length=\"60\" /\u003e\n    \u003cadd name=\"ShipCity\" length=\"15\" /\u003e\n    \u003cadd name=\"ShipRegion\" length=\"15\" /\u003e\n    \u003cadd name=\"ShipPostalCode\" length=\"10\" /\u003e\n    \u003cadd name=\"ShipCountry\" length=\"15\" /\u003e\n    \u003c!-- note: RowVersion must be aliased to co-exist with Order Details RowVersion --\u003e\n    \u003cadd name=\"RowVersion\" alias=\"OrdersRowVersion\" type=\"byte[]\" length=\"8\" /\u003e\n  \u003c/fields\u003e\n\u003c/add\u003e\n```\nBecause we're denormalizing entities, the field names in the output must ultimately be unique.  Since *RowVersion* is already used in *Order Details*, I need to `alias` it in *Orders*.\n\nMoreover, if I add another entity to my arrangement, I must relate it to the first entity.\n\n#### Relationships\n\n\u003e Introducing the **`\u003crelationships/\u003e`** section\n\nAll entities must be related to the first entity in the `\u003crelationships/\u003e` section which \nfollows `\u003centities/\u003e`.  To relate *Orders* to *Order Details*, add this to your arrangement:\n\n```xml\n\u003crelationships\u003e\n    \u003cadd left-entity=\"Order Details\" left-field=\"OrderID\" right-entity=\"Orders\" right-field=\"OrderID\" /\u003e\n\u003c/relationships\u003e\n```\n\nThis tells Transformalize to use `OrderID` to relate the two entities. Now re-initialize \nand run Transformalize:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003etfl -a NorthWind.xml -m init\u003c/strong\u003e\nwarn  | NorthWind | Order Details | Initializing\nwarn  | NorthWind | Orders        | Initializing\ninfo  | NorthWind | Order Details | 2155 from input\ninfo  | NorthWind | Order Details | 2155 inserts into output\n\u003cstrong\u003einfo  | NorthWind | Orders        | 830 from input\ninfo  | NorthWind | Orders        | 830 inserts into output\u003c/strong\u003e\ninfo  | NorthWind |               | Time elapsed: 00:00:01.02\n\n\u003cstrong\u003etfl -a NorthWind.xml\u003c/strong\u003e\ninfo  | NorthWind | Order Details | Change Detected: No.\n\u003cstrong\u003einfo  | NorthWind | Orders        | Change Detected: No.\u003c/strong\u003e\ninfo  | NorthWind |               | Time elapsed: 00:00:00.25\n\u003c/pre\u003e\n\n![Step05](./Files/Demo/step05-cp.gif \"Step 5\")\n\nLogging indicates records were processed from *Order Details* and *Orders*. In addition, \na view called `NorthWindStar` is created in the output.  *NorthWindStar* joins Transformalize's \n[star-schema](https://en.wikipedia.org/wiki/Star_schema) output so that it appears to be a \nsingle entity.\n\nQuery *NorthWindStar* to make sure Transformalize is working:\n\n```sql\nSELECT\n    ProductID,\n    Discount,\n    Quantity,\n    UnitPrice,\n    CustomerID,\n    EmployeeID,\n    Freight,\n    OrderDate,\n    RequiredDate,\n    ShipAddress,\n    ShipCity,\n    ShippedDate,\n    ShipPostalCode,\n    ShipRegion,\n    ShipVia\nFROM NorthWindStar\nLIMIT 10;\n```\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003eProductId   Discount    Quantity    UnitPrice   CustomerID  EmployeeID  Freight OrderDate   RequiredDate    ShipAddress ...\u003c/strong\u003e\n\u003cstrong\u003e---------   --------    --------    ---------   ----------  ----------  ------- ---------   ------------    -----------\u003c/strong\u003e\n11\t    0.0\t        12\t    14\t        VINET       5           32.38   1996-07-04  1996-08-01      59 rue de l'Abbaye\n42\t    0.0\t        10\t    9.8\t        VINET       5           32.38   1996-07-04  1996-08-01      59 rue de l'Abbaye\n72\t    0.0\t        5\t    34.8        VINET       5           32.38   1996-07-04  1996-08-01      59 rue de l'Abbaye\n14\t    0.0\t        9\t    18.6        TOMSP       6           11.61\t1996-07-05  1996-08-16      Luisenstr. 48\n51\t    0.0\t        40\t    42.4        TOMSP       6           11.61\t1996-07-05  1996-08-16      Luisenstr. 48\n41\t    0.0\t        10\t    7.7         HANAR       4           65.83\t1996-07-08  1996-08-05      Rua do Paço, 67\n51\t    0.15        35\t    42.4        HANAR       4           65.83\t1996-07-08  1996-08-05      Rua do Paço, 67\n65\t    0.15        15\t    16.8        HANAR       4           65.83\t1996-07-08  1996-08-05      Rua do Paço, 67\n22\t    0.05        6\t    16.8        VICTE       3           41.34\t1996-07-08  1996-08-05      2, rue du Commerce\n57\t    0.05        15\t    15.6        VICTE       3           41.34\t1996-07-08  1996-08-05      2, rue du Commerce\n\u003c/pre\u003e\n\n### Star Schema \u0026 Single \"Flat\" Entity\n\n\u003e Introducing the **`flatten`** attribute\n\nTransformalize de-normalizes in two phases.  First, it moves data \nfrom a relational model into a [star-schema](https://en.wikipedia.org/wiki/Star_schema).\nSecondly, it moves data into a completely de-normalized (flat) output. \n\n![Relational to Star](https://raw.githubusercontent.com/dalenewman/Transformalize/master/Files/transformalize-diagram.jpg)\n\nTo create a star-schema, it moves the foreign keys to the center.  Data retrieval is \nfaster because everything is directly related.\n\nTo create a flat output, it moves *everything* to the center.  Data retrieval is \neven faster because there aren't any relations.\n\nTo completely de-normalize, set `flatten` to `true` \nin the main `\u003ccfg/\u003e` like this:\n\n```xml\n\u003ccfg name=\"NorthWind\" flatten=\"true\"\u003e\n    \u003c!-- commented out for brevity --\u003e\n\u003c/cfg\u003e\n```\n\nWhen you re-initialize, a single output structure named *NorthWindFlat* is created and populated. \nYou may query it just as you queried *NorthWindStar*.\n\n### More Relationships\n\nTo add all the entities from NorthWind database (diagrammed above), follow the *Add an Entity* \nprocess (above) for *Products*, *Customers*, *Employees*, *Shippers*, *Suppliers*, and *Categories*.\n\nIn the end, the relationships should look like this:\n\n```xml\n\u003crelationships\u003e\n  \u003c!-- following Orders to Customers, Employees, and Shippers --\u003e\n  \u003cadd left-entity=\"Order Details\" left-field=\"OrderID\" right-entity=\"Orders\" right-field=\"OrderID\" /\u003e\n  \u003cadd left-entity=\"Orders\" left-field=\"CustomerID\" right-entity=\"Customers\" right-field=\"CustomerID\" /\u003e\n  \u003cadd left-entity=\"Orders\" left-field=\"EmployeeID\" right-entity=\"Employees\" right-field=\"EmployeeID\" /\u003e\n  \u003cadd left-entity=\"Orders\" left-field=\"ShipVia\" right-entity=\"Shippers\" right-field=\"ShipperID\" /\u003e\n\n  \u003c!-- following Products to Suppliers and Categories --\u003e\n  \u003cadd left-entity=\"Order Details\" left-field=\"ProductID\" right-entity=\"Products\" right-field=\"ProductID\" /\u003e\n  \u003cadd left-entity=\"Products\" left-field=\"SupplierID\" right-entity=\"Suppliers\" right-field=\"SupplierID\" /\u003e\n  \u003cadd left-entity=\"Products\" left-field=\"CategoryID\" right-entity=\"Categories\" right-field=\"CategoryID\" /\u003e\n\u003c/relationships\u003e\n```\n\nIf you'd rather not do all that work, you can use this pre-created [arrangement](./Files/NorthWindEntitiesRelated-cp.xml).\n\nNow when you initialize and run Transformalize, there's a lot going on:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003e\u003etfl -a \"NorthWind.xml\" -m init\u003c/strong\u003e\n\u003cspan style=\"color:#FF7F50;\"\u003ewarn  | NorthWind | Order Details | Initializing\nwarn  | NorthWind | Orders        | Initializing\nwarn  | NorthWind | Products      | Initializing\nwarn  | NorthWind | Customers     | Initializing\nwarn  | NorthWind | Employees     | Initializing\nwarn  | NorthWind | Shippers      | Initializing\nwarn  | NorthWind | Suppliers     | Initializing\nwarn  | NorthWind | Categories    | Initializing\u003c/span\u003e\ninfo  | NorthWind | Order Details | 2155 from input\ninfo  | NorthWind | Order Details | 2155 inserts into output\ninfo  | NorthWind | Orders        | 830 from input\ninfo  | NorthWind | Orders        | 830 inserts into output\ninfo  | NorthWind | Products      | 77 from input\ninfo  | NorthWind | Products      | 77 inserts into output\ninfo  | NorthWind | Customers     | 91 from input\ninfo  | NorthWind | Customers     | 91 inserts into output\ninfo  | NorthWind | Employees     | 9 from input\ninfo  | NorthWind | Employees     | 9 inserts into output\ninfo  | NorthWind | Shippers      | 3 from input\ninfo  | NorthWind | Shippers      | 3 inserts into output\ninfo  | NorthWind | Suppliers     | 29 from input\ninfo  | NorthWind | Suppliers     | 29 inserts into output\ninfo  | NorthWind | Categories    | 8 from input\ninfo  | NorthWind | Categories    | 8 inserts into output\ninfo  | NorthWind |               | 2155 records inserted into flat\ninfo  | NorthWind |               | Time elapsed: 00:00:06.58\n\n\u003cstrong\u003e\u003etfl -a \"NorthWind.xml\"\u003c/strong\u003e\ninfo  | NorthWind | Order Details | Change Detected: No.\ninfo  | NorthWind | Orders        | Change Detected: No.\ninfo  | NorthWind | Products      | Change Detected: No.\ninfo  | NorthWind | Customers     | Change Detected: No.\ninfo  | NorthWind | Employees     | Change Detected: No.\ninfo  | NorthWind | Shippers      | Change Detected: No.\ninfo  | NorthWind | Suppliers     | Change Detected: No.\ninfo  | NorthWind | Categories    | Change Detected: No.\ninfo  | NorthWind |               | Time elapsed: 00:00:01.40\n\u003c/pre\u003e\n\n![AllEntities](./Files/Demo/all-entities-cp.gif \"All Entities\")\n\n### Incrementals (Part 2)\n\nLet's simulate a data change.\n\n```sql\nUPDATE Customers\nSET CompanyName = 'Bottom Dollar Markets'\nWHERE CustomerID = 'BOTTM';\n```\nNow run Transformalize again:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003e\u003etfl -a \"NorthWind.xml\"\u003c/strong\u003e\ninfo  | NorthWind | Order Details | Change Detected: No.\ninfo  | NorthWind | Orders        | Change Detected: No.\ninfo  | NorthWind | Products      | Change Detected: No.\ninfo  | NorthWind | Customers     | Change Detected: Input: 0x75ad2 \u003e Output: 0x73bb5\n\u003cstrong\u003einfo  | NorthWind | Customers     | 1 from input\ninfo  | NorthWind | Customers     | 1 to output\ninfo  | NorthWind | Customers     | 1 updates to output\u003c/strong\u003e\ninfo  | NorthWind | Employees     | Change Detected: No.\ninfo  | NorthWind | Shippers      | Change Detected: No.\ninfo  | NorthWind | Suppliers     | Change Detected: No.\ninfo  | NorthWind | Categories    | Change Detected: No.\n\u003cstrong\u003einfo  | NorthWind |               | 35 records updated in flat\u003c/strong\u003e\ninfo  | NorthWind |               | Time elapsed: 00:00:01.79\n\u003c/pre\u003e\n\nUsing the version, Transformalize picked up the one change in *Customers*.  Since this \ncustomer has purchased 35 items (in *Order Details*), the flat table is updated as well.\n\n![Incrementals](./Files/Demo/incrementals-cp.gif \"Incrementals\")\n\n### Transformations to Make Life Easier\n\n\u003e * Introducing the **`copy`** transform\n\u003e * the **`datePart`** transform\n\u003e * the **`format`** transform\n\u003e * the **`toUpper`** transform\n\nMost often, in addition to de-normalization, you'll need to transform records too. \nTransformalize de-normalizes and transforms at the same time (thus, the name).\n\nLet's add some time [dimension](https://en.wikipedia.org/wiki/Dimension_(data_warehouse)) fields. \nModify the *Orders* entity to include a `\u003ccalculated-fields/\u003e` section like this:\n\n```xml\n\u003ccalculated-fields\u003e\n  \u003cadd name=\"OrderYear\" type=\"int\" t=\"copy(OrderDate).datePart(year)\" /\u003e\n  \u003cadd name=\"OrderMonthSortable\" t=\"format({OrderDate:MM-MMM}).toUpper()\" /\u003e\n  \u003cadd name=\"OrderDaySortable\" t=\"format({OrderDate:yyyy-MM-dd})\" /\u003e\n  \u003cadd name=\"OrderDayOfWeek\" t=\"copy(OrderDate).datePart(dayOfWeek)\" /\u003e\n\u003c/calculated-fields\u003e\t\t\n```\n\n**Note**: The **`copy`** method is mainly used to copy \nother fields into your transformation.  Generally speaking, when a \ntransform uses field names in it's expression (e.g. **`js`**, **`cs`**, and **`format`**), \nyou don't need to preceed it with a **`copy`** method.\n\nAfter re-initializing, *NorthWindFlat* has some helpful time related fields that allow you \nto run queries like:\n\n```sql\nSELECT OrderDayOfWeek AS \"Day\", SUM(Revenue) AS \"Sales\"\nFROM NorthWindFlat\nGROUP BY OrderDayOfWeek\n```\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003eDay         Sales\u003c/strong\u003e\nFriday      284393.64\nMonday      275256.90\nThursday    256143.26\nTuesday     272113.27\nWednesday   266546.72\n\u003c/pre\u003e\n\nNote that the query isn't dealing with joins or parsing dates. This is \nbecause we de-normalized it and pre-calculated useful fields.\n\n![Time Dimension](./Files/Demo/time-dimension-cp.gif \"Time Dimension\")\n\n## Post De-Normalization\n\n\u003e * Introducing system fields in output\n\u003e * the **`read-only`** attribute\n\nTransformalize must use a relational output to de-normalize (i.e. PostgreSQL).  However, now that it's flat, we can leverage the non-relational providers as well.\n\nTransformalize records four *system* fields that may \nbe used by additional `tfl` arrangements and/or other systems:\n\n* TflKey - a surrogate key (an auto-incrementing value)\n* TflBatchId - a version number corresponding to `tfl` runs\n* TflHashCode - a numerical value calculated from every field (used for comparisons)\n* TflDeleted - a boolean field tracking deletes (an optional setting)\n\n**Note:** You can disable system fields by setting `read-only` \nto `true` in the top-most `\u003ccfg/\u003e` element.\n\n### Leveraging Elasticsearch \u0026 Kibana\n\n\u003e Introducing the **elasticsearch** provider\n\nThis section demonstrates how to load the flattened Northwind \ndata into [Elasticsearch](https://www.elastic.co/products/elasticsearch) \nand view it with [Kibana](https://www.elastic.co/products/kibana).\n\n#### Elasticsearch\n\nStart a new arrangement with this in your XML editor:\n\n```xml\n\u003ccfg name=\"NorthWind\"\u003e\n  \u003cconnections\u003e\n    \u003cadd name=\"input\" provider=\"postgresql\" user=\"postgres\" password=\"Secret1!\" database=\"TflNorthwind\" /\u003e\n    \u003cadd name=\"output\" \n         provider=\"elasticsearch\" \n         server=\"localhost\" \n         port=\"9200\" \n         index=\"northwind\" \n         version=\"7.9.3\" /\u003e\n  \u003c/connections\u003e\n  \u003centities\u003e\n    \u003cadd name=\"NorthWindFlat\" version=\"TflBatchId\" \u003e\n      \u003cfields\u003e\n        \u003cadd name=\"TflKey\" alias=\"Key\" type=\"int\" primary-key=\"true\" /\u003e\n        \u003cadd name=\"TflBatchId\" alias=\"Version\" type=\"int\" /\u003e\n        \u003cadd name=\"Revenue\" type=\"decimal\" precision=\"19\" scale=\"2\" /\u003e\n        \u003cadd name=\"Freight\" type=\"decimal\" precision=\"19\" scale=\"4\" /\u003e\n        \u003cadd name=\"OrderDate\" type=\"datetime\" /\u003e\n        \u003cadd name=\"OrderYear\" type=\"int\" /\u003e\n        \u003cadd name=\"OrderMonthSortable\" /\u003e\n        \u003cadd name=\"Country\" length=\"15\" /\u003e\n        \u003cadd name=\"CategoryName\" length=\"15\" /\u003e\n      \u003c/fields\u003e\n    \u003c/add\u003e\n  \u003c/entities\u003e\n\u003c/cfg\u003e\n```\n\nThis arrangement uses an elasticsearch output.  Save as *NorthWindToES.xml* and run in it:\n\n\u003cpre style=\"font-size:smaller;\"\u003e\n\u003cstrong\u003e\u003etfl -a c:\\temp\\NorthWindToES.xml -m init\u003c/strong\u003e\nwarn  | NorthWind | NorthWindFlat | Initializing\ninfo  | NorthWind | NorthWindFlat | 2155 from input\ninfo  | NorthWind | NorthWindFlat | 2155 to output\ninfo  | NorthWind |               | Time elapsed: 00:00:02.40\n\n\u003cstrong\u003e\u003etfl -a c:\\temp\\NorthWindToES.xml\u003c/strong\u003e\ninfo  | NorthWind | NorthWindFlat | Starting\ninfo  | NorthWind | NorthWindFlat | Change Detected: No.\ninfo  | NorthWind |               | Time elapsed: 00:00:00.30\n\u003c/pre\u003e\n\nA quick query in your browser can confirm records loaded:\n\n[http://localhost:9200/northwind/northwindflat/_search?q=*:*\u0026size=0](http://localhost:9200/northwind/northwindflat/_search?q=*:*\u0026size=0)\n\n```json\n{\n    \"took\": 2,\n    \"timed_out\": false,\n    \"_shards\": {\n        \"total\": 5,\n        \"successful\": 5,\n        \"failed\": 0\n    },\n    \"hits\": {\n        \"total\": 2155,\n        \"max_score\": 0.0,\n        \"hits\": []\n    }\n}\n```\n\n![NorthWind in Kibana](./Files/Demo/elasticserach-kibana-cp.gif \"Northwind in Kibana\")\n\n### Leveraging the Orchard Core CMS Module\n\nThe [OrchardCore.Transformalize](https://github.com/dalenewman/OrchardCore.Transformalize) module allows you to:\n\n* edit, store, and secure your arrangements\n* run your arrangements as tasks (like the CLI does)\n* view and page through your arrangements as reports\n* export search results\n* compose bulk actions; select records from your report and run tasks on them.\n\nHere's a quick video of a Northwind report using the Elasticsearch provider we loaded earlier:\n\n![NorthWind in Orchard Core CMS](./Files/Demo/orchardcore-transformalize-cp.gif \"Northwind in Orchard Core\")\nThe arrangement for this is:\n\n```xml\n\u003ccfg name=\"NorthWind\"\u003e\n  \u003cconnections\u003e\n    \u003cadd name=\"input\" \n         provider=\"elasticsearch\" \n         server=\"host.docker.internal\" \n         index=\"northwind\" \n         version=\"7.9.3\" /\u003e\n  \u003c/connections\u003e\n  \u003centities\u003e\n    \u003cadd name=\"northwindflat\" alias=\"NorthWind\" page=\"1\" size=\"10\" sortable=\"true\" \u003e\n      \u003cfields\u003e\n        \u003cadd name=\"tflkey\" alias=\"Key\" type=\"long\" primary-key=\"true\" output=\"false\" /\u003e\n        \u003cadd name=\"orderyear\" type=\"long\" label=\"Year\" parameter=\"facet\" /\u003e\n        \u003cadd name=\"ordermonthsortable\" label=\"Month\" parameter=\"facet\" /\u003e\n        \u003cadd name=\"orderdate\" type=\"datetime\" label=\"Date\" format=\"yyyy-MM-dd\" /\u003e\n        \u003cadd name=\"country\" label=\"Country\" length=\"15\" /\u003e\n        \u003cadd name=\"categoryname\" length=\"15\" label=\"Category\" parameter=\"facet\" /\u003e\n        \u003cadd name=\"freight\" label=\"Freight\" type=\"decimal\" precision=\"19\" scale=\"4\" format=\"$#,###,###.00\" /\u003e\n        \u003cadd name=\"revenue\" label=\"Revenue\" type=\"decimal\" precision=\"19\" scale=\"2\" format=\"$#,###,###.00\" /\u003e\n      \u003c/fields\u003e\n    \u003c/add\u003e\n  \u003c/entities\u003e\n\u003c/cfg\u003e\n```\n\n\u003e * Introducing the `parameter` attribute in fields.\n\nThe report arrangement above is using the field's `parameter` attribute.  This is only for \nthe Orchard Core module.  It is a short-cut that sets up parameters and filters.  It has three settings:\n\n- *facet* : a drop-down for selecting a single value \n- *facets* : a drop-down for selecting multiple values\n- *search* : a text box for searching\n\nThe OrchardCore module has many specific \"web\" features.  More information can be found [here](https://github.com/dalenewman/OrchardCore.Transformalize).\n\n## To Be Continued\n\nThis is the end for now.  This article provides a brief overview of how you'd denormalize a relational database and make use of the flattened output in various ways. I will try to get more documentation created as time permits.\n\n**Note**: This readme is for the updated cross-platform version of Transformalize, the old version is [here](README_LEGACY.md).","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdalenewman%2Ftransformalize","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdalenewman%2Ftransformalize","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdalenewman%2Ftransformalize/lists"}