{"id":48822120,"url":"https://github.com/intersystems-ib/iris-datapipe","last_synced_at":"2026-04-14T15:34:31.096Z","repository":{"id":47167115,"uuid":"250022533","full_name":"intersystems-ib/iris-datapipe","owner":"intersystems-ib","description":"DataPipe an interoperability framework to ingest data in InterSystems IRIS in a flexible way","archived":false,"fork":false,"pushed_at":"2025-12-03T09:54:00.000Z","size":7258,"stargazers_count":0,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-06T10:33:51.155Z","etag":null,"topics":["data-processing","interoperability","intersystems-iris"],"latest_commit_sha":null,"homepage":"","language":"ObjectScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intersystems-ib.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-03-25T15:51:59.000Z","updated_at":"2025-12-03T09:54:04.000Z","dependencies_parsed_at":"2025-01-24T09:31:10.656Z","dependency_job_id":null,"html_url":"https://github.com/intersystems-ib/iris-datapipe","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/intersystems-ib/iris-datapipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intersystems-ib%2Firis-datapipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intersystems-ib%2Firis-datapipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intersystems-ib%2Firis-datapipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intersystems-ib%2Firis-datapipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intersystems-ib","download_url":"https://codeload.github.com/intersystems-ib/iris-datapipe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intersystems-ib%2Firis-datapipe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31803602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T11:13:53.975Z","status":"ssl_error","status_checked_at":"2026-04-14T11:13:53.299Z","response_time":153,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-processing","interoperability","intersystems-iris"],"created_at":"2026-04-14T15:34:30.329Z","updated_at":"2026-04-14T15:34:31.076Z","avatar_url":"https://github.com/intersystems-ib.png","language":"ObjectScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"DataPipe is an interoperability framework to ingest data in InterSystems IRIS in a flexible way.\n\n\u003cimg src=\"img/datapipe-diagram2.png\" width=\"800\" /\u003e\n\n# QuickStart\nYou can try out the included quickstart container to have a look at DataPipe. \n\n## Run container\n```bash\ndocker compose -f docker-compose.quickstart.yml up -d\n```\n\n## Interoperability production\nDataPipe uses interoperability framework.\n\nCheck [DataPipe.Test.Production](http://localhost:52773/csp/dpipe/EnsPortal.ProductionConfig.zen?PRODUCTION=DataPipe.Test.Production) interoperability production loggin-in as `superuser` / `SYS`.\n\nNow, you only need to generate some data.\n\n## Generate sample data\nNow you can generate some sample data that will be processed in your pipes:\n\n* Open an [WebTerminal](http://localhost:52773/terminal/) interactive session (terminal)\n\n* Generate data\n```objectscript\ndo ##class(DataPipe.Test.Helper).QuickStart()\n```\n\n* You can have a look at the [DataPipe.Test.Production](http://localhost:52773/csp/dpipe/EnsPortal.ProductionConfig.zen?PRODUCTION=DataPipe.Test.Production) and see how messages has been processed.\n\n## DataPipeUI\nDataPipe includes an UI that allows you to manage the data that has been processed.\n\nFollow these steps to run [datapipeUI](https://github.com/intersystems-ib/iris-datapipeUI).\n\n1. In other directory, clone [datapipeUI](https://github.com/intersystems-ib/iris-datapipeUI) repository\n\n```bash\ngit clone https://github.com/intersystems-ib/iris-datapipeUI\n```\n\n2. Run UI container:\n\n```bash\ncd iris-datapipeUI\ndocker compose up -d\n```\n\n3. Log-in the UI at http://localhost:8080 using `dpadmin` / `demo`\n\n* Have a look at the defined **Pipes**\n* Check the Activity **Dashboard** \n* Try out **filtering** the records that have been processed: \n\u003cimg src=\"img/dp2-overview-1.gif\" /\u003e\n\n* Click on some record details and inspect **Ingestion**, **Staging** and **Operation**\n* View the record history and even try to repeat some of the stages and see what happens.\n\u003cimg src=\"img/dp2-overview-2.gif\" /\u003e\n\n# Features\n\n## Defining Pipes\n* You need to define your Pipes: set a code and some description.\n* Optionally you can specify an InterSystems IRIS security resource that will be required to view that pipe.\n\n## Processing data\nWhen processing data you need to follow these steps:\n\n### 1) Define a DataPipe Model\nYou need to define a model for the data you want to ingest. \nA model is a class that extends from [DataPipe.Model.cls](src/DataPipe/Model.cls) where you must implement some methods.\n\n\u003cimg src=\"img/datapipe-model.png\" width=\"200\" /\u003e\n\nIn your model you will implement:\n* How to serialize / deserialize your data (e.g. using JSON or XML).\n* How to Normalize and Validate your data.\n* And finally, what operation you want to run with your data after it is normalized and validated.\n\nHave a look at these examples:\n* [DataPipe.Test.HL7.Models.A08.cls](src/DataPipe/Test/HL7/Models/A08.cls) - a model for processing information of incoming HL7 messages\n* [DataPipe.Test.REST.Models.Person.cls](src/DataPipe/Test/REST/Models/Person.cls) - a model for processing information of incoming REST requests with some person data\n\n### 2) Add Interoperability components\n* After defining your model, you need to setup an interoperability production using DataPipe components\n* DataPipe provides Business Processes and Operations that can work with a DataPipe model.\n* The only Business Process you must define is your Ingestion Process.\n\n\u003cimg src=\"img/datapipe-components.png\" width=\"800\" /\u003e\n\n#### 2.1) Create an Ingestion Process\nYou need to implement an **Ingestion process** using [DataPipe.Ingestion.BP.IngestionManagerContext](src/DataPipe/Ingestion/BP/IngestionManagerContext.cls) as context.\n\nThis process will receive an input and will implement:\n* `Input \u003e InboxAttributes`: \n    * Extract some input information and define the attributes of your data.\n    * The **Pipe** the data belongs to is an attribute you must specify.\n    * These attributes will describe your data and will be used afterwards for searching.\n* `Input \u003e Model`: \n    * Decide how your incoming data will be transformed into the DataPipe model you have previously defined.\n\nYou can find examples here:\n* [DataPipe.Test.HL7.BP.HL7Ingestion.cls](src/DataPipe/Test/HL7/BP/HL7Ingestion.cls) - ingestion process for incoming HL7 messages\n* [DataPipe.Test.REST.BP.RESTIngestion.cls](src/DataPipe/Test/REST/BP/RESTIngestion.cls) - ingestion process for incoming REST messages\n\n#### 2.2) Add the rest of components\nRest of the components are provided by DataPipe you need to add are provided by DataPipe.\nThese components will call the different methods you have in your model.\n\nUsually you will add:\n* The Ingestion process you have implemented before.\n* A [DataPipe.Staging.BP.StagingManager](src/DataPipe/Staging/BP/StagingManager.cls) process.\n* A [DataPipe.Oper.BP.OperManager](src/DataPipe/Oper/BP/OperManager.cls) process.\n* And a [DataPipe.Oper.BO.OperationHandler](src/DataPipe/Oper/BO/OperationHandler.cls) operation.\n\nHave a look at a full example in [DataPipe.Test.Production.cls](src/DataPipe/Test/Production.cls)\n\n# Installation\n1) Install [IPM package manager](https://github.com/intersystems/ipm) if you don't have already done it.\n2) Create a new namespace (e.g. `DPIPE`)\n3) Switch to the namespace you want to install DataPipe.\n4) Install DataPipe using ipm:\n\n```objectscript\nzpm \"install iris-datapipe\"\n```\n\n5) Set up users and roles as needed (see next section).\n\n## Users and privileges\n\nDataPipe uses different security resources you can assign to InterSystems IRIS user account:\n* `DP_ADMIN` - DataPipe Administrator\n* `DP_MENU_DASHBOARD` - Access to Dashboard menu option in UI\n* `DP_MENU_SEARCH` - Access to Search menu option in UI\n\nYou can use the following as an **EXAMPLE** to set up the users in your system:\n\nCreate resources:\n```objectscript\nzn \"%SYS\"\nwrite ##class(Security.Resources).Create(\"DP_ADMIN\",\"DataPipe Admin Privilege\")\nwrite ##class(Security.Resources).Create(\"DP_MENU_DASHBOARD\",\"DataPipe UI Dashboard Menu Access\")\nwrite ##class(Security.Resources).Create(\"DP_MENU_SEARCH\",\"DataPipe UI Search Menu Access\")\n```\n\nCreate a `DataPipe_Admin` role:\n```objectscript\nwrite ##class(Security.Roles).Create(\"DataPipe_Admin\",\"DataPipe Administrator\",\"DP_ADMIN:RWU,DP_MENU_DASHBOARD:RWU,DP_MENU_SEARCH:RWU,%DB_USER:RW,%DB_IRISSYS:R\")\n```\n\nGrant access to tables and views to `DataPipe_Admin` role:\n```sql\nGRANT INSERT,SELECT,UPDATE ON DataPipe_Data.Pipe, DataPipe_Data.Preference TO DataPipe_Admin\n```\n\n```sql\nGRANT SELECT ON DataPipe_Data.VInbox, DataPipe_Data.VIngestion, DataPipe_Data.VStaging, DataPipe_Data.VOper TO DataPipe_Admin\n```\n\nCreate a new user that belongs to `DataPipe_Admin` role:\n```objectscript\nwrite ##class(Security.Users).Create(\"dpadmin\",\"DataPipe_Admin\",\"demo\")\n```\n\nYou can also check out the [DataPipe_Admin](http://localhost:52773/csp/sys/sec/%25CSP.UI.Portal.Role.zen?PID=DataPipe_Admin) role definition in InterSystems IRIS. \n\n## Using DataPipe in multiple namespaces\n\nYou can install DataPipe in one namespace and use it in different namespaces using the same UI.\n\nTo do so, you need to:\n* Install DataPipe in source namespace\n* Map DataPipe and Restforms2 package and globals to target namespace\n* Grant access to tables and views on target namespace\n\nThe following example uses the **quickstart container** to configure Datapipe also in `USER` namespace:\n\n1) DataPipe is already installed in `DPIPE` namespace.\n2) Map DataPipe and Restforms2 package and globals from `DPIPE` to `USER`:\n\n```objectscript\nzn \"%SYS\"\n\n// package mapping\nset props(\"Database\")=\"DPIPE-DATA\"\nwrite ##class(Config.MapPackages).Create(\"USER\", \"DataPipe\", .props)\nwrite ##class(Config.MapPackages).Create(\"USER\", \"Form\", .props)\n\n// global mapping\nwrite ##class(Config.MapGlobals).Create(\"USER\", \"DataPipe*\", .props)\nwrite ##class(Config.MapGlobals).Create(\"USER\", \"Form*\", .props)\n\n// routine mapping\nwrite ##class(Config.MapRoutines).Create(\"USER\", \"Form*\", .props)\n```\n\n3) Grant access to tables and views on `USER` namespace:\n\n```sql\nGRANT INSERT,SELECT,UPDATE ON DataPipe_Data.Pipe, DataPipe_Data.Preference TO DataPipe_Admin\n```\n\n```sql\nGRANT SELECT ON DataPipe_Data.VInbox, DataPipe_Data.VIngestion, DataPipe_Data.VStaging, DataPipe_Data.VOper TO DataPipe_Admin\n```\n\n4) Load a sample production in `USER` which already have DataPipe components:\n\n```objectscript\nzn \"USER\"\ndo $system.OBJ.Load(\"/app/install/SampleProduction.cls\", \"ck\")\n```\n\n5) Start the production\n\n6) Send a test message\n\n```objectscript\nzn \"USER\"\nwrite ##class(DataPipe.Test.REST.Helper).SendHTTPRequests(1, \"/test/user/api/message\")\n```\n\n6) Check datapipeUI\n\n## DataPipeUI considerations\n\nWhen enabling [datapipeUI](https://github.com/intersystems-ib/iris-datapipeUI), you must consider the following:\n\n### CORS\nYou must consider CORS restrictions.\nYou can see a basic example [here](https://github.com/intersystems-ib/iris-datapipe/blob/master/src/Form/REST/Abstract.cls#L18) that allows any incoming connection (this is only recommended for testing).\n\n### Database resource\nMake sure that `CSPSystem` user in InterSystems IRIS have read permission on the resource of the database where you have installed DataPipe.\n\n# Upgrading from previous version\n\nSee [CONTRIB.md](./CONTRIB.md) *Upgrading from previous versions* section for detailed instructions. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintersystems-ib%2Firis-datapipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintersystems-ib%2Firis-datapipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintersystems-ib%2Firis-datapipe/lists"}