{"id":21068951,"url":"https://github.com/cloudprivacylabs/leap-sh","last_synced_at":"2025-03-14T02:23:43.554Z","repository":{"id":59427243,"uuid":"515645320","full_name":"cloudprivacylabs/leap-sh","owner":"cloudprivacylabs","description":"Semantic Interoperability for Electronic Health Data Using the Layered Schemas Architecture","archived":false,"fork":false,"pushed_at":"2023-08-18T14:41:54.000Z","size":64495,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-20T21:39:30.494Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudprivacylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-19T15:39:32.000Z","updated_at":"2023-08-08T23:06:48.000Z","dependencies_parsed_at":"2024-11-19T18:54:34.505Z","dependency_job_id":null,"html_url":"https://github.com/cloudprivacylabs/leap-sh","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fleap-sh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fleap-sh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fleap-sh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fleap-sh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudprivacylabs","download_url":"https://codeload.github.com/cloudprivacylabs/leap-sh/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243510056,"owners_count":20302296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T18:29:39.584Z","updated_at":"2025-03-14T02:23:43.528Z","avatar_url":"https://github.com/cloudprivacylabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Semantic Interoperability for Electronic Health Data Using the Layered Schemas Architecture\n\nThis repository contains the artifacts for the ONC LEAP Project titled\n\"Semantic Interoperability for Electronic Health Data Using the\nLayered Schemas Architecture\". For more information, questions, and\nsupport, contact:\n\n * [DARTNet Institute](https://dartnet.info/contact.htm)\n * [Cloud Privacy Labs](https://cloudprivacylabs.com/)\n\nThe project's vision is to build an integrated ecosystem of tools and\ntechnologies that connect and harmonize health and health-related data\nobtained from organizations of all sizes, as well as individuals,\nsupported by a person-centric and privacy-conscious approach to data\naccess and sharing among patients, clinical settings, and researchers.\n\n![Project Vision](assets/leap-vision.png)\n\nAll data files included in the repository are synthetic data files.\n\n## Introduction\n\nThe longitudinal health data from large diverse populations with\nvarying social, economic, geographic, and environmental conditions is\na highly valuable resource for medical and public health researchers\nthrough the creation of various data commons where disparate data are\nstructured and harmonized to expand research options. Many challenges\nhinder the complete and efficient capture and exchange of health data,\nincluding:\n\n  1. a lack of semantic interoperability across systems\n  2. the varying adoption of data standards within and between systems\n  3. a lack of standardized metadata, and\n  4. the poor integration of electronic health records (EHR) data with data from other relevant\nsources such as social services, environmental measurements,\npatient-entered, and data collected from wearable devices.\n\nEHR data can be highly variable between different sources and even\nwithin a single EHR system, due to the differences between EHR\nvendors, data models, and underlying data codification approaches; as\nwell as institutional conventions, and end users’ data capture\ndecisions amongst many other reasons. Customized\nExtraction-Transformation-Loading (ETL) pipelines are the key for\nsemantic harmonization in a traditional data warehousing\noperation. However, these data transformations are usually lossy, even\nif not intended to be so, because of inexact mapping of semantics to\nthe target data model. The process does not scale well when numerous\ndata sources are involved as it requires many customizations that are\nspecific to a particular problem, location, and/or data source.\n\nThis project uses layered schemas as a scalable semantic harmonization\nsolution for data warehousing applications. Layered schemas technology\nconsists of a schema base that represents core attributes of the\ncaptured data, along with multiple interchangeable overlays that\nrepresent variations due to different EHR vendors, jurisdictions, or\norganizational conventions. A layered schema also defines a mapping\nbetween the input data elements and an abstract data model represented\nas a labeled property graph. The annotated graph representation can be\ntransformed into a common data model or used directly, such as into a\ntraining data set for an AI application using the source data and the\nmetadata associated with it.\n\n## Scope\n\nEven though the project scope included different types of EHR data at\nthe outset, there was a lot of intereset to incorporate non-EHR health\nrelated data. Because of this, we worked on the harmonization of the\nfollowing types of data:\n\n  * Raw EHR data captured CSV files,\n  * FHIR data in JSON format,\n  * CCDAs in XML format,\n  * Social health needs assessment tools captured as spreadsheets in\n    various forms\n  * Pharmacy data captured as CSV files.\n  \nThis repository contains synthetic data that reflects the shape and\nvariations of data we collected from our partners.\n\n## Layered Schemas\n\nA **schema** is a machine-readable document that describes the\nstructure of data. JSON and XML schemas are widely used to generate\nexecutable code from specifications and to check structural validity\nof data. LSA extends schemas (such as FHIR or OMOP schemas) with\nlayers (overlays) to add semantic annotations. The semantic\nannotations add ontology mappings, contextual metadata, tags, and\nprocessing instructions that control data ingestion and\nnormalization. A **schema variant** is composed of a schema and a\nset of overlays, and contains the combination of annotations given in\nthe schema and the overlays.\n\nA schema variant is represented using a **labeled property graph**\n(LPG) that has a node for each data field. An LPG is a directed graph\nwhere every node and edge contain a set of **labels** describing its\ntype or class, and a set of **properties** that represent named\nvalues. An LPG allows assigning tags that represent different types of\nmetadata to fields. A field may be a simple value, a structured object\n(e.g a JSON object, array, polymorphic object), or a reference to\nanother schema.\n\n![Layered schemas for data ingestion](assets/layers_ingestion.png)\n\nDifferent schema variants can be used to ingest data that shows\nvariations depending on the data source. Data variations can be\nstructural (e.g. additional data fields, extensions) or semantic\n(e.g. measurements in different units, different ontologies or coding\nsystems), and can be due to different vendor implementations, local\nconventions, or regulations. Ingesting structured data using a schema\nvariant creates an LPG whose nodes combine the annotations from the\nschema variant and data values from the input.\n\n## Layered Schemas for Interoperability\n\nThe focus of layered schema architecture is achieving interoperability\nin an environment with multiple data standards. These standards can be\n\n  * structural, such as data represented as a FHIR message, or CCDA,\n  * representational, e.g. different date formats, data entry conventions,\n  * semantic, e.g. different coding systems, measurement units.\n\nThe goal is **not** to ingest and normalize data with an ETL process,\nbut to ingest data verbatim with annotations that capture context and\nmetadata. The actual interpretation of data happens closer to the\nanalytics side of the process.\n\nStructural differences in ingested data are solved using labeled\nproperty graphs as the core data model. A labeled property graph\nallows storing individual data elements and the relationships between\nthose data elements together with semantic annotations. Data is\ningested using a layered schema describing the input data elements,\nand converted into an LPG. Then source-specific transformations are\nperformed on that LPG to map it to a graph model. The schemas and the\ntransformations used to process the input can be reused and shared.\n\nRepresentational differences are solved using a type system built into\nthe layered schema tools, and valueset lookups. For example, the input\nmay contain a date field:\n\n```\n   dob\n----------\n2001-01-04\n```\n\nThe schema that ingests this field can be defined as:\n\n```\n\"birthDate\": {\n  \"@type\": \"Value\",\n  \"valueType\": \"xs:date\",\n  \"attributeName\": \"dob\"\n}\n```\n\n`xs:date` is the XML date format that says the date is of the form\n`YYYY-MM-DD`, Which causes the field to be ingested as as graph node:\n\n```\n+-----------------------+\n|   :Value              |\n+-----------------------+\n| id: birthDate         |\n| valueType: xs:date    |\n| attributeName: dob    |\n| value: 2001-01-04     |\n+-----------------------+\n```\nWhen data is extracted from this node using a schema that looks like:\n\n```\n\"birthYear\": {\n  \"@type\": \"Value\",\n  \"valueType\": \"xs:gYear\"\n```\n\nwhere `xs:gYear` is the XML data type \"Gregorian Year\", the output will be:\n\n```\nbirthYear\n--------\n2001\n```\n\nSo the ingestion phase ingests data with type information, and when\nexported using a different type, it is converted if possible.\n\n[This is the complete set of data types supported by LSA.](https://layeredschemas.org/docs/data_types/)\n\n\nData entry and data capture conventions for enumerated data may\nrequire dictionary lookups to correctly interpret data. For example,\none data source may report:\n\n```\ngender\n------\nF\nM\n```\n\nand another may report:\n\n```\ngender\n-----\nFemale\nMale\n```\n\nThis can be modeled using a schema:\n\n```\n\"gender\": {\n   \"@type\": \"Value\",\n   \"attributeName\": \"gender\"\n}\n```\n\nAnd an overlay for data source 1 with:\n\n```\n\"gender\": {\n   \"@type\": \"Value\",\n   \"vsValuesets\": \"gender_source_1\",\n   \"vsResultValues\": [ \"normalized_gender\"]\n},\n\"normalizedGender\": {\n   \"@type\": [\"Value\",\"OMOPConcept\"]\n}\n```\n\nAnd a valueset:\n\n```\n{\n  \"id\": \"gender_source_1\",\n  \"values\": [\n      {\n         \"values\": [\"F\"],\n         \"result\": \"8532\"\n      },\n      {\n         \"values\": [\"M\"],\n         \"result\": \"8507\"\n      }\n    ]\n}\n```\n\nAnd for data source 2 with:\n\n```\n\"gender\": {\n   \"@type\": \"Value\",\n   \"vsValuesets\": \"gender_source_2\",\n   \"vsResultValues\": [ \"normalized_gender\"]\n},\n\"normalizedGender\": {\n   \"@type\": [\"Value\",\"OMOPConcept\"]\n}\n```\n\n```\n{\n  \"id\": \"gender_source_1\",\n  \"values\": [\n      {\n         \"values\": [\"Female\"],\n         \"result\": \"8532\"\n      },\n      {\n         \"values\": [\"Male\"],\n         \"result\": \"8507\"\n      }\n    ]\n}\n```\n\nThe codes 8532 and 8507 are OMOP concept IDs for \"female\" and \"male\".\n\nThe ingested data becomes:\n\n```\n+-----------------------+   +-----------------------+\n|   :Value              |   | :Value:OMOPConcept    |\n+-----------------------+   +-----------------------+\n| id: gender            |   |id: normalizedGender   |\n| attributeName: gender |   |value: 8532            |\n| value: F              |   |                       |\n+-----------------------+   +-----------------------+\n\n```\n\n```\n+-----------------------+   +-----------------------+\n|   :Value              |   | :Value:OMOPConcept    |\n+-----------------------+   +-----------------------+\n| id: gender            |   |id: normalizedGender   |\n| attributeName: gender |   |value: 8532            |\n| value: Female         |   |                       |\n+-----------------------+   +-----------------------+\n\n```\n\nSuch lookups allow creation of codified nodes as data is\ningested. This way, the stored data contains both the raw input and\nthe concept normalized based on a terminology. Multiple terminologies\ncan also be used. The data model is not strict and can be extended\nad-hoc as needed.\n\nSuch valueset lookups are feasible for a small number of\noptions. There are cases where an terminology database is needed. The\nLSa tooling provides facilities to call out to third-party servers to\nperform terminology lookups. \n\n## Data Flow Overview\n\nThe overall process of ingestion, harmonization, and the extraction of\ndata in a common model is depicted in the following diagram. This\nproject primarily works with OMOP, but the model is flexible enough to\nbe adapted to other common data models.\n\n![Semantic Data Warehouse](assets/dw-arch.png)\n\nThe input to the ingestion process contains support for the following\ndata formats:\n\n * FHIR data in JSON format. Every patient record is a FHIR Bundle\n   resource stored in a separate JSON file.\n * CCDA data in XML format. Every patient record is a separate XML\n   file.\n * Raw EHR data in CSV format. This data files are obtained by running\n   EHR reports, thus show a great deal of structural and semantic\n   variability. There are usually multiple files, each representing a\n   single object (like patient, or measurement), but there are cases\n   where the result of a SQL JOIN is provided as a single file.\n\nFor most CSV data, we write custom schemas to describe the input\nformat.  For CCDAs, we developed a set of interlinked schemas and\noverlays to extract relevant parts of a CCDA document. FHIR ingestion\nuses HL7 published FHIR schemas with additional overlays to link\nvocabulary lookups, type variations, and other processing directives.\nThen, data files are ingested and converted into labeled property\ngraph form.\n\nFor FHIR and CCDA files, a single file can generate a graph containing\nall the necessary elements. For CSV files, each row usually contains a\nsingle data object (like a patient, or an observation for a\npatient). Thus, when ingested, these records must be linked to form a\ncommon patient view. This linking is done by using special annotations\non the schemas. For instance, the schema for an observation table\ncontains a foreign key to the patient table.\n\nThe harmonization process does **not** normalize the data elements to\na common format. Instead, input data elements are tagged with type\ninformation and additional annotations that describe how they can be\ninterpreted.\n\nOnce ingested, all data elements are harmonized and translated into an\nintermediate graph model. This graph model is not a strict structural\nconstruct. The model is Person centric, and contains the following\nelements:\n\n * Person: Demographic information and identity\n   * Condition: Multiple conditions are linked to a Person. Condition\n     entries include start/end dates, raw input data, and\n     codifications.\n   * DrugExposure: All drug related activity is recorded\n     here. Multiple DrugExposure entries are linked to a Person.\n   * Measurement: All laboratory measurements are recorded\n     here. Multiple measurement entries are linked to a Person. The\n     same measurement may appear multiple times, in different dates.\n   * Observation: All observations are recored here. Multiple\n     observation entries are linked to a Person.\n   * Procedure: All procedures are recoded here. Multiple procedure\n     entries are linked to a Person.\n\nThe graph model is stored in a graph database. This project uses Neo4j\ncommunity version to store the ingested data, but other graph\ndatabases that have openCypher support can be adapted as well. It is\nalso possible to change the data ingestion pipelines so that a\nrelational database, or a document database can be used to store data.\n\nOMOP output is generated from this data model by mapping individual\nfields to OMOP schema fields.\n\n## Building and Running the Tools\n\nThis repository contains a `pipeline` program that can be built and\nused to ingest data contained in this repository. You can get a\npre-built binary for your platform, or build it yourself using the Go\nlanguage build system.\n\nTo build, after cloning this repository, run:\n\n```\ncd pipeline\ngo build\n```\n\nThis will build the `pipeline` binary in the current directory.\n\n## Acknowledgments\n\nThis project is supported by the Office of the National Coordinator\nfor Health Information Technology (ONC) of the U.S. Department of\nHealth and Human Services (HHS) under grant number 90AX0034, Semantic\nInteroperability for Electronic Health Data Using the Layered Schemas\nArchitecture, total award $999,990 with 100% financed with federal\ndollars and 0% financed with non-governmental sources. This\ninformation or content and conclusions are those of the author and\nshould not be construed as the official position or policy of, nor\nshould any endorsements be inferred by ONC, HHS, of the\nU.S. Government.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudprivacylabs%2Fleap-sh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudprivacylabs%2Fleap-sh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudprivacylabs%2Fleap-sh/lists"}