{"id":21068950,"url":"https://github.com/cloudprivacylabs/lsa","last_synced_at":"2025-05-16T03:33:49.691Z","repository":{"id":39991835,"uuid":"350044660","full_name":"cloudprivacylabs/lsa","owner":"cloudprivacylabs","description":"Layered Schema Architecure","archived":false,"fork":false,"pushed_at":"2024-01-22T17:30:48.000Z","size":70517,"stargazers_count":7,"open_issues_count":8,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-21T06:17:19.507Z","etag":null,"topics":["graphs","interoperability","linked-data","schema"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudprivacylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-21T15:50:47.000Z","updated_at":"2023-11-02T21:01:31.000Z","dependencies_parsed_at":"2023-02-17T08:16:55.568Z","dependency_job_id":"2c84340b-b446-40f0-9f20-31ee52533ba8","html_url":"https://github.com/cloudprivacylabs/lsa","commit_stats":{"total_commits":876,"total_committers":4,"mean_commits":219.0,"dds":"0.40753424657534243","last_synced_commit":"fd5437e34441555016019dcb835821b9883fbeb9"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Flsa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Flsa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Flsa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Flsa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudprivacylabs","download_url":"https://codeload.github.com/cloudprivacylabs/lsa/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225405266,"owners_count":17469317,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["graphs","interoperability","linked-data","schema"],"created_at":"2024-11-19T18:29:39.563Z","updated_at":"2024-11-19T18:29:40.187Z","avatar_url":"https://github.com/cloudprivacylabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![GoDoc](https://godoc.org/github.com/cloudprivacylabs/lsa?status.svg)](https://godoc.org/github.com/cloudprivacylabs/lsa)\n[![Go Report Card](https://goreportcard.com/badge/github.com/cloudprivacylabs/lsa)](https://goreportcard.com/report/github.com/cloudprivacylabs/lsa)\n[![Build Status](https://github.com/cloudprivacylabs/lsa/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/cloudprivacylabs/lsa/actions/workflows/CI.yml)\n# Layered Schemas\n\nLayered Schema Architecture (LSA) enables semantic interoperability\nbetween heterogeneous systems. LSA uses a schema (such as FHIR schemas\nfor health data) to define data structures, and overlays to add\nsemantics, contextual metadata, and processing directives. A schema\nvariant is the composition of the schema base and use-case specific\noverlays. Different schema variants can be used to ingest data from or\nexport data to disparate systems. Each variant encodes source specific\nmetadata and rules to ingest data into a knowledge graph, or target\nspecific metadata and rules to translate the knowledge graph into\nanother format.\n\nA layered schema defines a mapping from structured data (in JSON, XML,\ntabular format) to a labeled property graph. For instance, when a JSON\ndocument is ingested using a layered schema, a labeled property graph\nis constructed by combining schema information with the input data\nelements. The resulting graph is self-contained, and includes both the\ninput data and the schema. This allows processing data based on its\nsemantic attributes instead of its attributes names or other\nstructural properties. For example, it is possible to use layered\nschema to ingest health data, mark certain attributes as\n\"identifiable\", and then, remove all attributes that are marked as\nsuch. The algorithm to remove \"identifiable\" elements can be written\nwithout any knowledge of the input structure. Same algorithm can be\nused to de-identify health data as well as marketing data.\n\nThe main documentation site for layered schemas is:\n\nhttps://layeredschemas.org\n\nThis Go module contains the reference implementation of the layered\nschema specification. It contains a layered schema processor to\nslice/compose schemas, import JSON schemas, and ingest data from\nJSON/CSV/XML sources.\n\n\n## Overview\n\nA traditional schema (a JSON schema, an XML schema, etc.) defines one\nor more data types. Such schemas are primarily used for documentation,\ndata validation, and automated code generation. A schema includes\ndefinitions about the shape of the data object such as the list of\nfields, nested fields, or constraints on field contents. A layered\nschema adds additional interchangeable layers to a traditional schema\nso the same schema can be adjusted for different use cases, or can be\nprocessed with different processing directives to handle data\nvariations. The layered schema architecture provides the framework to\nextend traditional schemas using such layers.\n\n### Schemas, overlays, and bundles\n\nA *schema* defines a data structure (attributes). A schema has a\n`valueType` which is the the type of data structure defined by the\nschema (e.g. `Person`).\n\nAn *overlay* defines semantics for attributes, and may override schema\ndefinitions. A `Person` schema can only be composed with `Person`\noverlays.\n\nA *schema variant* is a schema that is the result of the composition\nof a schema with zero or more overlays. \n\nA *bundle* defines the schema variants for multiple `valueType`s. That\nis, a bundle specifies all the schemas and overlays for each type of\nobject used in a particular use case. For instance, a bundle may\nspecify that `Person` means the composition of a `Person` schema and\nan overlay containing terms from data privacy vocabulary. Data\ningested using this bundle will have annotations using data privacy\nvocabulary terms. \n\nBelow is an example schema for a `Person` data type containing\n`firstName` and `lastName`.\n\n```\nperson.schema.json:\n{\n  \"@context\": \"https://lschema.org/v1/ls.json\",\n  \"@id\": \"https://example.org/Person/schema\",\n  \"@type\": \"Schema\",\n  \"valueType\": \"https://example.org/Person\",\n  \"layer\": {\n    \"@type\": \"Object\",\n    \"@id\": \"https://example.org/Person\",\n    \"attributes\": {\n      \"https://example.org/Person/firstName\": {\n        \"@type\": \"Value\"\n      },\n      \"https://example.org/Person/lastName\": {\n        \"@type\": \"Value\"\n     }\n   }\n }\n}\n```\n\nThe `Person` object is defined as an `Object` that contains two\n`Value` attributes: `firstName` and `lastName`. These are the unique\nattribute IDs. An overlay can be defined to add new semantics to these\nattributes. The following overlay adds personal data category `Name`\nto the `lastName` attribute:\n\n```\nperson-dpv.overlay.json:\n{\n  \"@context\": \"https://lschema.org/v1/ls.json\",\n  \"@id\": \"https://example.org/Person/dpv-ovl\",\n  \"@type\": \"Overlay\",\n  \"valueType\": \"https://example.org/Person\",\n  \"layer\": {\n    \"https://example.org/lastName\": {\n      \"@type\": \"Value\",\n      \"http://www.w3.org/ns/dpv#hasPersonalDataCategory\": \"http://www.w3.org/dpv#Name\"\n    }\n  }\n}\n```\n\nA bundle that defines `Person` by composing the person schema and the\ndpv overlay is:\n\n```\n{\n    \"typeNames\" : {\n        \"https://example.org/Person\": {\n            \"schema\" : \"person.schema.json\",\n            \"overlays\" : [\n                {\n                    \"schema\" : \"person-dpv.overlay.json\"\n                }\n            ]\n        }\n    }\n}\n```\n\n### Composing Schema Variants\n\nA schema variant is a schema and zero or more overlays that modify the\nsemantics of the schema for a use-case. A schema variant is composed\nby combining attributes of overlays with matching attributes of the\nschema. By default, the terms specified for the attributes of the\noverlays are combined as a set with the same terms of the schema. That\nis, if a schema has\n\n```\n{\n   \"@id\": \u003cattributeId\u003e,\n   \"@type\": \"Value\",\n   \"someAnnotation\": \"someValue\"\n}\n```\n\nand the overlay has:\n\n```\n{\n   \"@id\": \u003cattributeId\u003e,\n   \"@type\": [\"Value\",\"someType\"],\n   \"someAnnotation\": \"otherValue\"\n}\n```\n\nthe schema variant will have:\n\n```\n{\n   \"@id\": \u003cattributeId\u003e,\n   \"@type\": [\"Value\",\"someType\"],\n   \"someAnnotation\": [\"someValue\", \"otherValue\"]\n}\n```\n\nSome of the terms have override semantics, and can be used to modify\nschema behavior using overlays. For example:\n\n```\n{\n   \"@id\": \u003cattributeId\u003e,\n   \"@type\": \"Value\",\n   \"valueType\": \"string\"\n}\n```\n\nThe `valueType` can be overridden by an overlay:\n\n```\n{\n   \"@id\": \u003cattributeId\u003e,\n   \"@type\": \"Value\",\n   \"valueType\": \"xsd:date\"\n}\n```\n\nTo compose schema variants using the `layers` binary, use:\n\n```\nlayers compose --bundle person-dpv.bundle.json --type https://example.org/Person\n```\n\nThis will compose and output the schema variant for `Person` as defined in the `person-dpv.bundle`.\n\n\n### Compiling Schemas\n\nA schema variant may reference other schema variants. For example, the\n`Person` object in the `examples/contact` directory refers to a\n`Contact` object to represent contact information. Compiling a schema\nvariant resolves these references and creates a single self-contained\nschema. Such a schema can be used for data ingestion.\n\nTo compile the `Person` schema, use:\n\n```\nlayers compile --bundle person-dpv.bundle.json --type https://example.org/Person\n```\n\nThis operation will attach the `Contact` schema variant as configured\nin the bundle to the `Person` variant, resulting in a schema that has\nno external references.\n\n### Ingesting Data\n\nData ingestion process parses an input object, combines the data\nattributes with the matching schema attributes and builds a labeled\nproperty graph (LPG). The LPG contains both the input object and the\nrelevant schema information, so it can be processed using the\nannotations in the graph without any external references.\n\nTo ingest data using the `layers` binary, use:\n\n```\nlayers ingest json person_sample.json --bundle person-dpv.bundle.json --type https://example.org/Person\n```\n\nThis operation will output a graph containing the nodes for the input\nas well as nodes for the schema. All the input nodes will have the\nlabel `https://lschema.org/DocumentNode`, and all schema nodes will\nhave the label `https://lschema.org/Attribute`. Every input node will\nbe connected to its corresponding schema node with a\n`https://lschema.org/instanceOf` edge. The input nodes will contain\n`https://lschema.org/schemaNodeId` annotation giving the ID of the\nschema node for the attribute.\n\nThe schema nodes and input nodes can be combined using the\n`embedSchemaNodes` flag:\n\n```\nlayers ingest json person_sample.json --bundle person-dpv.bundle.json --type https://example.org/Person --embedSchemaNodes\n```\n\nThe output graph will have nodes containing only\n`https://lschema.org/DocumentNode`s that include all the annotations\ngiven in the corresponding schema node. The original schema node can\nstill be found using the `https://lschema.org/schemaNodeId`\nannotation.\n\n\n## Building\n\nOnce you clone the repository, you can build the schema compiler using\nthe Go build system.\n\n```\ncd layers\ngo build\n```\n\nThis will build the `layers` binary in the current directory.\n\n## Examples\n\nThe `examples/` directory contains some example data processing\nscenarios.\n\n## Commercial Support\n\nCommercial support for this library is available from Cloud Privacy Labs: support@cloudprivacylabs.com\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudprivacylabs%2Flsa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudprivacylabs%2Flsa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudprivacylabs%2Flsa/lists"}