{"id":21068946,"url":"https://github.com/cloudprivacylabs/json2rdf","last_synced_at":"2025-08-23T10:02:59.736Z","repository":{"id":180810763,"uuid":"656992820","full_name":"cloudprivacylabs/json2rdf","owner":"cloudprivacylabs","description":"Layered schema approach for converting JSON data into RDF","archived":false,"fork":false,"pushed_at":"2023-07-26T15:24:55.000Z","size":1014,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-03T11:02:28.427Z","etag":null,"topics":["go","golang","json","json-schema","jsonld","rdf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudprivacylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-22T04:49:41.000Z","updated_at":"2025-03-22T18:20:10.000Z","dependencies_parsed_at":"2024-06-21T19:04:12.005Z","dependency_job_id":null,"html_url":"https://github.com/cloudprivacylabs/json2rdf","commit_stats":null,"previous_names":["cloudprivacylabs/json2rdf"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fjson2rdf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fjson2rdf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fjson2rdf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudprivacylabs%2Fjson2rdf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudprivacylabs","download_url":"https://codeload.github.com/cloudprivacylabs/json2rdf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254110374,"owners_count":22016391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","json","json-schema","jsonld","rdf"],"created_at":"2024-11-19T18:29:38.927Z","updated_at":"2025-05-16T03:33:41.993Z","avatar_url":"https://github.com/cloudprivacylabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# json2rdf - JSON-LD vs. Layered Schemas\n\nLayered JSON schemas can be used instead of JSON-LD to translate JSON\ndocuments to knowledge graphs (and thus, to RDF). This is a\nproof-of-concept showing how that's done.\n\nJSON-LD is a format for encoding linked data. It allows mapping of\nJSON properties to a web ontology to enable interoperability between\ndifferents systems. JSON-LD also offers a JSON-based encoding of\nknowledge graphs. A JSON-LD document can be translated into RDF and\nvice-versa.\n\nA layered JSON schema is simply a JSON schema with additional overlays\nthat annotate the schema. This annotated schema (the \"schema variant\")\nis used to ingest a JSON document. The added annotations encode the\nnecessary terminology mappings and graph-shaping instructions to build\na knowledge graph, and thus, RDF.\n\nThere are several advantages of using layered JSON schemas: JSON\nschemas are widely available in the industry to specify data\nstandards. They describe valid JSON documents expected by data\nexchange partners or API users in a machine-readable manner so they\ncan be used to validate JSON documents or to generate code for\ndifferent languages.\n\n## Layered Schema Architecture (LSA)\n\nThe LSA tools are available\n[here](https://github.com/cloudprivacylabs/lsa). Here's an overview\nof the idea:\n\nA **schema variant** is composed of a schema that defines data\nstructures, and zero or more overlays (hence the name \"layered\nschemas\") that annotate the schema with semantic information and\nmetadata. These annotations adjust and enrich the schema by adding or\nremoving constraints, metadata, processing information such as\npointers to normalization tables, or mappings to an ontology.\n\nSchema variants are useful in an environment where there are multiple\nvarying implementation of a standard, or where there are multiple\nstandards or proprietary data structures within an\necosystem. Different schema variants can be used to ingest and\nharmonize disparate data structures.\n\nWhen a document is ingested using LSA tools, the ingestion process\ncreates a labeled property graph (LPG) containing both the input data\nelements and the schema variant. Each node in this graph contains the\ningested data value and the schema information corresponding to the\ndata element. Thus, the ingested LPG is a self-describing object that\ncontains the input values and schema annotations for each data\nelement. The annotations can also contain graph-shaping instructions\nto fine-tune the ingestion process.\n\nThis proof-of-concept uses the LSA packages to compose schema variants\nand ingest JSON documents. The `layers` program from the LSA\nrepository can also be used together with `json2rdf` program in this\nrepository. \n\n## JSON-LD/RDF Translation\n\nLet's consider the following simple JSON-LD document describing a\n`Person`. It uses the `https://schema.org` ontology to describe a\nperson object:\n\n``` javascript\n{\n    \"@context\": \"https://schema.org\",\n    \"@type\": \"Person\",\n    \"@id\": \"http://linkedin.com/jane-doe\",\n    \"address\": {\n        \"@type\": \"PostalAddress\",\n        \"addressLocality\": \"Denver\",\n        \"addressRegion\": \"CO\"\n    },\n    \"colleague\": [\n        \"http://example.com/John.html\",\n        \"http://example.com/Amy.html\"\n    ],\n    \"email\": \"jane@example.com\",\n    \"name\": \"Jane Doe\",\n    \"sameAs\" : [ \"https://facebook.com/jane-doe\",\n                 \"https://twitter.com/jane-doe\"]\n}\n```\n\nJSON-LD uses the `context` to map JSON keys to RDF. This is done by\nmapping individual object properties to concepts in an ontology (the\n`context` defines the \"semantics\" of the data.) The graphical RDF\nrepresentation for this object is:\n\n![Person](person-rdf.png)\n\nAs you can see, the structure of the output graph depends on both the\nJSON-LD context mappings and the structure of the JSON input file. A\nJSON object in the input file is represented as a node in the output\ngraph, and a JSON property is represented as an edge (predicate). If\nthe JSON object does not have an `@id`, then it is translated into a\nblank node.\n\nThis process can be summarized as:\n\n![JSON-LD to RDF](jsonld-to-rdf.png)\n\n## JSON/RDF Translation using a Layered JSON Schema\n\n### Idea\n\nThe goal is to produce RDF from a JSON document using a JSON schema,\nas opposed to using a JSON-LD document and context. For this, we will\nannotate the JSON schema using an overlay (remember: schema + overlay\n= schema variant), ingest the JSON document, and translate the\nresulting LPG to RDF using the annotations embedded in the ingested\nobject.\n\nFirst, we write a JSON schema to describe the data structures\n(`Person` and `PostalAddress`). Then we combine it with an overlay\nthat describes how to translate JSON data points into RDF. For\nexample, the JSON key-value pair:\n\n``` javascript\n\"email\": \"jane@example.com\"\n```\n\nshould be translated as an RDF predicate `http://schema.org/email` and\nan RDF literal object `jane@example.com`. To do this, the overlay\nannotates the schema for the `email` property with the IRI\n`http://schema.org/email`. The overlay should also specify that it\nshould be an RDF predicate. The following annotation serves this\npurpose:\n\n``` javascript\n\"rdfPredicate\": \"http://schema.org/email\"\n```\n\nAs another example, consider the `Person` object:\n\n``` javascript\n{\n   \"@id\": \"http://linkedin.com/jane-doe\",\n   ...\n}\n```\n\nThis should be translated to an RDF IRI node\n`http://linkedin.com/jane-doe`, which is given in the `@id`\nattribute. So we should be able to tell that an RDF IRI node should be\ncreated from a given data element to represent this object:\n\n``` javascript\n\"rdfIRI\": \"ref:\u003cpointer to the @id field\u003e\n```\n\nThe following diagram summarizes the approach:\n\n![JSON to RDF](json-to-rdf.png)\n\n\nBased on this, we devise the following tags:\n\n  * `rdfPredicate`: Declares a JSON property as an RDF predicate\n    (edge), with a mapping to a term. This is similar to mapping a\n    JSON property using JSON-LD context. The difference here is that\n    using a layered schema, we can explicitly declare that a JSON\n    property should be translated as a predicate instead of a node.\n  * `rdfIRI`: Declares a JSON property as an RDF node, while also\n    providing its IRI mapping. The IRI can be a fixed value, or it can\n    be collected from another node in the input document (like, the\n    `@id` property in our example.)\n  * `rdfType`: Defines the type of the literal, or the type of the\n    node.\n  * `rdfLang`: Defined the language of a literal.\n  \n### The Overlay\n\nThe schema defines the object `Person` as follows:\n\n``` javascript\n{\n    \"definitions\": {\n        \"Person\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"name\": {\n                 \"type\": \"string\"\n              },\n             ...\n```\n\nThe overlay follows the same structure, but adds the tags under the\n`x-ls` object:\n\n``` javascript\n{\n    \"definitions\": {\n        \"Person\": {\n            \"x-ls\": {\n                \"rdfType\": \"http://schema.org/Person\",\n                \"rdfIRI\": \"ref:http://schema.org/Person/@id\"\n            },\n            \"properties\": {\n              \"@id\": {\n                 \"type\": \"string\"\n              },\n              \"name\": {\n                 \"x-ls\": {\n                   \"rdfPredicate\": \"http://schema.org/name\"\n                 }\n              }\n            ...\n```\n\nThe `@id` property does not exist in the original schema. The overlay\nadds it.\n\nAll annotations go under the `x-ls` JSON object. This is the\nrecommended way of adding extensions to a JSON schema: it starts with\n`x-`. `ls` standard for `layered schema`. The layered schema processor\nbuilds a composite schema by adding `x-ls` objects to corresponding\nplaces in the schema.\n\nThe schema variant is a composition of the two:\n\n``` javascript\n{\n    \"definitions\": {\n        \"Person\": {\n            \"type\": \"object\",\n            \"x-ls\": {\n                \"rdfType\": \"http://schema.org/Person\",\n                \"rdfIRI\": \"ref:http://schema.org/Person/@id\"\n            },\n            \"properties\": {\n              \"@id\": {\n                 \"type\": \"string\"\n              },\n              \"name\": {\n                 \"type\": \"string\",\n                 \"x-ls\": {\n                    \"rdfPredicate\": \"http://schema.org/name\"\n                  }\n              },\n            ...\n```\nHere's the complete [person schema](person.schema.json) and the [overlay](person.ovl.json).\n\nFor `rdfIRI`, `json2rdf` uses these conventions:\n\n   * The following uses the given value as the IRI node:\n\n``` javascript\n\"rdfIRI\": \"value\"\n```\n\nExample:\n\nInput:\n``` javascript\n\"PostalAddress\": {\n    \"x-ls\": {\n       \"rdfIRI\": \"http://schema.org/PostalAddress\"\n    }\n```\nOutput:\n\nThe RDF node corresponding to the \"PostalAddress\" property with IRI:\n\"http://schema.org/PostalAddress\"\n\n\n   * The following creates a blank node for the JSON property:\n    \n``` javascript\n\"rdfIRI\": \"blank\"\n```\n\nExample:\n\nInput:\n``` javascript\n\"PostalAddress\": {\n    \"x-ls\": {\n       \"rdfIRI\": \"blank\"\n    }\n```\nOutput:\n\nThe RDF node corresponding to the \"PostalAddress\" property will be a blank node.\n\n\n   * The following uses the referenced node value to create an IRI node. The\n   node value must be an IRI. The first node accessible from the\n   current node that has `schemaNodeId: \u003creference\u003e` value will be\n   used.\n\n``` javascript\n\"rdfIRI\": \"ref:\u003creference\u003e\"\n```\n\nExample:\n\nInput:\n\n``` javascript\n\"Person\": {\n    \"x-ls\": {\n        \"rdfIRI\": \"ref:http://schema.org/Person/@id\"\n    },\n```\n\nOutput:\n\nThe RDF node corresponding to the \"Person\" property will have the IRI\nextracted from the \"@id\" property (the LPG node with `schemaNodeId:\nhttp://schema.org/Person/@id`) under the \"Person\" object.\n\n   * The following uses the JSON property value to create an IRI\n   node. The JSON property value must be an IRI:\n\n``` javascript\n\"rdfIRI\": \".\"\n```\n\n\n\n \n### LSA Data Ingestion\n\nLSA already provides tools to ingest data and produce a labeled\nproperty graph, so we will use those. LSA ingests a data file based on\na schema variant and produces a labeled property graph. The following\nimage illustrates the data ingestion process.\n\n![Ingestion Pipeline](pipeline.png)\n\nThe JSON schema [person.schema.json](person.schema.json) defines the\nstructure of the JSON objects, in this example, `Person` and\n`PostalAddress`. The overlay [person.ovl.json](person.ovl.json)\nannotates this schema to define the mappings to `schema.org` terms\nusing the above tags. The bundle file\n[person.bundle.yaml](person.bundle.yaml) combines the schema and the\noverlay, and defines the schema variant. [The schema variant itself is\nan LPG](compiled-schema-variant.svg?raw=1).  This schema variant LPG contains a node for every JSON data\npoint (every object, array, and value.) Data ingestion process takes\nthe input data file [person-sample.json](person-sample.json) and\ninterprets it using the schema variant LPG, [creating a new LPG for the\ndata object](ingested-json-lpg.svg?raw=1). This LPG becomes a self-describing object that contains\nall input data values and corresponding schema annotations. We then\ntake this LPG, use the RDF annotations at each node, and produce the\nRDF output.\n\nThe LPG for the ingested data contains the schema annotations as well\nas the input data. We can process the annotations in each node to\ncreate the RDF output. This creates an RDF IRI node with value taken\nfrom the `@id` property under `Person`, and with type\n`http://schema.org/Person`. Note that the original schema does not\ncontain the `@id` property. That is added by the overlay. The output\nlooks like:\n\n![Person id and type](id-type.png)\n\n\nFor `Person/name`, we get:\n\n![Person name](name.png)\n\n\n### Algorithm\n\nThe algorithm to convert an ingested data LPG into RDF using these\nannotations is implemented in [graph2rdf.go](graph2rdf.go). The\nalgorithm sketch is as follows:\n\n\n  * We first build the top-level nodes. These are nodes that have\n    `rdfIRI` annotation. We keep a mapping between input LPG nodes and\n    the RDF nodes. This step builds a list of input graph nodes for\n    which RDF nodes are built.\n  * Using the list of graph nodes built in the previous step of the\n    previous iteration, we process all graph nodes that are connected\n    with `rdfPredicate`, and extend the RDF graph in a breadth-first\n    manner. We put every new input graph nodes for which a non-literal\n    RDF node is generated to the list of nodes, and iterate as long as\n    the list of nodes is nonempty.\n\n## Running\n\nYou can run the `json2rdf` program as follows:\n\n``` \n  json2rdf --bundle person.bundle.yaml --type http://schema.org/Person person-sample.json\n```\n\nAlternatively, you can first ingest the JSON file using LSA tools, and\nthen pass the output graph to json2rdf:\n\n``` \n  layers ingest json --bundle person.bundle.yaml --type http://schema.org/Person person-sample.json | json2rdf \n```\n\nWhich produces:\n\n``` \n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://www.w3.org/1999/02/22-rdf-syntax-ns#type\u003e \u003chttp://schema.org/Person\u003e .\n_:b0 \u003chttp://www.w3.org/1999/02/22-rdf-syntax-ns#type\u003e \u003chttps://schema.org/PostalAddress\u003e .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/email\u003e \"info@example.com\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/name\u003e \"Jane Doe\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/birthPlace\u003e \"Boulder, CO\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/birthDate\u003e \"1972-11-12\"^^\u003chttp://schema.org/Date\u003e .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/height\u003e \"71\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/gender\u003e \"female\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/address\u003e _:b0 .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/colleague\u003e \"http://www.example.com/John.html\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/colleague\u003e \"http://www.example.com/Jane.html\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/sameAs\u003e \"https://www.facebook.com/\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/sameAs\u003e \"https://www.linkedin.com/\" .\n\u003chttp://linkedin.com/jane-doe\u003e \u003chttp://schema.org/sameAs\u003e \"http://twitter.com/\" .\n_:b0 \u003chttp://schema.org/addressLocality\u003e \"Denver\" .\n_:b0 \u003chttp://schema.org/addressRegion\u003e \"CO\" .\n_:b0 \u003chttp://schema.org/postalCode\u003e \"80123\" .\n_:b0 \u003chttp://schema.org/streetAddress\u003e \"100 Main Street\" .\n```\n\nSo this is how you can generate RDF from a JSON document using a JSON\nschema describing the format of the input documents, a JSON schema\noverlay that describes the RDF mapping, and a bundle that combines the\nschema and the overlay.\n\nThe important takeaways are:\n\n  * The shape of the RDF output can be better controlled by extending\n    the tags and the translation algorithm. A JSON document can be\n    translated in multiple ways to produce different RDF outputs.\n  * The input to `json2rdf` is an LPG. LSA supports other schema\n    formats. It is possible to translate an XML document or a CSV file\n    to an RDF using the same framework.\n  * LSA supports multiple data types. It is possible to normalize and\n    translate non-standard date/time representations.\n  * Unlike a JSON-LD document, JSON schemas provide structural\n    validation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudprivacylabs%2Fjson2rdf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudprivacylabs%2Fjson2rdf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudprivacylabs%2Fjson2rdf/lists"}