{"id":25205179,"url":"https://github.com/sparna-git/ngsi-ld-converters","last_synced_at":"2025-04-05T01:10:11.199Z","repository":{"id":75663822,"uuid":"585638816","full_name":"sparna-git/ngsi-ld-converters","owner":"sparna-git","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-30T08:55:25.000Z","size":143,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-10T09:19:14.850Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sparna-git.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-01-05T17:26:40.000Z","updated_at":"2023-10-13T10:04:02.000Z","dependencies_parsed_at":"2023-10-10T17:02:47.869Z","dependency_job_id":"d1a79512-8f6e-47aa-88df-efec97ad2f1e","html_url":"https://github.com/sparna-git/ngsi-ld-converters","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparna-git%2Fngsi-ld-converters","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparna-git%2Fngsi-ld-converters/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparna-git%2Fngsi-ld-converters/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sparna-git%2Fngsi-ld-converters/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sparna-git","download_url":"https://codeload.github.com/sparna-git/ngsi-ld-converters/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247271528,"owners_count":20911587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-10T09:19:16.950Z","updated_at":"2025-04-05T01:10:11.181Z","avatar_url":"https://github.com/sparna-git.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NGSI-LD Semantic Converter\n\n## Installing the Python script\n\n1. Clone this repository\n\n```\ngit clone git@github.com:sparna-git/ngsi-ld-converters.git\ncd ngsi-ld-converters\n```\n\n2. Install pip\n\n```\nsudo apt install python3-pip\n```\n\nOn wWindows, PIP is already included for versions of Python \u003e 3.4.\n\n3. Install virtualenv\n\n```\npip install virtualenv\n# You may need to do this on Linux :\n# sudo apt install python3.10-venv\n```\n\n4. Create virtualenv\n\n```\npython3.10 -m venv virtualenv\n```\n\n5. Activate virtualenv\n\n```\nWindows : virtualenv/Scripts/activate.bat\nLinux : source virtualenv/bin/activate\n```\n\n6. Once in the virtual env, installer the necessary dependencies from `requirements.txt` :\n\n```\npip install -r requirements.txt\n```\n\n## Running the Python script\n\n1. See the available options by running the script with `--help` flag\n\n```\npython3.10 convert_shacl_json.py --help\n```\n\nYou will see the usage message.\n\n2. Run by passing as input the data file to convert, the SHACL rules to apply, and the JSON-LD framing spec to apply, and where to write the final output JSON file. Use the rules and JSON-LD framing files provided in the \"rules\" folder :\n\n```\npython3.10 convert_shacl_json.py --rules ../rules/datacube-2-statdcatap.ttl  --frame ../rules/framing-context.jsonld --data ../examples/structure-tourism-minimal/structures-tourism-minimal-pretty.ttl --output output.json\n```\n\n\n## Approach\n\nThe approach taken to convert input RDF data to NGSI-LD is the following :\n\n1. Map source input data to target data structure at the semantic (= RDF triple) level\n   - this can be done in a number of ways, the proposed approach is to describe a serie of transformation rules using SHACL rules : https://www.w3.org/TR/shacl-af/#SPARQLRule\n   - this way each rule can be precisely identified, described, maintained, shared, etc. following linked data principles\n   - rules are executed using a SHACL processor (I have develop https://shacl-play.sparna.fr/play/home?lang=es which wraps TopBraid SHACL library : https://github.com/TopQuadrant/shacl)\n   - mapping rules deal exclusively with the semantic mapping from one model to another, they don't care about serialization. They could be reused in other technical/implementation contexts.\n   - The approach is only applicable to input RDF data.\n   - Sample rules to give you an idea are at https://github.com/sparna-git/ngsi-ld-converters/blob/main/rules/datacube-2-statdcatap/datacube-2-statdcatap.ttl\n2. Then serialize the output RDF in JSON-LD (any RDF library can do that)\n3. Then use JSON-LD Framing (https://www.w3.org/TR/json-ld11-framing/) to specify how the output triples should be serialized in a clean JSON-LD (use the JSON-LD contexts to map classes/properties to JSON keyx, specify which entity should come first, etc.)\n   - sample JSON-LD framing specification is at https://github.com/sparna-git/ngsi-ld-converters/blob/main/contexts/framing-context.jsonld\n4. Then, if needed, split the JSON-LD in multiple files\n   - splitting could be done earlier in the process, I don't know\n\n## Documentation pointers\n\nExisting FIWARE converter documentation : \n- https://github.com/flopezag/IoTAgent-Turtle\n- https://github.com/INTERSTAT/Statistics-Contextualized/tree/main/pilots/Deliverable%203.2#sdmx-to-etsi-ngsi-ld\n\nSmart data models :\n- https://www.fiware.org/smart-data-models/ et https://smartdatamodels.org/\n- STAT-DCAT-AP data model : https://github.com/smart-data-models/dataModel.STAT-DCAT-AP\n\nDCAT problematics pointers :\n- https://hackmd.io/@WeYQtPurSw-OcPBBOdwKLQ/HkB1jchXs\n\n## Questions to Fernando and his answers\n\n\u003e In existing converter, why are some properties reified (dct:description), and some not (dct:title, dct:identifier) ?\n\nThe problem is that some properties are defined in different contexts. There is a solution to correct it if inside \nthe Smart Data Models Subject, we define unambiguously the property to be used in the scope of this subject. Then, \nwe could dismiss the use of prefix in the JSON-LD. \n\nIt also means that the IoTAgent-Turtle need to make an update to align with this approach. We have started to modify \nthe Data Models to reflect it and afterwards we will plan the steps to modify the parser with the new version of the \ndata models.\n\n\n\n\u003e In existing converter, why is URI mapped to dct:title and why is rdfs:label mapped to dct:description ?\n\nFirst, it seems that something is missing in the question regarding the dct:title.\nWe are mapping the SDMX into DCAT-AP/statDCAT-AP, this is the reason why we map rdfs:label to dct:description.\n\n\n\n\u003e Existing converter generates separate files in the output. Is there a reason for this ? could the output be in a\nsingle file ?\n\nETSI NGSI-LD API defines the world based on entities. NGSI-LD Entity is the informational representative of something \nthat is supposed to exist in the real world, physically or conceptually. Each of the data models are making reference \nto a concrete Entity and they are sent to the FIWARE Context Brokers using different Entities that in the end are \ndifferent files.\n\n\n\n\u003e The STAT-DCAT-AP json-ld context at https://smart-data-models.github.io/dataModel.STAT-DCAT-AP/context.jsonld maps \nkeys to wrong URIs. Is it intended ? This is a blocking issue as it forces us to use these URIs (and not the typical \ndct ones) so that we can then regenerate a JSON-LD valid according to the spec.\n\nIn the context.jsonld there are automatically generated IRI based on the smart data models domain. These IRIs are \nprovided for all terms and lead to a web page with basic information about the term, definition, data type, and data \nmodels where the term is also used. Now in some subjects, there is a complementary file ‘notes_context.jsonld’ with \nthe original IRI whenever they are available. In order to create the proper context (with the original IRI) there is a \nservice at the front page of smartdatamodels.org home -\u003e tools -\u003e map @context with external ontologies that whenever \nit detects a notes_context.jsonld file, make the change from the original terms producing a original @context.\n\n\n\n\u003e I have the feeling that the JSON-LD context of StatDCAT-AP is incomplete; it does not contain the classes \n documented at https://github.com/INTERSTAT/Statistics-Contextualized/tree/main/pilots/Deliverable%203.2#dcat-ap-data-models\n \n\u003e We can read at https://github.com/INTERSTAT/Statistics-Contextualized/tree/main/pilots/Deliverable%203.2#statdcat-ap-data-models \n  that stat:numSeries or dqv:qualityAnnotation are not created in StatDCAT-AP Smart Data Model. Why ? what is the reason\n  for not declaring a complete Stat DCAt-AP Smart Data Model ?\n  \nNot all StatDCAT-AP is used in the project and only those classes used in the project have been mapped. Keep in mind \nthat the Smart Data Models program is based on real cases and therefore, we define the data models to cover those real \ncases. For those classes, It should have all the first-level terms in the context.jsonld, because it is generated \nautomatically, but errors in the scripts are always possible if you could be more precise we could fix it.  The other \nStatDACT-AP classes are not used because we are adopting the DCAT-AP analogue classes.\n\n\n\n\u003e Same strange thing with dataModel.DCAT-AP at https://raw.githubusercontent.com/smart-data-models/dataModel.DCAT-AP/master/context.jsonld, \n  which uses its own namespace, and fails to declare the types, e.g. `CatalogueDCAT-AP` used by the converter to convert\n  Catalogs\n\nCan you elaborate a little more the question. It is not clear for us.\n\n\n\n\u003e It seems there is an NGSI-V2 with key-values syntax for which it is not necessary to decompose the properties : \nhttps://github.com/smart-data-models/dataModel.STAT-DCAT-AP/blob/master/Dataset/doc/spec.md#datasetstat-dcat-ap-ngsi-v2-key-values-example is this interesting ?\n\nSmart Data Models program is agnostic to the data model representation, and its data models look compatible with \nseveral formats, including NGSIv2, NGSI-LD, but also SQL, DTDL, etc. Then, these examples in the spec are useful for \nNGSIv2 users.\n\n\n\n\u003e The prefix stat in StatDCAT-AP is not yet assigned : http://data.europa.eu/(xyz)/statdcat-ap/ \n\nCorrect, at the beginning of the implementation. We cannot find the URI in the specification. Nevertheless, during the \nmodification of the data models to answer the first question, we discover that joinup has published the URI for the \nstat prefix (http://data.europa.eu/s1n). See https://joinup.ec.europa.eu/collection/semantic-interoperability-community-semic/solution/statdcat-application-profile-data-portals-europe/discussion/namespace-uri-still-be-finalised \ncomment about it.\n\n\n\n\u003e Is there any constraint on the URIs to generate ? the current generators construct URIs like \n`urn:ngsi-ld:Measure:m1000` (using the local part of existing URI and crafting a new URI with urn:ngsi-ld prefix)\n\nOnly the limitation by the ETSI NGSI-LD standard (https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.06.01_60/gs_cim009v010601p.pdf, \nAnnex A3). NGSI-LD defines a specific URN namespace. As it is based on URNs, the usage of this identification approach \nis not recommended when dereferenceable URIs are needed (fully-fledged linked data scenarios).\n\nThe referred namespace is defined as follows (to be registered with IANA):\n• Namespace identifier: NID = \"ngsi-ld\"\n• Namespace specific string: NSS = EntityTypeName \":\" EntityIdentificationString\n\nEntityTypeName shall be an Entity Type Name which can be expanded to a URI as per the @context.\nEntityIdentificationString shall be a string that allows uniquely identifying the subject Entity in combination with \nthe other items being part of the NSS.\n\nEXAMPLE: urn:ngsi-ld:Person:28976543.\n\nIt is recommended that applications use this URN namespace when applicable. \n\n\n\n\u003e Should we create any dcat:Distribution ?\n\nAt the moment, taking into account the current real cases, we did not need to use dcat:Distribution. If in the incoming \nmonths, we see a special scenario in which it is needed the definition of this class we work on its definition.\n\n\n\n\u003e Does generating the Catalogue automatically has any interest ?\n\nSDMX DataStructureDefinition is mapped to the Dataset in DCAT-AP and the SDMX Dataset is mapped to the Catalogue DCAT-AP.\n\n\n\n\u003e What is the scope exactly of what we need to transform ? the current converter converts properties, concepts, concept\nschemes, etc. but it looks like simple copy of the data. How should we deal with this ?\n\nThe scope of the IoTAgent-Turtle is to parse terse RDF TTL file format into the JSON-LD file format in order to apply \nETSI NGSI-LD API and send the metadata and data to the FIWARE Context Brokers. It will demonstrate the integration of \nstatistical scenarios into the FIWARE Architecture, facilitating the integration with third-parties services that can \nconsume NGSI-LD API data.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparna-git%2Fngsi-ld-converters","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsparna-git%2Fngsi-ld-converters","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparna-git%2Fngsi-ld-converters/lists"}