{"id":13781046,"url":"https://github.com/RMLio/rmlmapper-java","last_synced_at":"2025-05-11T14:34:46.459Z","repository":{"id":34882331,"uuid":"138707242","full_name":"RMLio/rmlmapper-java","owner":"RMLio","description":"The RMLMapper executes RML rules to generate high quality Linked Data from multiple originally (semi-)structured data sources","archived":false,"fork":false,"pushed_at":"2025-04-25T11:21:20.000Z","size":89398,"stargazers_count":167,"open_issues_count":48,"forks_count":63,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-05-01T12:04:53.625Z","etag":null,"topics":["knowledge-graph","mapping-languages","rml","rml-mapping"],"latest_commit_sha":null,"homepage":"http://rml.io","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RMLio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-26T08:22:17.000Z","updated_at":"2025-04-07T11:37:35.000Z","dependencies_parsed_at":"2023-01-15T10:00:51.484Z","dependency_job_id":"58568553-31a9-4efd-a6d2-873532c95417","html_url":"https://github.com/RMLio/rmlmapper-java","commit_stats":{"total_commits":971,"total_committers":28,"mean_commits":34.67857142857143,"dds":0.7950566426364573,"last_synced_commit":"15c9c02c78aa743df02f7e773c2b1a229c3bc377"},"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2Frmlmapper-java","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2Frmlmapper-java/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2Frmlmapper-java/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2Frmlmapper-java/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RMLio","download_url":"https://codeload.github.com/RMLio/rmlmapper-java/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252954380,"owners_count":21830902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knowledge-graph","mapping-languages","rml","rml-mapping"],"created_at":"2024-08-03T18:01:22.431Z","updated_at":"2025-05-11T14:34:41.445Z","avatar_url":"https://github.com/RMLio.png","language":"Java","funding_links":[],"categories":["KGC Materializers","Mapping","Open Data and Statistics"],"sub_categories":["Groups"],"readme":"# RMLMapper \u003c!-- omit in toc --\u003e\n\n[![Maven Central](https://img.shields.io/maven-central/v/be.ugent.rml/rmlmapper.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22be.ugent.rml%22%20AND%20a:%22rmlmapper%22)\n\nThe RMLMapper executes RML rules to generate Linked Data.\nIt is a Java library, which is available via the command line ([API docs online](https://javadoc.io/doc/be.ugent.rml/rmlmapper)).\nThe RMLMapper loads all data in memory, so be aware when working with big datasets.\n\nWant to get started quickly? Check out [Releases](#releases) on where to find the latest CLI build as a jar,\nand see [Usage](#cli) on how to use the commandline interface!\n\n## Table of contents \u003c!-- omit in toc --\u003e\n\n- [Features](#features)\n  - [Supported](#supported)\n  - [Future](#future)\n- [Releases](#releases)\n- [Build](#build)\n- [Usage](#usage)\n  - [CLI](#cli)\n  - [Library](#library)\n  - [Docker](#docker)\n  - [Including functions](#including-functions)\n  - [Generating metadata](#generating-metadata)\n- [Testing](#testing)\n  - [RDBs](#rdbs)\n- [Dependencies](#dependencies)\n- [Commercial Support](#commercial-support)\n- [Remarks](#remarks)\n  - [Typed spreadsheet files](#typed-spreadsheet-files)\n  - [XML file parsing performance](#xml-file-parsing-performance)\n  - [Language tag support](#language-tag-support)\n  - [Duplicate removal and serialization format](#duplicate-removal-and-serialization-format)\n  - [I have a question! Where can I get help?](#i-have-a-question-where-can-i-get-help)\n- [Documentation](#documentation)\n  - [UML Diagrams](#uml-diagrams)\n\n## Features\n\n### Supported\n- local data sources:\n  - Excel (.xlsx)\n  - LibreOffice (.ods)\n  - CSV files (including CSVW)\n  - JSON files (JSONPath)\n  - XML files (XPath)\n- remote data sources:\n  - relational databases (MySQL, PostgreSQL, Oracle, and SQLServer)\n  - Web APIs with W3C Web of Things\n  - SPARQL endpoints\n  - files via HTTP urls (via GET)\n    - CSV files\n    - JSON files (JSONPath (`@` can be used to select the current object.))\n    - XML files (XPath)\n- functions (most cases)\n  - For examples on how to use functions within RML mapping documents, you can have a look at the [RML+FnO test cases](https://github.com/RMLio/rml-fno-test-cases)\n- configuration file\n- metadata generation\n- output formats: nquads (default), turtle, trig, trix, jsonld, hdt\n- join conditions\n- targets:\n  - local file\n  - VoID dataset\n  - SPARQL endpoint with SPARQL UPDATE\n\n### Future\n- functions (all cases)\n- conditions (all cases)\n- data sources:\n  - NoSQL databases\n  - TPF servers\n\n## Releases\n\nThe standalone jar file (that has a [commandline interface](#cli)) for every release can be found on the release's page on GitHub.\nYou can find the latest release [here](https://github.com/RMLio/rmlmapper-java/releases/latest).\nThis is the recommended way to get started with RMLMapper.\nDo you want to build from source yourself? Check [Build](#build).\n\n## Build\nThe RMLMapper is built using Maven.\nAs it is also tested against Oracle (check [here](#accessing-oracle-database) for details),\nit needs a specific set-up to run all tests.\nThat's why we recommend to build without testing: `mvn install -DskipTests=true`.\nIf you want, you can install with tests, and just skip the Oracle tests: `mvn test -Dtest=!Mapper_OracleDB_Test`.\n\nA standalone jar can be found in `/target`.\n\nTwo jars are found in `/target`: a slim jar without bundled dependencies, and a standalone jar (suffixed with `-all.jar`) with all dependencies bundled.\n\nBuilding with profile `no-buildnumber` disables using and updating `buildNumber.properties` (and uses `0` as build number), e.g.:\n```\nmvn clean package -P no-buildnumber\n```\noutputs for example `target/rmlmapper-\u003cversion\u003e-r0.jar`\n\n## Usage\n\n### CLI\nThe following options are most common.\n\n- `-m, --mapping \u003carg\u003e`: one or more mapping file paths and/or strings (multiple values are concatenated).\n- `-o, --output \u003carg\u003e`: path to output file\n- `-s,--serialization \u003carg\u003e`: serialization format (nquads (default), trig, trix, jsonld, hdt)\n\nAll options can be found when executing `java -jar rmlmapper.jar --help`,\nthat output is found below.\n\n```\nusage: java -jar mapper.jar \u003coptions\u003e\noptions:\n -b,--base-iri \u003carg\u003e                 Base IRI used to expand relative IRIs\n                                     in generated terms in the output.\n -c,--configfile \u003carg\u003e               path to configuration file\n -d,--duplicates                     remove duplicates in the HDT,\n                                     N-Triples, or N-Quads output\n    --disable-automatic-eof-marker   Setting this option assumes input\n                                     data has a kind of End-of-File\n                                     marker. Don't use unless you're\n                                     absolutely sure what you're doing!\n -dsn,--r2rml-jdbcDSN \u003carg\u003e          DSN of the database when using R2RML\n                                     rules\n -e,--metadatafile \u003carg\u003e             path to output metadata file\n -f,--functionfile \u003carg\u003e             one or more function file paths\n                                     (dynamic functions with relative\n                                     paths are found relative to the cwd)\n -h,--help                           show help info\n -l,--metadataDetailLevel \u003carg\u003e      generate metadata on given detail\n                                     level (dataset - triple - term)\n -m,--mappingfile \u003carg\u003e              one or more mapping file paths and/or\n                                     strings (multiple values are\n                                     concatenated). r2rml is converted to\n                                     rml if needed using the r2rml\n                                     arguments.RDF Format is determined\n                                     based on extension.\n -o,--outputfile \u003carg\u003e               path to output file (default: stdout)\n -p,--r2rml-password \u003carg\u003e           password of the database when using\n                                     R2RML rules\n -psd,--privatesecuritydata \u003carg\u003e    one or more private security files\n                                     containing all private security\n                                     information such as usernames,\n                                     passwords, certificates, etc.\n -s,--serialization \u003carg\u003e            serialization format (nquads\n                                     (default), turtle, trig, trix,\n                                     jsonld, hdt)\n    --strict                         Enable strict mode. In strict mode,\n                                     the mapper will fail on invalid IRIs\n                                     instead of skipping them.\n -t,--triplesmaps \u003carg\u003e              IRIs of the triplesmaps that should\n                                     be executed in order, split by ','\n                                     (default is all triplesmaps)\n -u,--r2rml-username \u003carg\u003e           username of the database when using\n                                     R2RML rules\n -v,--verbose                        show more details in debugging output                                                      \n```\n\n#### Accessing Web APIs with authentication\n\nThe [W3C Web of Things Security Ontology](https://www.w3.org/2019/wot/security)\nis used to describe how Web APIs authentication should be performed \nbut does not include the necessary credentials to access the Web API.\nThese credentials can be supplied using the `-psd \u003cPATH\u003e` CLI argument.\nThe `PATH` argument must point to one or more private security files\nwhich contain the necessary credentials to access the Web API.\n\nAn example can be found in the test cases \n[src/test/resources/web-of-things](src/test/resources/web-of-things).\n\n### Library\n\nAn example of how you can use the RMLMapper as an external library can be found\nat [./src/test/java/be/ugent/rml/readme/ReadmeTest.java](https://github.com/RMLio/rmlmapper-java/blob/master/src/test/java/be/ugent/rml/readme/ReadmeTest.java)\n\n### Docker\n\n#### Dockerhub\n\nWe publish our Docker images automatically on Dockerhub for every release.\nYou can find our images here: [rmlio/rmlmapper-java](https://hub.docker.com/r/rmlio/rmlmapper-java).\n\n#### Build image\n\nYou can use Docker to run the RMLMapper by following these steps:\n\n- Build the Docker image: `docker build -t rmlmapper .`.\n- Run a Docker container: `docker run --rm -v $(pwd):/data rmlmapper -m mapping.ttl`.\n\nThe same parameters are available as via the CLI.\nThe RMLMapper is executed in the `/data` folder in the Docker container.\n\n### Including functions\n\nThere are three ways to include (new) functions within the RML Mapper\n  * dynamic loading: you add links to java files or jar files, and those files are loaded dynamically at runtime\n  * preloading: you register functionality via code, and you need to rebuild the mapper to use that functionality\n  * add as dependency\n\nRegistration of functions is done using a Turtle file, which you can find in `src/main/resources/functions.ttl`\n\n\n\n#### Dynamic loading\n\nCreate a Turtle file that describe the functions that need to be included and add the jar which contains those functions.\n\n\u003e Note: the java or jar-files are found relative to the cwd.\nYou can change the functions.ttl path (or use multiple functions.ttl paths) using a commandline-option (`-f`).\n\nFor example the snippets below dynamically link an fno:Function to a library, provided by a jar-file (`CustomFunctions.jar`). The example links a function that parses the latitude (`50.2`) out of the following string `\"POINT (50.2 5.3)\"`.\n\n `functions.ttl` contains the description of the function in Turtle:\n```turtle\n@prefix dcterms: \u003chttp://purl.org/dc/terms/\u003e .\n@prefix doap:    \u003chttp://usefulinc.com/ns/doap#\u003e .\n@prefix fno:     \u003chttps://w3id.org/function/ontology#\u003e .\n@prefix fnoi:    \u003chttps://w3id.org/function/vocabulary/implementation#\u003e .\n@prefix fnom:    \u003chttps://w3id.org/function/vocabulary/mapping#\u003e .\n@prefix grel:    \u003chttp://users.ugent.be/~bjdmeest/function/grel.ttl#\u003e .\n@prefix grelm:   \u003chttp://fno.io/grel/rmlmapping#\u003e .\n@prefix rdfs:    \u003chttp://www.w3.org/2000/01/rdf-schema#\u003e .\n\ngrel:parsePointLat a fno:Function ;\n  fno:name \"parsePointLat\" ;\n  rdfs:label \"parsePointLat\" ;\n  dcterms:description \"Parse the latitude from a point.\" ;\n  fno:expects ( grel:valueParam ) ;\n  fno:returns ( grel:stringOut ) .\n\ngrelm:javaString\n    a                  fnoi:JavaClass ;\n    doap:download-page \"CustomFunctions.jar\" ;\n    fnoi:class-name    \"CustomFunctions\" .\n\ngrelm:parsePointLat\n    a                    fno:Mapping ;\n    fno:function         grel:parsePointLat ;\n    fno:implementation   grelm:javaString ;\n    fno:methodMapping    [ a                fnom:Function ;\n                           fnom:method-name \"parsePointLat\" ] .\n```\nThe accompanying java file `CustomFunctions.java`:\n```java\npublic class CustomFunctions {\n  public static String parsePointLat(String s) {\n    return s.replace(\"POINT \", \"\").replace('(', ' ').replace(')', ' ').trim().split(\"\\\\s+\")[0];\n  }\n}\n```\nTo dynamically include the custom function, compile the java-file and include `functions.ttl` with the `-f` option:\n```bash\njavac CustomFunctions.java \u0026\u0026 jar cvf CustomFunctions.jar CustomFunctions.class\njava -jar mapper.jar -f functions.ttl \u003cother options\u003e\n```\n\n#### Preloading\n\nThis overrides the dynamic loading.\nAn example of how you can use Preload a custom function can be found\nat [./src/test/java/be/ugent/rml/readme/ReadmeFunctionTest.java](https://github.com/RMLio/rmlmapper-java/blob/master/src/test/java/be/ugent/rml/readme/ReadmeFunctionTest.java)\n\n#### Adding as dependency\n\nThis is most interesting if you use RMLMapper as a library in your own project.\nJust add the dependency to the function library you want to use in your project.\n\nYou can also add a function library as a Maven dependency in `pom.xml` of RMLMapper.\nYou'll have to rebuild RMLMapper to use it.\n\n### Generating metadata\n\nConform to how it is described in the scientific paper [1],\nthe RMLMapper allows to automatically generate [PROV-O](https://www.w3.org/TR/prov-o/) metadata.\nSpecifically, you need the CLI arguments below.\nYou can specify in which output file the metadata should be stored,\nand up to which level metadata should be stored (dataset, triple, or term level metadata).\n\n```\n -e,--metadatafile \u003carg\u003e          path to output metadata file\n -l,--metadataDetailLevel \u003carg\u003e   generate metadata on given detail level\n                                  (dataset - triple - term)\n```\n\n## Testing\n\n### Command line\nRun the tests via `test.sh`. \n\n### IntelliJ\nRight-click `src/test/java` directory and select \"Run 'All tests'\". \n\n#### Derived tests\nSome tests (Excel, ODS) are derived from other tests (CSV) using a script (`./generate_spreadsheet_test_cases.sh`)\n\n### RDBs\nMake sure you have [Docker](https://www.docker.com) running. On Unix, others read-write permission (006) is required on `/var/run/docker.sock` in order to run the tests.\nThe tests will fail otherwise, as Testcontainers can't spin up the container. \n\n#### Problems\n* A problem with Docker (can't start the container) causes the SQLServer tests to fail locally. These tests will always succeed locally.\n* A problem with Docker (can't start the container) causes the PostgreSQL tests to fail locally on Windows 7 machines.\n\n## Dependencies\n\n|                   Dependency                   | License                                                            |\n|:----------------------------------------------:|--------------------------------------------------------------------|\n|         ch.qos.logback logback-classic         | Eclipse Public License 1.0 \u0026 GNU Lesser General Public License 2.1 |\n|      com.github.fnoio function-agent-java      | MIT                                                                |\n|      com.github.fnoio grel-functions-java      | MIT                                                                |\n|     com.github.fnoio idlab-functions-java      | MIT                                                                |\n|           com.github.rdfhdt hdt-java           | GNU Lesser General Public License v3.0                             |\n|      com.github.tomakehurst:wiremock-jre8      | Apache License 2.0                                                 |\n|       com.microsoft.sqlserver mssql-jdbc       | MIT                                                                |\n|         com.mysql mysql-connector-java         | GNU General Public License v2.0                                    |\n|        com.oracle.database.jdbc:ojdbc11        | Oracle Free Use Terms and Conditions                               |\n|             net.minidev json-smart             | Apache License 2.0                                                 |\n|          org.apache.jena fuseki-main           | Apache License 2.0                                                 |\n|         org.eclipse.rdf4j rdf4j-client         | Eclipse Distribution License v1.0                                  |\n|      org.junit.jupiter junit-jupiter-api       | Eclipse Public License v2.0                                        |\n|     org.junit.jupiter junit-jupiter-engine     | Eclipse Public License v2.0                                        |\n|     org.junit.jupiter junit-jupiter-params     | Eclipse Public License v2.0                                        |\n|     org.junit.vintage junit-vintage-engine     | Eclipse Public License v2.0                                        |\n|           org.postgresql postgresql            | BSD                                                                |\n|            org.testcontainers jdbc             | MIT                                                                |\n|        org.testcontainers junit-jupiter        | MIT                                                                |\n|         org.testcontainers mssqlserver         | MIT                                                                |\n|            org.testcontainers mysql            | MIT                                                                |\n|          org.testcontainers oracle-xe          | MIT                                                                |\n|         org.testcontainers postgresql          | MIT                                                                |\n\n## Commercial Support\n\nDo you need...\n\n-   training?\n-   specific features?\n-   different integrations?\n-   bugfixes, on _your_ timeline?\n-   custom code, built by experts?\n-   commercial support and licensing?\n\nYou're welcome to [contact us](mailto:info@rml.io) regarding\non-premise, enterprise, and internal installations, integrations, and deployments.\n\nWe have commercial support available.\n\nWe also offer consulting for all-things-RML.\n\n## Remarks\n\n### Typed spreadsheet files\nAll spreadsheet files are as of yet regarded as plain CSV files. No type information like Currency, Date... is used.\n\n### XML file parsing performance\n\nThe RMLMapper's XML parsing implementation (`javax.xml.parsers`) has been chosen to support full XPath.\nThis implementation causes a large memory consumption (up to ten times larger than the original XML file size).\nHowever, the RMLMapper can be easily adapted to use a different XML parsing implementation that might be better suited for a specific use case.\n\n### Language tag support\n\nThe processor checks whether correct language tags are not, using a regular expression.\nThe regex has no support for languages of length 5-8, but this currently only applies to 'qaa..qtz'.\n\n### Duplicate removal and serialization format\n\nPerformance depends on the serialization format (`--serialization \u003cformat\u003e`)\nand if duplicate removal is enabled (`--duplicates`).\nExperimenting with various configurations may lead to better performance for \nyour use case.\n\n### I have a question! Where can I get help?\n\nDo you have any question related to writing RML mapping rules, \nthe RML specification, etc., feel free to ask them \nhere: https://github.com/kg-construct/rml-questions !\nIf you have found a bug or need a feature for the RMLMapper itself, \nyou can make an issue in this repository.\n\n## Documentation\nGenerate static files at /docs/apidocs with:\n```\nmvn javadoc:javadoc\n```\n\n### UML Diagrams\n\n#### Architecture UML Diagram\n##### How to generate with IntelliJ IDEA\n(Requires Ultimate edition)\n\n* Right click on package: \"be.ugent.rml\"\n* Diagrams \u003e Show Diagram \u003e Java Class Diagrams\n* Choose what properties of the classes you want to show in the upper left corner\n* Export to file \u003e .png  | Save diagram \u003e .uml\n\n#### Sequence Diagram\n##### Edit on [draw.io](https://www.draw.io)\n* Go to [draw.io](https://www.draw.io)\n* Click on 'Open Existing Diagram' and choose the .html file\n\n[1]: A. Dimou, T. De Nies, R. Verborgh, E. Mannens, P. Mechant, and R. Van de Walle, “Automated metadata generation for linked data generation and publishing workflows,” in Proceedings of the 9th Workshop on Linked Data on the Web, Montreal, Canada, 2016, pp. 1–10.\n[PDF](http://events.linkeddata.org/ldow2016/papers/LDOW2016_paper_04.pdf)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRMLio%2Frmlmapper-java","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRMLio%2Frmlmapper-java","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRMLio%2Frmlmapper-java/lists"}