{"id":13781067,"url":"https://github.com/RMLio/RMLStreamer","last_synced_at":"2025-05-11T14:34:44.248Z","repository":{"id":34882330,"uuid":"176524785","full_name":"RMLio/RMLStreamer","owner":"RMLio","description":"The RMLStreamer executes RML rules to generate high quality Linked Data from multiple originally (semi-)structured data sources in a streaming way.","archived":false,"fork":false,"pushed_at":"2024-02-21T00:20:49.000Z","size":5332,"stargazers_count":50,"open_issues_count":30,"forks_count":18,"subscribers_count":7,"default_branch":"development","last_synced_at":"2025-05-01T12:04:55.081Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://rml.io/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RMLio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-19T14:03:18.000Z","updated_at":"2025-04-28T09:19:41.000Z","dependencies_parsed_at":"2023-01-15T10:00:51.302Z","dependency_job_id":"25bf000b-f98e-4025-8910-1fbcc6e0a036","html_url":"https://github.com/RMLio/RMLStreamer","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2FRMLStreamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2FRMLStreamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2FRMLStreamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RMLio%2FRMLStreamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RMLio","download_url":"https://codeload.github.com/RMLio/RMLStreamer/tar.gz/refs/heads/development","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252954378,"owners_count":21830902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T18:01:22.607Z","updated_at":"2025-05-11T14:34:41.205Z","avatar_url":"https://github.com/RMLio.png","language":"Scala","funding_links":[],"categories":["KGC Materializers"],"sub_categories":[],"readme":"## RMLStreamer\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3887065.svg)](https://doi.org/10.5281/zenodo.3887065)\n\nRMLStreamer generates [RDF](https://www.w3.org/2001/sw/wiki/RDF) from files or data streams\nusing [RML](http://rml.io/). The difference with other RML implementations is that it can handle\n*big* input files and *continuous data streams*, like sensor data.\n\nDocumentation regarding the use of (custom) functions can be found [here](documentation/README_Functions.md).\n\n### Quick start (standalone)\n\n* Download `RMLStreamer-\u003cversion\u003e-standalone.jar` from the [latest release](https://github.com/RMLio/RMLStreamer/releases/latest).\n* Run it as\n```\n$ java -jar RMLStreamer-\u003cversion\u003e-standalone.jar \u003ccommands and options\u003e\n```\n\nSee [Basic commands](#basic-commands) (where you replace `$FLINK_BIN run \u003cpath to RMLStreamer jar\u003e` with `java -jar RMLStreamer-\u003cversion\u003e-standalone.jar`)\nand [Complete RMLStreamer usage](#complete-rmlstreamer-usage) for\nexamples, possible commands and options.\n\n### Quick start (Docker - the fast way to test)\n\nThis runs the stand-alone version of RMLStreamer in a Docker container.\nThis is a good way to quickly test things or run RMLStreamer on a single machine, \nbut you don't have the features of a Flink cluster set-up (distributed, failover, checkpointing). \nIf you need those features, see [docker/README.md](docker/README.md). \n   \n#### Example usage:\n\n```\n$ docker run -v $PWD:/data --rm rmlio/rmlstreamer toFile -m /data/mapping.ttl -o /data/output\n```\n\n#### Build your own image:\n\nThis option builds RMLStreamer from source and puts that build into a Docker container ready to run.\nThe main purpose is to have a one-time job image.\n\n```\n$ ./buildDocker.sh\n```\n\nIf the build succeeds, you can invoke it as follows.\nIf you go to the directory where your data and mappings are,\nyou can run something like (change tag to appropriate version):\n\n```\n$ docker run -v $PWD:/data --rm rmlstreamer:v2.5.1-SNAPSHOT toFile -m /data/mapping.ttl -o /data/output.ttl \n```\n\nThere are more options for the script, if you want to use specific tags or push to Docker Hub:\n```\n$ ./buildDocker.sh -h\n\nBuild and push Docker images for RMLStreamer\n\nbuildDocker.sh [-h]\nbuildDocker.sh [-a][-n][-p][-u \u003cusername\u003e][-v \u003cversion\u003e]\noptions:\n-a   Build for platforms linux/arm64 and linux/amd64. Default: perform a standard 'docker build'\n-h   Print this help and exit.\n-n   Do NOT (re)build RMLStreamer before building the Docker image. This is risky because the Docker build needs a stand-alone version of RMLStreamer.\n-u \u003cusername\u003e  Add an username name to the tag name as on Docker Hub, like \u003cusername\u003e/rmlstreamer:\u003cversion\u003e.\n-p   Push to Docker Hub repo. You must be logged in for this to succeed.\n-v \u003cversion\u003e       Override the version in the tag name, like \u003cusername\u003e/rmlstreamer:\u003cversion\u003e. If not given, use the current version found in pom.xml.\n```\n\n### Moderately quick start (Docker - the recommended way)\n\nIf you want to get RMLStreamer up and running within 5 minutes using Docker, check out [docker/README.md](docker/README.md)\n\n### Not so quick start (deploying on a cluster)\n\nIf you want to deploy it yourself, read on.\n\nIf you want to develop, read [these instructions](documentation/README_DEVELOPMENT.md).\n\n### Installing Flink\nRMLStreamer runs its jobs on Flink clusters.\nMore information on how to install Flink and getting started can be found [here](https://ci.apache.org/projects/flink/flink-docs-release-1.14/try-flink/local_installation.html).\nAt least a local cluster must be running in order to start executing RML Mappings with RMLStreamer.\nPlease note that this version works with Flink 1.14.5 with Scala 2.11 support, which can be downloaded [here](https://archive.apache.org/dist/flink/flink-1.14.5/flink-1.14.5-bin-scala_2.11.tgz).\n\n### Grabbing RMLStreamer...\n\nDownload `RMLStreamer-\u003cversion\u003e.jar` from the [latest release](https://github.com/RMLio/RMLStreamer/releases/latest).\n\n### ... or building RMLStreamer\n\nIn order to build a jar file that can be deployed on a Flink cluster, you need:\n- a Java JDK \u003e= 11 and \u003c= 13 (We develop and test on JDK 11)\n- Apache Maven 3 or higher\n\nClone or download and then build the code in this repository:\n\n```\n$ git clone https://github.com/RMLio/RMLStreamer.git \n$ cd RMLStreamer\n```\nand then run:\n```\n$ mvn -DskipTests clean package\n```\n\n`-DskipTests` just builds and packages without running tests. If you want to run the tests, just omit this parameter.\n\n`clean` cleans any cached builds before packaging. While not strictly necessary, it is considered good practice to do\nso.\n\nThe resulting `RMLStreamer-\u003cversion\u003e.jar`, found in the `target` folder, can be deployed on a Flink cluster.\n\n**Note**: To build a *stand-alone* RMLStreamer jar, add `-P 'stand-alone'` to the build command, e.g.:\n```\n$ mvn clean package -DskipTests -P 'stand-alone'\n```\n\n**Note**: If you want to update the version of RMLStreamer (e.g. when developing or releasing), run the script\n`change-version.sh \u003cyour-new-version\u003e`. It updates the version on relevant places in the repository.\n\n### Executing RML Mappings\n\n*This section assumes the use of a CLI. If you want to use Flink's web interface, check out\n[this section](docker/README.md#3-deploy-rmlstreamer-using-the-flink-web-interface) in the Docker README.*\n\nHere we give examples for running RMLStreamer from the command line. We use `FLINK_BIN` to denote the Flink CLI tool,\nusually found in the `bin` directory of the Flink installation. E.g. `/home/myuser/flink-1.14.0/bin/flink`.\nFor Windows a `flink.bat` script is provided.\n\nThe general usage is:\n\n```\n$ FLINK_BIN run [Flink options] -c io.rml.framework.Main \u003cpath to RMLStreamer jar\u003e [toFile|toKafka|toTCPSocket] [options]\n```\n\n`FLINK HOME`  | The path to the provided Flink CLI script.\nFlink options | Options to the Flink run script. Example: `-p 4` sets the `parallelism` to 4.\n`-c io.rml.framework.Main` | This is the application class of RMLStreamer.\nPath to RMLStreamer jar | The absolute path to the RMLStreamer jar file.\nRMLStreamer options | The actual program arguments for RMLStreamer. See below for a full list.\n\n#### Basic commands:\n```shell script\n# write output to file(s)\n$FLINK_BIN run \u003cpath to RMLStreamer jar\u003e toFile --mapping-file \u003cpath to mapping file\u003e --output-path \u003cpath to output file\u003e  \n\n# write output to a listening socket (only if logical source(s) are streams)\n$FLINK_BIN run \u003cpath to RMLStreamer jar\u003e toTCPSocket --output-socket \u003chost:port\u003e\n\n# write output to kafka topic (only if logical source(s) are streams)\n$FLINK_BIN run \u003cpath to RMLStreamer jar\u003e toKafka --broker-list \u003chost:port\u003e --topic \u003ctopic name\u003e\n```\n\n#### Complete RMLStreamer usage:\n\n```\nUsage: RMLStreamer [toFile|toKafka|toTCPSocket|toMQTT|noOutput] [options]\n\n  -j, --job-name \u003cjob name\u003e\n                           The name to assign to the job on the Flink cluster. Put some semantics in here ;)\n  -i, --base-iri \u003cbase IRI\u003e\n                           The base IRI as defined in the R2RML spec.\n  --disable-local-parallel\n                           By default input records are spread over the available task slots within a task manager to optimise parallel processing,at the cost of losing the order of the records throughout the process. This option disables this behaviour to guarantee that the output order is the same as the input order.\n  -p, --parallelism \u003ctask slots\u003e\n                           Sets the maximum operator parallelism (~nr of task slots used)\n  -m, --mapping-file \u003cRML mapping file\u003e\n                           REQUIRED. The path to an RML mapping file. The path must be accessible on the Flink cluster.\n  --json-ld                Write the output as JSON-LD instead of N-Quads. An object contains all RDF generated from one input record. Note: this is slower than using the default N-Quads format.\n  --bulk                   Write all triples generated from one input record at once, instead of writing triples the moment they are generated.\n  --checkpoint-interval \u003ctime (ms)\u003e\n                           If given, Flink's checkpointing is enabled with the given interval. If not given, checkpointing is enabled when writing to a file (this is required to use the flink StreamingFileSink). Otherwise, checkpointing is disabled.\n  --auto-watermark-interval \u003ctime (ms)\u003e\n                           If given, Flink's watermarking will be generated periodically with the given interval. If not given, a default value of 50ms will be used.This option is only valid for DataStreams.\n  -f, --function-descriptions \u003cfunction description location 1\u003e,\u003cfunction description location 2\u003e...\n                           An optional list of paths to function description files (in RDF using FnO). A path can be a file location or a URL.\nCommand: toFile [options]\nWrite output to file \nNote: when the mapping consists only of stream triple maps, a StreamingFileSink is used. This sink will write the output to a part file at every checkpoint.\n  -o, --output-path \u003coutput file\u003e\n                           The path to an output file. Note: when a StreamingFileSink is used (the mapping consists only of stream triple maps), this path specifies a directory and optionally an extension. Part files will be written to the given directory and the given extension will be used for each part file.\nCommand: toKafka [options]\nWrite output to a Kafka topic\n  -b, --broker-list \u003chost:port\u003e[,\u003chost:port\u003e]...\n                           A comma separated list of Kafka brokers.\n  -t, --topic \u003ctopic name\u003e\n                           The name of the Kafka topic to write output to.\n  --partition-id \u003cid\u003e      EXPERIMENTAL. The partition id of kafka topic to which the output will be written to.\nCommand: toTCPSocket [options]\nWrite output to a TCP socket\n  -s, --output-socket \u003chost:port\u003e\n                           The TCP socket to write to.\nCommand: toMQTT [options]\nWrite output to an MQTT topic\n  -b, --broker \u003chost:port\u003e\n                           The MQTT broker.\n  -t, --topic \u003ctopic name\u003e\n                           The name of the MQTT topic to write output to.\nCommand: noOutput\nDo everything, but discard output\n\n```\n\n#### Examples\n\n##### Processing a stream\n\nAn example of how to define the generation of an RDF stream from a stream in an RML Mapping via TCP.\n```\n \u003c#TripleMap\u003e\n\n    a rr:TriplesMap;\n    rml:logicalSource [\n        rml:source [\n            rdf:type rmls:TCPSocketStream ;\n            rmls:hostName \"localhost\";\n            rmls:port \"5005\"\n        ];\n        rml:referenceFormulation ql:JSONPath;\n    ];\n\n    rr:subjectMap [\n        rml:reference \"$.id\";\n        rr:termType rr:IRI;\n        rr:class skos:Concept\n    ];\n\n    rr:predicateObjectMap [\n            rr:predicateMap [\n                rr:constant dcterms:title;\n                rr:termType rr:IRI\n            ];\n            rr:objectMap [\n                rml:reference \"$.id\";\n                rr:termType rr:Literal\n            ]\n        ].\n```\nThe RML Mapping above can be executed as follows:\n\nThe input and output in the RML Framework are both TCP clients when streaming. Before running stream mappings the input and output ports must be listened to by an application. For testing purposes the following commands can be used:\n ```\n$ nc -lk 5005 # This will start listening for input connections at port 5005\n$ nc -lk 9000 # This will start listening for output connections at port 9000\n # This is for testing purposes, your own application needs to start listening to the input and output ports. \n ```\nOnce the input and output ports are listened to by applications or by the above commands, the RML Mapping can be executed. RMLStreamer will open the input and output sockets so it can act upon data that will be written to the input socket.\n```\n$FLINK_BIN run \u003cpath to RMLStreamer jar\u003e toTCPSocket -s localhost:9000 -m .../framework/src/main/resources/json_stream_data_mapping.ttl\n# The -m paramater sets the mapping file location\n# The -s parameter sets the output socket port number\n```\n\nWhenever data is written (every data object needs to end with `\\n`) to the socket, this data will be processed by the RML Framework.\n\n##### Generating a stream from a Kafka Source\n\nAn example of how to define the generation of an RDF stream from a stream in an RML Mapping via Kafka.\n```\n \u003c#TripleMap\u003e\n\n    a rr:TriplesMap;\n    rml:logicalSource [\n        rml:source [\n            rdf:type rmls:KafkaStream ;\n            rmls:broker \"localhost:9092\" ;\n            rmls:groupId \"groupId\";\n            rmls:topic \"topic\";\n        ];\n        rml:referenceFormulation ql:JSONPath;\n    ];\n```\n\n**Note on using Kafka with Flink**: As a consumer, the Flink Kafka client never *subscribes* to a topic, but it is\n*assigned* to a topic/partition (even if you declare it to be in a *consumer group* with the `rmls:groupId` predicate). This means that it doesn't do\nanything with the concept *\"consumer group\"*, except for committing offsets. This means that load is not spread across\nRMLStreamer jobs running in the same consumer group. Instead, each RMLStreamer job is assigned a partition. \nThis has some consequences:\n* When you add multiple RMLStreamer jobs in a consumer group, and the topic it listens to has one partition,\nonly one instance will get the input.\n* If there are multiple partitions in the topic and multiple RMLStreamer jobs, it could be that two (or more) jobs\nare assigned a certain partition, resulting in duplicate output.\n\nSee also https://stackoverflow.com/questions/38639019/flink-kafka-consumer-groupid-not-working .\n\nThe only option for spreading load is to use multiple topics, and assign one RMLStreamer job to one topic.\n\n##### Generating a stream from a dataset\n\n```\n \u003c#TripleMap\u003e\n\n    a rr:TriplesMap;\n    rml:logicalSource [\n        rml:source \"/home/wmaroy/github/rml-framework/akka-pipeline/src/main/resources/io/rml/framework/data/books_small.json\";\n        rml:referenceFormulation ql:JSONPath;\n        rml:iterator \"$.store.books\"\n    ];\n\n    rr:subjectMap [\n        rml:reference \"id\";\n        rr:termType rr:IRI;\n        rr:class skos:Concept\n    ];\n\n    rr:predicateObjectMap [\n            rr:predicateMap [\n                rr:constant dcterms:title;\n                rr:termType rr:IRI\n            ];\n            rr:objectMap [\n                rml:reference \"id\";\n                rr:termType rr:Literal\n            ]\n        ] .\n        \n ```\n\n##### Generating a stream from a relational database\nRMLStreamer supports relational databases as a logical source. JDBC is used to establish a connection and perform a query against a database. See example mapping below.\n```ttl\n\u003cTriplesMap1\u003e\n  a rr:TriplesMap;\n\n  rml:logicalSource [\n    rml:source \u003c#DB_source\u003e;\n    rr:sqlVersion rr:SQL2008;\n    rr:tableName \"country_info\";\n  ];\n\n  rr:subjectMap [ rr:template \"http://example.com/{Country Code}/{Name}\" ];\n\t\n  rr:predicateObjectMap [ \n    rr:predicate ex:name ;\n    rr:objectMap [ rml:reference \"Name\" ]\n  ] .\n\n\u003c#DB_source\u003e a d2rq:Database;\n  d2rq:jdbcDSN \"CONNECTIONDSN\";\n  d2rq:jdbcDriver \"org.postgresql.Driver\";\n  d2rq:username \"postgres\";\n  d2rq:password \"\" .\n\n```\n \n#### RML Stream Vocabulary (non-normative)\n\nNamespace: \u003chttp://semweb.mmlab.be/ns/rmls#\u003e \n\nThe RML vocabulary have been extended with rmls to support streaming logical sources. \nThe following are the classes/terms currently used:\n* **rmls:[stream type]** \n    * **rmls:TCPSocketStream** specifies that the logical source will be a tcp socket stream.\n    * **rmls:FileStream** specifies that the logical source will be a file stream (to be implemented). \n    * **rmls:KafkaStream** specifies that the logical source will be a kafka stream.\n   \n \n* **rmls:hostName** specifies the desired host name of the server, from where data will be streamed from.\n\n\n* **rmls:port** specifies a port number for the stream mapper to connect to. \n    \nExample of a valid json logical source map using all possible terms: \n\n```\n\nrml:logicalSource [\n        rml:source [\n            rdf:type rmls:TCPSocketStream ;\n            rmls:hostName \"localhost\";\n            rmls:port \"5005\"\n        ];\n        rml:referenceFormulation ql:JSONPath;\n    ];\n```\n\n### Logging\n\nRMLStreamer uses Flink's Log4j 2 system. It can be configured in \n`$FLINK_HOME/conf/log4j.properties`.\n\nTo adjust the log level for RMLStreamer specifically, add two lines like this:\n\n```properties\nlogger.rmlstreamer.name = io.rml.framework\nlogger.rmlstreamer.level = DEBUG\n\n```\n\n### Benchmark \n\nRMLStreamer is benchmarked with this [repo.](https://github.com/s-minoo/rmlstreamer-benchmark-rust)\n\n### References ([preprint](./paper/RMLStreamer_ISWC.pdf))\n[1] S. Min Oo, G. Haesendonck, B. De Meester, A. Dimou. RMLStreamer - an RDF stream\ngenerator from streaming heterogeneous data. The Semantic Web – ISWC 2022. Springer International Publishing, (2022)\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRMLio%2FRMLStreamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRMLio%2FRMLStreamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRMLio%2FRMLStreamer/lists"}