https://github.com/engineering-research-and-development/fiware-orion-pyspark-connector
Bidirectional Orion/Orion-LD <--> PySpark Connector
https://github.com/engineering-research-and-development/fiware-orion-pyspark-connector
cognitive fiware ngsi ngsi-ld ngsi-v2 orion orion-context-broker orion-ld processing pyspark python spark
Last synced: 3 months ago
JSON representation
Bidirectional Orion/Orion-LD <--> PySpark Connector
- Host: GitHub
- URL: https://github.com/engineering-research-and-development/fiware-orion-pyspark-connector
- Owner: Engineering-Research-and-Development
- License: agpl-3.0
- Created: 2022-04-07T13:21:26.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-06T14:31:36.000Z (over 1 year ago)
- Last Synced: 2024-10-11T14:07:34.914Z (8 months ago)
- Topics: cognitive, fiware, ngsi, ngsi-ld, ngsi-v2, orion, orion-context-broker, orion-ld, processing, pyspark, python, spark
- Language: Python
- Homepage:
- Size: 2.39 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.txt
- Code of conduct: CODE_OF_CONDUCT.md
- Roadmap: docs/roadmap.md
Awesome Lists containing this project
README
[](https://www.fiware.org/developers/catalogue/)
[](https://opensource.org/licenses/AGPL-3.0)
[](https://stackoverflow.com/questions/tagged/fiware+pyspark)
[](https://quay.io/repository/fiware/pyspark-connector)
[](https://hub.docker.com/r/rdlabengpa/fiware-orion-pyspark-connector)
[](https://fiware-orion-pyspark-connector.readthedocs.io)
[](https://github.com/Engineering-Research-and-Development/fiware-orion-pyspark-connector/actions?query=workflow%3ACI)
[](https://bestpractices.coreinfrastructure.org/projects/6113)
[](https://coveralls.io/github/Engineering-Research-and-Development/fiware-orion-pyspark-connector?branch=increase-coverage-2)
The FIWARE PySpark Connector is a FIWARE Generic Enabler (GE) creating a data bridge between the FIWARE Context Brokers and PySpark
This project is part of [FIWARE](https://www.fiware.org/) For more information check the [FIWARE Catalogue](https://github.com/FIWARE/catalogue/tree/master/core)
| :books: [Documentation](https://fiware-orion-pyspark-connector.readthedocs.io/en/latest/) | 
[quay.io](https://quay.io/repository/fiware/pyspark-connector) | :whale: [Docker Hub](https://hub.docker.com/r/rdlabengpa/fiware-orion-pyspark-connector) | :dart: [Roadmap](https://github.com/Engineering-Research-and-Development/fiware-orion-pyspark-connector/blob/main/docs/roadmap.md) |
| -------------------------------------------------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --- |
## Table of Contents
- [What Is FIWARE PySpark Connector](#what-is-fiware-pyspark-connector)
- [Why Use FIWARE PySpark Connector](#why-use-fiware-pyspark-connector)
- [Quick start](docs/step_by_step.md#how-to-develop-an-ngsi-integrated-custom-algorithm-with-fiware-pyspark-connector)
- Admin Installation Instructions
- [Standalone Installation](docs/requirements.md)
- [Docker Installation](docs/docker.md)
- [User Documentation](docs/quick_start.md)
- [Step-by-Step Tutorial](docs/step_by_step.md)
- [Roadmap](docs/roadmap.md)
## What Is FIWARE PySpark Connector
FIWARE PySpark Connector is a FIWARE Generic Enabler (GE) made of a receiver and a replier subcomponents allowing a bidirectional communication between the FIWARE Context Brokers (CB) and PySpark. The component works on a low-level socket communication implementing a message passing interface between the two aforementioned counterparts. This interface is equipped with a parser function, hence permitting the creation of both NGSIv2 and NGSI-LD entities ready to use in a custom PySpark algorithm. Once data are preprocessed inside the PySpark evnironment, the component also provide a write-back interface (via REST API) to the CBs.
## Why Use FIWARE PySpark Connector
FIWARE PySpark Connector was created with the idea of expanding the FIWARE CB's environment to merge with python's one. Python is currently one of the most used programming languages for data analysis, providing lots of scientific libraries for data processing and visualization. Context brokers such as Orion, on the other hand, are powerful tools allowing the management of context elements and sending updates via subscriptions. This connector is able to open a communication path between these two instruments and to combine the advantages they provide at a little-to-none cost.
## Component Architecture
The FIWARE-PySpark receiver is currently a custom tool capable of receiving HTTP messages from a context broker and transform them to produce a batched stream of NGSI events to be processed by a PySpark job.
The tool is represented in the above diagram and it is made of:
- **`Connector Library`**: Set of functions to receive and reply to the context broker
- **`Connector configuration file`**: Configuration file to configure the connector servers and mode, class definition for NGSI Events and Parameter configuration for replier side.
- **`Subscribing Tool`**: Optional Library browsing the available entities in the context broker and allowing an easier subscription.
## License
The FIWARE PySpark Connector is licensed under [Affero General Public License (GPL) version 3](https://github.com/Engineering-Research-and-Development/fiware-orion-pyspark-connector/blob/main/LICENSE.txt).
© 2022 Engineering Ingegneria Informatica S.p.A.
PySpark Connector has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements No 870062 [CAPRI](https://www.capri-project.com/).
### Are there any legal issues with AGPL 3.0? Is it safe for me to use?
No problem in using a product licensed under AGPL 3.0. Issues with GPL (or AGPL) licenses are mostly related with the
fact that different people assign different interpretations on the meaning of the term “derivate work” used in these
licenses. Due to this, some people believe that there is a risk in just _using_ software under GPL or AGPL licenses
(even without _modifying_ it).
For the avoidance of doubt, the owners of this software licensed under an AGPL 3.0 license wish to make a clarifying
public statement as follows:
"Please note that software derived as a result of modifying the source code of this software in order to fix a bug or
incorporate enhancements is considered a derivative work of the product. Software that merely uses or aggregates (i.e.
links to) an otherwise unmodified version of existing software is not considered a derivative work, and therefore it
does not need to be released as under the same license, or even released as open source."