https://github.com/engineering-research-and-development/o2k-connector
Lightweight connector that enables context data sharing from Orion Context Broker towards Apache Kafka
https://github.com/engineering-research-and-development/o2k-connector
Last synced: 10 months ago
JSON representation
Lightweight connector that enables context data sharing from Orion Context Broker towards Apache Kafka
- Host: GitHub
- URL: https://github.com/engineering-research-and-development/o2k-connector
- Owner: Engineering-Research-and-Development
- License: agpl-3.0
- Created: 2022-06-16T13:19:14.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2025-01-31T10:35:47.000Z (over 1 year ago)
- Last Synced: 2025-04-03T16:21:55.742Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 318 KB
- Stars: 2
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# O2K-Connector: a FIWARE Orion to Kafka Connector
[](https://www.fiware.org/developers/catalogue/)
[](https://opensource.org/licenses/AGPL-3.0)
[](http://stackoverflow.com/questions/tagged/fiware+o2k)
[](https://quay.io/repository/fiware/o2k-connector)
[](https://registry.hub.docker.com/r/rdlabengpa/o2k-connector)
O2K-Connector is a lightweight connector, built in Python, that enables context data sharing from Orion Context Broker towards Apache Kafka. In more detail, O2K-Connector subscribes to Orion Context Broker and publishes the received context-data to a built-in kafka topic.
| 
[quay.io](https://quay.io/repository/fiware/o2k-connector) | :whale: [Docker Hub](https://hub.docker.com/r/rdlabengpa/o2k-connector) |
|---|---|
## O2K-Connector example placement in a FIWARE/Apache Architecture
# 
## Contents
- [Install](#install)
- [Configuration](#configuration)
- [Docker Install](#docker---recommended)
- [API](#api)
- [Testing](#testing)
- [License](#license)
## Configuration
O2K-Connector can be fully customized and configured by the means of environment variables listed on __*o2k.env*__ file.
Environment variables
Description
LOG_LEVEL
Level of O2K Debug. Possible values: INFO, DEBUG, ERROR
LOGS_PATH
Path of O2K-Connector logs
O2K_HOST
Listener for O2K-Connector
O2K_PORT
Port of O2K-Connector
SCHEMA_URL
Kafka Schema url for avro messages
BOOTSTRAP_SERVERS
Kafka Boostrap Servers
KAFKA_TOPIC
User-specified Kafka Topic, if left blank, it will be auto-generated on the fly taking into account FIWARE_SERVICE and FIWARE_SERVICEPATH environments for NGSI V2 and entity id for NGSI-LD
ORION_HOST
Hostname of Orion Context Broker
ORION_PORT
Port of Orion Context Broker
FIWARE_SERVICE
Fiware Service for NGSI V2
FIWARE_SERVICEPATH
Fiware ServicePath for NGSI V2s
MQTT_HOST
Hostname of MQTT Server, if used
MQTT_PORT
Port of MQTT Server, if used
MQTT_TOPIC
Topic to which publish on MQTT Server, if used
SUBSCRIPTION_JSON_PATH
Path of subscription configuration files
SUBSCRIPTION_JSON_FILENAME
Filename of subscription configuration file for NGSI V2 subscription
SUBSCRIPTION_JSON_FILENAME_LD
Filename of subscription configuration file for NGSI-LD subscription
SUBSCRIPTION_JSON_FILENAME_MQTT
Filename of subscription configuration file for NGSI-V2 subscription and MQTT underlying subscription engine
SUBSCRIPTION_SCHEMA_FILE_PATH
Filename of subscription schema file for NGSI V2 subscription
SUBSCRIPTION_SCHEMA_FILE_PATH_LD
Filename of subscription schema file for NGSI-LD subscription
SUBSCRIPTION_SCHEMA_FILE_PATH_MQTT
Filename of subscription schema file for NGSI-V2 subscription and MQTT underlying subscription engine
ORION_VERSION
Orion Context Broker version. Possible values: V2, LD
ORION_SUBSCRIPTION
Orion Context Broker subscription engine. Possible values: http, mqtt
MULTIPLE_SUBSCRIPTIONS
For NGSI V2 only: set to "false" to handle a unique subscription with multiple attributes
KAFKA_ENABLE_SSL
Set to "true" if want to use SSL connection to Kafka
KAFKA_SSL_CA
Certificate Authority key (path file inside the container) to use for SSL connection to Kafka
KAFKA_SSL_KEY
Client key (path file inside the container) to use for SSL connection to Kafka
KAFKA_SSL_CERTIFICATE
Certificate key (path file inside the container) to use for SSL connection to Kafka
## Getting Started - Install
To instantiate O2K-Connector you can use docker-compose which will take care of the creation of the container running python. Alternatively you can locally run the connector by running the python script.
N.B:
The connector will create a Kafka Topic structured as follows:
{FIWARE_SERVICE}\_{FIWARE_SERVICEPATH*}\_{NGSI_ENTITY_ID}\_{NGSI_ENTITY_TYPE}
*FIWARE_SERVICEPATH is deprived of '/' character
### Docker - Recommended
1. Configure subscription payload configuration _'subscription.json_' inside "conf" folder.
2. Configure your environment variables in the _'docker-compose.yml'_ file
3. Run docker-compose command into project root folder:
```sh
docker-compose up
```
## API
HTTP Method
Service
Description
POST
/notify
V2 Context data ingestion route
POST
/notifyld?subscriptionId=urn:ngsi-ld:Type:Entity
LD Context data ingestion route
## Testing
To test O2K-Connector, few services are required, in particular:
- Orion Context Broker
- A fully working Kafka Environment
At startup O2K-Connector subscribes to Orion Context Broker which is required to be running. Context-data will start being published to a given Kafka Topic (generate on-the-fly) as soon as possible.
To text context-data consumpion, run kafka-console-consumer command:
kafka-console-consumer --bootstrap-server broker-1:29092,broker-2:29093,broker-3:29094 --topic {FIWARE_SERVICE}\_{FIWARE_SERVICEPATH*}\_{NGSI_ENTITY_ID}\_{NGSI_ENTITY_TYPE}
*FIWARE_SERVICEPATH is deprived of '/' character
## License
O2K-Connector is licensed under
*GNU Affero General Public License v3.0*