{"id":13726034,"url":"https://github.com/vogler75/automation-gateway","last_synced_at":"2025-05-16T17:07:19.794Z","repository":{"id":41086769,"uuid":"336974682","full_name":"vogler75/automation-gateway","owner":"vogler75","description":"A OPC UA gateway which gives you access to your OPC UA values via MQTT or GraphQL (HTTP). If you have an OPC UA server in your PLC, or a SCADA system with an OPC UA server, you can query data from there via MQTT and GraphQL (HTTP). In addition, the gateway can also log value changes from OPC UA nodes in an InfluxDB, IoTDB, Kafka, and others.","archived":false,"fork":false,"pushed_at":"2025-02-28T08:20:52.000Z","size":21881,"stargazers_count":267,"open_issues_count":2,"forks_count":42,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-05-10T16:04:56.300Z","etag":null,"topics":["cratedb","gateway","graphql","influxdb","iotdb","java","jdbc","mqtt","neo4j","opc-ua","opcua","plc4x"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vogler75.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"vogler75"}},"created_at":"2021-02-08T05:55:12.000Z","updated_at":"2025-04-26T08:59:04.000Z","dependencies_parsed_at":"2023-10-17T09:34:42.488Z","dependency_job_id":"54836d59-ffc8-4451-98fb-6ca9f25ce379","html_url":"https://github.com/vogler75/automation-gateway","commit_stats":{"total_commits":451,"total_committers":4,"mean_commits":112.75,"dds":"0.026607538802660757","last_synced_commit":"61702f91588aaecb1ee32490d8f71afe06491178"},"previous_names":[],"tags_count":52,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vogler75%2Fautomation-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vogler75%2Fautomation-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vogler75%2Fautomation-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vogler75%2Fautomation-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vogler75","download_url":"https://codeload.github.com/vogler75/automation-gateway/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254573588,"owners_count":22093731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cratedb","gateway","graphql","influxdb","iotdb","java","jdbc","mqtt","neo4j","opc-ua","opcua","plc4x"],"created_at":"2024-08-03T01:02:49.565Z","updated_at":"2025-05-16T17:07:19.741Z","avatar_url":"https://github.com/vogler75.png","language":"Kotlin","funding_links":["https://github.com/sponsors/vogler75"],"categories":["Kotlin"],"sub_categories":[],"readme":"# Frankenstein Automation Gateway\n\nConnect one or more OPC UA servers, PLC4X devices or MQTT brokers to the gateway and access the data with a GraphQL, a MQTT, or an OPC UA client. The Gateway additionally offers functionality to log value changes from OPC UA, MQTT and PLC4X in a range of databases and platforms, including QuestDB, InfluxDB, Kafka, among others. Tested with up to 250000 value changes per second on comodity hardware.  \n\nAvailable logger sinks:  \n* TimescaleDB (JDBC)\n* InfluxDB\n* QuestDB\n* CrateDB\n* IoTDB\n* Neo4J\n* JDBC (PostgreSQL, MySQL, SQL Server)\n* OpenSearch (ElasticSearch)\n* Imply.io (Apache Druid)\n\n* Kafka\n* MQTT\n\nThe Gateway passes values through, it does not store data internally. The MQTT Broker is not a fully complient MQTT Broker. It does not keep values in memory. If you subscribe to a virtual MQTT-Topic, which must follow certain rules, then it will connect to the tags in OPC UA or PLC4X and will pass the values to the client. If multiple clients subscribe to the same virtual topic, then only one subscription to the device is made and the Gateway will distribute the values to all the clients.  \n\nThe Gateway also has an integrated OPC UA server. You can define what kind of data from MQTT brokers, other OPC UA servers and from PLC4X devices you want to have in the integraded OPC UA server. \n\nDocker images can be found on [Docker Hub](https://hub.docker.com/r/rocworks/automation-gateway).\n\nNews and Blog posts can be found [here](https://www.rocworks.at/wordpress/?cat=39).  \n\nHere is a tutorial on [YouTube](https://youtu.be/1xL89R0NBsk?si=6rXNbYKleGygzCui).\n\n![Gateway](doc/Automation-Gateway.png)\n![Architecture](doc/Architecture-Overview.png)\n\n# Content\n\n- [Build and Run](#build-and-run)\n- [Configuration](#configuration)\n- [OPCUA Schema in GraphQL](#opcua-schema-in-graphql)\n- [Topic Mapping](#topic-mapping)\n- [Logger Configuration](#logger-configuration)\n- [OPCUA Driver](#opcua-driver)\n- [PLC4X Driver](#plc4x-driver)\n- [MQTT Driver](#mqtt-driver)\n- [Build Docker Image](#build-docker-image)\n- [Version History](#version-history)\n\n# Build and Run\n\nIt needs [Java 17](https://openjdk.java.net/projects/jdk/17/) or higher. There is an issue with Java 8 update 292 and bouncycastl encryption. [here](https://github.com/bcgit/bc-java/issues/941) you can find more information about this issue..\n\nYou can open the project in IntelliJ IDEA IDE and build it there or use grade to build it from command line. Use the included \"gradlew\" command. Tt will download and use the right gradle version.\n\n```\n\u003e cd source/app\n\u003e ../gradlew build\n\n\u003e export GATEWAY_CONFIG=config.yaml  # Set configuration file (default is config.yaml)\n\u003e ../gradlew run\n```\n\nYou can run the application using Gradle, or alternatively, you can use the ZIP file located in the `build/distributions` directory (either `app.zip` or `app.tar`). Extract the ZIP file, and you'll find the executable file at `app/bin/app` (or `app.bat` on Windows), which you can use to run the gateway. \n\nOn Windows, you’ll need to modify the line that sets the CLASSPATH. Replace:\n```bash\nset CLASSPATH=...long list of libs...\n```\nwith:\n```bash\nset CLASSPATH=%APP_HOME%\\lib\\*\n```\n\nThis is because the full CLASSPATH, which includes all the libraries, is too long for Windows to handle. Otherwise, you'll encounter the error: \"The input line is too long.\"\n\nYou can also pass the configuration filename as an argument.\n\nApp is with GraphQL, MQTT and the OPC UA connections.    \nThere is also \"App-plc4x\" which includes the [plc4x](https://plc4x.apache.org/) connectivity.  \n\n## Configuration\n\nSee config.yaml in the app directory for an example how to configure the Gateway. You can pass a configuration file name to the program as the first argument or by setting a environment variable GATEWAY_CONFIG. If no argument is given then config.yaml will be used. \n* There is a YAML schema in the doc directory. This can be used with Visual Studio Code to create a valid YAML configuration for the gateway.\n\n* In Visual Studio Codee install the \"YAML Language Support by Red Hat\" Extension. In the settings of this extension you will find the \"schema\" section, open the settings.json and add one line to the yaml.schemas sections: \n```\"\nyaml.schemas\": {        \n        \"your-path-to-gateway/doc/yaml-json-schema.json\": [\"config*.yaml\"]\n}\n```\n\nIf you enable [GraphiQL](https://github.com/graphql/graphiql), a graphical ui to build and execute GraphQL queries, then you can access GraphiQL with  \n\u003e http://localhost:4000/graphiql/ **! trailing slash is important !**\n\n```yaml\nServers:\n  Mqtt:\n    - Id: Mqtt\n      Port: 1883\n      Host: 0.0.0.0\n      LogLevel: INFO # ALL | INFO\n      \n  OpcUa:\n    - Port: 4841\n      LogLevel: INFO\n      Topics:\n        - Topic: opc/demo1/path/#\n        - Topic: opc/demo2/path/#  \n\n  GraphQL:\n    - Id: GraphQL\n      Port: 4000\n      LogLevel: INFO\n      GraphiQL: true\n```\n\n## OPCUA Schema in GraphQL\n\nThe GraphQL server can read the OPC UA object schema and convert it to a GraphQL schema. The starting NodeIds can be set to reduce the amount of browsed items. Browsing can take some while if the OPC UA server holds a huge structure of tags!\n```yaml\nServers:\n  GraphQL:\n    - Id: GraphQL\n      Port: 4000\n      Enabled: true\n      LogLevel: INFO                    # ALL | INFO\n      GraphiQL: true\n      WriteSchemaToFile: false\n      Schemas:                          # This systems will be browsed and converted to GraphQL\n        - System: \"unified\"             # Id of OPC UA system, must correlate with the drivers id\n          FieldName: BrowseName         # Use \"BrowseName\" or \"DisplayName\" as item name in GraphQL\n          RootNodes: \n            - ns=2;s=Simulation         # Node will be browsed and added to GraphQL schema \n            - ns=2;s=SimulationMass     # Node will be browsed and added to GraphQL schema          \n```\n\n```yaml\nDrivers:\n  OpcUa\n  - Id: \"unified\" \n    Enabled: true\n    LogLevel: INFO\n    EndpointUrl: \"opc.tcp://scada-server:4890\"\n    UpdateEndpointUrl: scada-server\n    SecurityPolicyUri: http://opcfoundation.org/UA/SecurityPolicy#None  \n```\n\nExample GraphQL Query with two OPC UA systems:\n\n```\n{\n  Systems {\n    unified {\n      HmiRuntime {\n        HMI_RT_5 {\n          Structure_instances{\n            A1 {\n              Velocity { ...Value }\n              RefPoint { ...Value }\n            }\n          }\n        }\n      }\n    }    \n    ignition {\n      Tag_Providers {\n        default {\n          Pump_1 {\n            flow { ...Value }\n            speed { ...Value }\n          }\n        }\n      }\n    }\n  }\n}\n\nfragment Value on Node {\n  Value {\n    Value\n    SourceTime\n  }\n}\n\n```\n\n## Topic Mapping\nWith the built in MQTT Interface you can get access to the connected OPC UA servers by subscribing to MQTT Topics.  \n\nThe Topic name follows a certain rule. When a MQTT client subscribes to such MQTT topics, then the Gateway will create a subscription to the Node in the OPC UA Server. If multiple clients subscribe to the same Topic/Node, then the Gateway will act as a distributor and only one connection to the node is made.  \n\nNote: In the following examples \"test\" is the Id of the OPC UA Client in the configuration file.\n\nNote: Remove the blanks between the slashe! Just here for better readabilty. \n\nUsing the NodeId \n\u003e opc / test / node / ns=2;s=ExampleDP_Float.ExampleDP_Arg1   \n\u003e opc / test / node / 2 / ExampleDP_Float.ExampleDP_Arg1   \n\nValue as JSON with timestamp, quality, ...  \n\u003e opc / test / node:**json** / ns=2;s=ExampleDP_Float.ExampleDP_Arg1  \n\u003e opc / test / node:**json** / 2 / ExampleDP_Float.ExampleDP_Arg1  \n\nUsing the browse path instead of the NodeId  \n\u003e opc / test / **path** / *root-node-id* / *browse-name* / *browse-name* /...   \n\nWildcard \"+\" can also be used as a browsename\n\u003e opc / ua / **path**:json / *ns=1;s=16|Tags* / **+**  \n\n\"Objects\" can be used as root node and will be replace with \"i=85\"\n\u003e opc / test / path / **Objects** / Test / Test00003 / float  \n\u003e opc / test / path / **Objects** / Test / Test00003 / +\n\nBe careful when using wildcards when there are a lot of nodes, it can lead to a lot of browsing round trips  \n\u003e opc / test / path / Objects / Test / + / float\n\n## Logger Configuration\nLoggers for different type of sinks can be defined in the configuration file. All of them share a common configuration and can have additonal sink specific configuration. Sink specific configuration can be found in the example configuration files or in the version history.\n\nIn the \"Logging\" section we can specify the Topics which should be logged to the sink. The Topics follow the same rule as MQTT Topics and will map to sources like OPC UA or PLC4X. See [Topic Mapping](#topic-mapping).\n```yaml\nLoggers:\n  InfluxDB:\n    - Id: influx1\n      Enabled: true\n      Url: http://192.168.1.13:8086\n      Database: test\n      Username: \"\"\n      Password: \"\"\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_SByte/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Byte/+\n  \n```\n\nEvery logger has an internal topic where the throughput is updated every second.\nTopic: logger/**logger-id**/metrics\nValue: {\"Input v/s\":20932,\"Output v/s\":20932}  # just an example\n\n## OPCUA Driver\nSee config directory for more example configurations.\n```yaml\nDrivers:\n  OpcUa:\n  - Id: \"unified\"\n    Enabled: true\n    LogLevel: INFO\n    EndpointUrl:  \"opc.tcp://desktop-9o6hthf:4890\"\n    SecurityPolicy: Basic128Rsa15\n    SubscriptionSamplingInterval: 0\n    UsernameProvider:\n      Username: opcuauser\n      Password: password1\n    WriteParameters:\n      QueueSize: 1000\n      BlockSize: 100\n      WithTime: true\n    MonitoringParameters:\n      BufferSize: 10\n      SamplingInterval: 0.0\n      DiscardOldest: true\n```\n\n## PLC4X Driver\n\nYou the application app-plc4x to get values from various source supported by PLC4X. See config.yaml in each folder of the app. Because some PLC4X drivers/plc's do not support subscriptions we have added a simple polling options (currently only one polling time per connection). \n\n\u003e cd app-plc4x  \n\u003e cat config.yaml\n\n```yaml\nDrivers:\n  Plc4x:\n  - Id: \"machine1\"\n    Enabled: true\n    Url: \"modbus://127.0.0.1:502\"\n    Polling:\n      Time: 1000 # ms\n      Timeout: 900 # ms\n      OldNew: true\n    WriteTimeout: 100 # ms\n    ReadTimeout: 100 # ms\n    LogLevel: ALL\n```\n\u003e gradle run\n\n\nExample GraphQL Query:\n```\n{\n  a: NodeValue(\n    Type: Plc,\n    System: \"mod\"\n    NodeId: \"coil:1\"\n  ) {\n    Value\n  }\n  b: NodeValue(\n    Type: Plc\n    System: \"mod\"\n    NodeId: \"holding-register:1:INT\"\n  ) {\n    Value\n    SourceTime\n  }  \n}\n```\n\nExample MQTT Topic:\n\u003e plc/mod/node:json/holding-register:1:INT  \n\u003e plc/mod/node/holding-register:2:INT  \n\u003e plc/mod/node:json/coil:1  \n\u003e plc/mod/node/coil:1  \n\n## MQTT Driver\n\nMQTT Brokers can be connected with MQTT Driver. Withte the RAW Format the values are strings. JSON Format will be the Gateway's JSON format. If you add \"CustomJson\" to the config, then the JSON format can be customized. You can set JSON-Paths (without $.) to get the value out of a JSON. Currently only JSON objects are supported (not arrays).\n```yaml\nDrivers:  \n  Mqtt:\n    - Id: \"mqtt1\"\n      LogLevel: INFO\n      Host: 192.168.1.3\n      Port: 1883\n      Format: Json\n      CustomJson: \n          Value: \"Value\"\n          TimestampMs: \"TimeMS\"    \n          TimestampIso: \"TimeISO\"\n```\n\n## Build Docker Image\n\nYou have to build the program before with gradle. Then you can use the shell script `docker/build.sh` to build a docker image.  \n`docker run --rm --name gateway -p 4000:4000 -p 1883:1883 -v $PWD/config.yaml:/app/config.yaml gateway`\n\n\u003e C:\\Workspace\\automation-gateway\\source\u003e gradle build  \n\u003e C:\\Workspace\\automation-gateway\\docker\u003e build.bat  \n\u003e C:\\Workspace\\automation-gateway\\docker\\examples\\hazelcast\u003e docker compose up -d  \n\n# Version History\n- [1.37 Support for Imply.io \u0026 reactivated QuestDB](#137-support-for-implyio--reactivated-questdb)\n- [1.36 Support for InfluxDB V2](#136-support-for-influxdb-v2)\n- [1.35 Enhancements and Bug Fixes](#135-enhancements-and-bug-fixes)\n- [1.34 Added Config Upload Page](#134-added-config-upload-page)\n- [1.33 Rework of Logging](#133-rework-of-logging)\n- [1.32 Zenoh Logger](#132-zenoh-logger)\n- [1.31 QuestDB Logger](#131-questdb-logger)\n- [1.30 OpenSearch Logger](#130-opensearch-logger)\n- [1.29 Extended OPC UA Browsing](#129-extended-opc-ua-browsing)\n- [1.28 Various changes and code rework](#128-various-changes-and-code-rework)\n- [1.27 Add target option to MQTT Logger for UNS](#127-add-target-option-to-mqtt-logger-for-uns)\n- [1.26 Reactivated Neo4j Logger](#126-reactivated-neo4j-logger)\n- [1.25 MQTT Driver Custom JSON Format](#125-mqtt-driver-custom-json-format)\n- [1.24 Added OPC UA server](#124-added-opc-ua-server)\n- [1.23 Upgrade to VertX 4.4.6](#123-upgrade-to-vertx-446)\n- [1.22 Config file changes](#122-config-file-changes)\n- [1.21.2 Fixes and SparkplugB for Kafka \\\u0026 MQTT Logger](#1212-fixes-and-sparkplugb-for-kafka--mqtt-logger)\n- [1.21.1 Fixes and SparkplugB for MQTT Client](#1211-fixes-and-sparkplugb-for-mqtt-client)\n- [1.21 IoTDB, MQTT SparkplugB Logger, YAML Schema, Native-Image](#121-iotdb-mqtt-sparkplugb-logger-yaml-schema-native-image)\n- [1.20.3 Moved Neo4J to separate branches](#1203-moved-neo4j-to-separate-branches)\n- [1.20.2 Modifed JSON Format of Kafka Logger](#1202-modifed-json-format-of-kafka-logger)\n- [1.20.1 Kafka properties in the config file](#1201-kafka-properties-in-the-config-file)\n- [1.20 Cleanup and GraalVM Native Build](#120-cleanup-and-graalvm-native-build)\n- [1.19 Neo4j Logger](#119-neo4j-logger)\n- [1.18.3 Added MQTT Websocket Option and simple Authentication](#1183-added-mqtt-websocket-option-and-simple-authentication)\n- [1.18.2 Raw value to engineering value conversion for PLC4X driver](#1182-raw-value-to-engineering-value-conversion-for-plc4x-driver)\n- [1.18.1 Features and fixes in PLC4X driver](#1181-features-and-fixes-in-plc4x-driver)\n- [1.18 Removed Apache Ignite](#118-removed-apache-ignite)\n- [1.17 Added CrateDB as supported JDBC database for logging](#117-added-cratedb-as-supported-jdbc-database-for-logging)\n- [1.16 JDBC Logger to write field values to relational databases](#116-jdbc-logger-to-write-field-values-to-relational-databases)\n- [1.15 Nats Logger to write field values to a Nats server](#115-nats-logger-to-write-field-values-to-a-nats-server)\n- [1.14 Fixes and optimizations](#114-fixes-and-optimizations)\n- [1.13 MQTT Logger to write field values to a MQTT Broker](#113-mqtt-logger-to-write-field-values-to-a-mqtt-broker)\n- [1.12 MQTT Driver with Groovy script transformer](#112-mqtt-driver-with-groovy-script-transformer)\n- [1.11 Apache Kafka Database Logger](#111-apache-kafka-database-logger)\n- [1.10 Apache IoTDB Database Logger](#110-apache-iotdb-database-logger)\n- [1.9 Apache Ignite as Cluster option and Ignite as Memory-Store](#19-apache-ignite-as-cluster-option-and-ignite-as-memory-store)\n- [1.8 Upgrade to VertX 4.0.3](#18-upgrade-to-vertx-403)\n- [1.7 DDS Driver (subscribe and publish)](#17-dds-driver-subscribe-and-publish)\n- [1.6 Added GraphiQL (http://localhost:4000/graphiql/)](#16-added-graphiql-httplocalhost4000graphiql)\n- [1.5 OPC UA Schemas to GraphQL Schema Importer](#15-opc-ua-schemas-to-graphql-schema-importer)\n\n## 1.37 Support for Imply.io \u0026 reactivated QuestDB\n\nSupport has been added for [Imply.io](https://docs.imply.io/polaris/api-stream/) to enable pushing event data via API directly into **Imply.io** (Apache Druid in the Cloud). This eliminates the need for a separate Apache Kafka instance for data ingestion. The automation-gateway will create the connection and the job for data ingestion directly in imply.io.\n\nTo ensure successful integration, the API key used must have the following permissions:\n\n+ ManageConnections: To create and edit connections.\n+ ManageTables: To create and modify tables.\n+ ManageIngestionJobs: To create ingestion jobs for connections.  \n\n\n### Example Configuration\n\n```yaml\nImply:\n  - Id: Druid1\n    Enabled: true\n    LogLevel: INFO\n    Host: domain.region.aws.api.imply.io\n    ApiKey: \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n    ProjectId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx               \n    ConnectionName: gateway\n    TableName: gateway\n    Logging:\n      - Topic: opc/scada/path/Objects/Mqtt/home/Original/Meter_Output/WattAct\n      - Topic: opc/scada/path/Objects/Mqtt/home/Original/Meter_Input/WattAct\n      - Topic: opc/scada/path/Objects/Mqtt/home/Original/PV/Calc/#\n```\n\n\n## 1.36 Support for InfluxDB V2\n\nAdded support for InfluxDB V2 connection to be able to connect with Token, Org and Bucket. \n```yaml\nLoggers:\n  InfluxDB:\n    - Id: influxdb\n      Enabled: true\n      LogLevel: INFO\n      Version: 2\n      Url: \"https://xxxxxxxxxx.aws.cloud2.influxdata.com\"\n      Token: \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n      Org: rocworks\n      Bucket: scada\n      Measurement: opc  # Optionally, if not set the system name of the source topic will be used\n      WriteParameters:\n        QueueSize: 1000000\n      Logging:\n        - Topic: opc/scada/path/Objects/Mqtt/home/Original/Meter_Input/WattAct\n        - Topic: opc/scada/path/Objects/Mqtt/home/Original/Meter_Output/WattAct\n        - Topic: opc/scada/path/Objects/Mqtt/home/Original/PV/Calc/FlowWatt\n```\n\n## 1.35 Enhancements and Bug Fixes\n\nRemoved GraphiQL due to deprecation in Vert.x.\nAdded necessary files for building a SIEMENS Industrial Edge App.\nIntroduced a web page for config file upload functionality.\nFixed an issue in the MQTT Driver where reconnecting to the MQTT Broker caused incoming messages to be sent multiple times to the bus.\n  \n\n## 1.34 Added Config Upload Page\n\nA simple web page has been added to facilitate the upload of a configuration file. This feature is particularly useful when the gateway is installed on an Edge Device, such as a SIMATIC HMI Comfort Panel. Additionally, an option is provided for the gateway to attempt to read the configuration file at regular intervals. Once the configuration file is uploaded, the gateway will read the file and initiate the startup process.\n\nEnable Upload Page: Set the environment variable GATEWAY_CONFIG_HTTP to enable the configuration file upload page. Example: GATEWAY_CONFIG_HTTP=8080 (You can specify any desired port number).\n\nEnable Periodic Configuration File Reading: Set the environment variable GATEWAY_CONFIG_RETRY to enable the periodic reading of the configuration file. Example: GATEWAY_CONFIG_RETRY=10 (The gateway will retry every 10 seconds to locate a configuration file). \n\n## 1.33 Rework of Logging\n\nLogging ensures now no data loss. If a connection is lost, values are temporarily stored in memory or on disk and written once the connection is restored. The default storage is memory. To specify the storage type, use the QueueType parameter in WriteParameters. If DISK is chosen, QueueSize sets the file size in bytes. If memory is used, QueueSize is the number of data points. This specifies the maximum space available for buffered data. The file will be pre-allocated to this size. Currently, each tag value consumes around 1400 bytes due to Java Object serialization, but we must optimize this to decrease storage requirements.\n\n```yaml\n Jdbc:\n    - Id: postgres      \n      Enabled: false\n      Url: jdbc:postgresql://linux0:5432/scada\n      WriteParameters:\n        QueueType: DISK\n        QueueSize: 1073741824 # Filesize in bytes (1GB)\n        DiskPath: /data/buffers # Storage location for disk files\n``` \n\nWe have removed Zenoh and DuckDB from the main branch for maintenance purposes.\n\n## 1.32 Zenoh Logger\n\nAdded logger for Zenoh.\n\nFor **Zenoh** only the default configuration settings are currently available. Zenoh is also commented out in the \"App\", because Zenoh uses native libs and the size of the libs is high. Also the native libs maybe hinder the compilation of a native executable with GraalVM (not tested). If you need Zenoh, then go to the App.kt and remove the comments from the Zenoh related lines.\nYou need to set the following variables in ~/.gradle/gradle.properties, because the current Zenoh Java libs are not yet available at maven and must be fetched from github.\n```\ngithub_user=xxx\ngithub_token=xxx\n```\nAlso comment in the github maven repository for Zenoh in the settings.gradle file in the source directory\n\n## 1.31 QuestDB Logger\n\nLogger for QuestDB.\n\nYou should create the logging table before you start the logger.\n```sql\nCREATE TABLE gateway (\n    time timestamp,\n    system symbol,\n    address symbol,\n    value double,\n    text varchar\n) TIMESTAMP(time) PARTITION BY MONTH;\nALTER TABLE gateway DEDUP ENABLE UPSERT KEYS(time, system, address)\nALTER TABLE gateway ALTER COLUMN system ADD INDEX;\nALTER TABLE gateway ALTER COLUMN address ADD INDEX;\n```\n\nExample configuration:\n```yaml\nLoggers:\n  QuestDB:\n    - Id: Qdb0\n      Enabled: true\n      Config: http::addr=nuc1.rocworks.local:9001;\n      Table: home1\n      Logging:\n        - Topic: mqtt/home/path/Original/#\n```\n\n## 1.30 OpenSearch Logger\n\nLogger for OpenSearch / Elasticsearch. \n\nYou must create an index *template* \\\u003cindex name\\\u003e with an index *pattern* \"\\\u003cindex name\\\u003e-*\" with the following JSON index mapping:\n```json\n{\n  \"properties\": {\n    \"topicName\": { \"type\": \"text\" },\n    \"systemType\": { \"type\": \"text\" },\n    \"systemName\": { \"type\": \"text\" },\n    \"topicType\": { \"type\": \"text\" },\n    \"topicPath\": { \"type\": \"text\" },\n    \"topicNode\": { \"type\": \"text\" },\n    \"browsePath\": { \"type\": \"text\" },\n    \"valueAsString\": { \"type\": \"text\" },\n    \"valueAsNumber\": { \"type\": \"double\" },\n    \"statusCode\": { \"type\": \"text\" },\n    \"sourceTime\": { \"type\": \"date\" },\n    \"serverTime\": { \"type\": \"date\" }\n  }\n}\n```\n\nExample configuration:\n```yaml\nLoggers:\n  OpenSearch:\n    - Id: Search1\n      Enabled: true\n      LogLevel: INFO\n      Host: linux0\n      Port: 9200\n      Index: gateway\n      Logging:\n        - Topic: mqtt/home/path/Original/#\n```\n\n## 1.29 Extended OPC UA Browsing \n\nNodes of type Variable will now be browsed. Before browsing stopped, when it reached a node of type Variable. But some OPC UA servers, like the one from SIEMENS PLCs, are using a Variable node type for User-Defined-Datatypes or Structs. \n\n## 1.28 Various changes and code rework\n\n- Upgrade to PLC4X 0.9\n- Upgrade to Vertx 4.5.1\n- Added Duckdb Logger\n- Added HSQLDB Support for JDBC Logger\n- Added ExecuteSQL GraphQL function \n- IoTDB Logger improvements\n- Rework of SparkplugB messages\n- Rework of BrowsePath in messages\n\n## 1.27 Add target option to MQTT Logger for UNS\n\nWith the \"Target\" option at the Logging Topics, a transformation of the input topic to the target topic can now be done.  With that a Unified Namespace (UNS) can be created at the target MQTT broker, with incomding non UNS topic names.  \n\nIf a wildcard is used at the source topic, like opc/home1/path/Objects/Mqtt/home/Original/Gas/#, then the browsed/resolved names can be added to the end of the target topic. Just add a wildcard \"#\" also at the end of the target topic. If there is no wildcard at the target, then all the resolved topics of the source are all written to the same target topic.\n```yaml\nLoggers:\n  Mqtt:\n    - Id: \"mqtt1\"\n      Enabled: true\n      Host: 192.168.1.4\n      Port: 1883\n      Topic: demo\n      Format: JsonSimple\n      LogLevel: INFO\n      Retained: false\n      Logging:\n        - Topic: opc/demo2/path/Objects/Demo/SimulationMass/SimulationMass_Boolean/Boolean_00\n          Target: uns/rocworks/site1/area1/line1/sim1/bool00\n        - Topic: opc/demo2/path/Objects/Demo/SimulationMass/SimulationMass_Byte/#\n          Target: uns/rocworks/site1/area1/line1/sim2/#\n        - Topic: opc/home1/path/Objects/Mqtt/home/Original/Gas/#\n          Target: uns/rocworks/site1/area1/line1/gas/#\n        - Topic: opc/home1/path/Objects/Mqtt/home/Original/Meter_Input/#\n          Target: uns/rocworks/site1/area1/line1/meter/input/#\n        - Topic: opc/home1/path/Objects/Mqtt/home/Original/Meter_Output/#\n          Target: uns/rocworks/site1/area1/line1/meter/output/#\n        - Topic: opc/home1/path/Objects/Mqtt/home/Original/PV/#\n          Target: uns/rocworks/site1/area1/line1/pv/# \n```\n\n## 1.26 Reactivated Neo4j Logger  \nAdded Neo4j as an option to log values from MQTT or OPC UA to the graph database. Additionally the OPC UA node structure can also be replicated to the graph database. This will be done only once at the startup of the Automation Gateway. For MQTT the node structure will be built during runtime, as new topics are coming in, the structure will be created.\n\n```yaml\nLoggers:\n  Neo4j:\n    - Id: neo4j\n      Enabled: true\n      Url: bolt://nuc1.rocworks.local:7687\n      Username: \"neo4j\"\n      Password: \"neo4j\"\n      Schemas:\n        - System: demo1\n          RootNodes:\n            - \"ns=2;s=Variables\"\n        - System: demo2\n          RootNodes:\n            - \"ns=2;s=Demo\"\n      Logging:\n        - Topic: mqtt/mqtt1/path/Original/#\n        - Topic: opc/demo1/path/Objects/Variables/#\n        - Topic: opc/demo2/path/Objects/Demo/SimulationMass/#\n```\n## 1.25 MQTT Driver Custom JSON Format\nWith format JSON it is now possible to define the JSON-Path for the value and for the timestamp in milliseconds since epoch or ISO 8601. If CustomJson is not defined, then the JSON content is a defined JSON format of the Gateway. The format is used for reading and writing.  \n```yaml\nDrivers:  \n  Mqtt:\n    - Id: \"mqtt1\"\n      LogLevel: INFO\n      Host: 192.168.1.3\n      Port: 1883\n      Format: Json\n      CustomJson: \n          Value: \"Value\"\n          TimestampMs: \"TimeMS\"    \n          TimestampIso: \"TimeISO\"\n```\n## 1.24 Added OPC UA server\nThe Gateway now also has an integrated OPC UA server. You can define what kind of data from MQTT brokers, other OPC UA servers and from PLC4X devices you want to have in the integraded OPC UA server. Data will be mapped to structured nodes in the OPC UA server. It is also possible to change the values in the OPC UA server and the changed values will be written back to the source (MQTT broker, other OPC UA server, PLC4X connected device).  \n\nThere is now also a GraphQL interface for the configuration of the gateway. This GraphQL Server can be enabled by setting the environment variable \"GATEWAY_CONFIG_PORT=9999\". 9999 will be the port for the configuration GraphQL serer. This configuration GraphQL server can be used to build a configuration Ui for the gateway.  \n\n## 1.23 Upgrade to VertX 4.4.6\nThe GraphQL server websocket subprotocol now changed to the new one: \"graphql-transport-ws\". The older Apollo subprotocol \"graphql-ws\" is not supported anymore.  \n\nNote: In some GraphQL context the naming of the two protocols can be confusing. From [apollographql.com](https://www.apollographql.com/docs/react/data/subscriptions/): Confusingly, the subscriptions-transport-ws library calls its WebSocket subprotocol graphql-ws, and the graphql-ws library calls its subprotocol graphql-transport-ws! The names of the protocol and the websocket subprotocol are exchangend.\n\n## 1.22 Config file changes\n!!! Config file structure has changed !!! \n\nTo have a consistant layout of the config file, it was necessary to change the structure of it. Existing config files must be changed! Please use the Visual Studio Code YAML plugin to change your existing configs. See [Configuration](#configuration).\n\nExample of the new config file:\n```yaml\nServers:\n  GraphQL:\n    - Id: \"GraphQL\"        \n  Mqtt:\n    - Id: \"Mqtt\"\n\nDrivers:   \n  Mqtt:\n    - Id: remote\n      Host: bd9c43f59b7a42deba3248fca439f378.s1.eu.hivemq.cloud\n      Port: 8883\n\n  OpcUa:\n    - Id: demo1\n      EndpointUrl: \"opc.tcp://192.168.1.3:62540/server\"\n      SecurityPolicy: None\n\n    - Id: demo2\n      EndpointUrl: \"opc.tcp://192.168.1.3:62541\"\n      SecurityPolicy: None     \n\nLoggers:    \n  InfluxDB:\n    - Id: InfluxLogger1      \n      Url: http://nuc1b.rocworks.local:8086\n      Database: test\n      Logging:\n        - Topic: opc/demo1/path/Objects/Variables/#\n        - Topic: opc/demo2/path/Objects/Demo/SimulationMass/#\n        - Topic: mqtt/remote/path/Austria/Sparkplug/#\n\n  Mqtt:\n    - Id: MqttLogger1\n      Host: linux0.rocworks.local\n      Port: 1883\n      Format: Raw\n      Topic: test1\n      Logging:\n        - Topic: mqtt/remote/path/Austria/Sparkplug/#\n  \n```\n\n## 1.21.2 Fixes and SparkplugB for Kafka \u0026 MQTT Logger\nKafka and MQTT Logger can now publish SparkplugB message format. \n\n## 1.21.1 Fixes and SparkplugB for MQTT Driver\nThere is now a Format option for the MQTT driver. It can now read SparkplugB messages from topics. You can use now a logger to log values from a MQTT broker which are in SparkplugB message format. You can also write values from GraphQL or publish a value from the MQTT server to the MQTT driver. If the format of the MQTT driver is set to SparkplugB, it will publish a SparkplubB message.\n```yaml\nDriver:\n  Mqtt:\n    - Id: \"mqttclient1\"\n      Host: linux0.rocworks.local\n      Port: 1883    \n      Format: SparkplugB # RAW | JSON\n```\nMQTT Publish Example:   \n* mqtt/mqttclient1/node:value/Enterprise/Test =\u003e Hallo World  \n* mqtt/mqttclient1/node:json/Enterprise/Test =\u003e {\"value\": \"Hello World\", \"sourceTime\":\"2023-10-19T18:23:55.389Z\"}  \n\n## 1.21 IoTDB, MQTT SparkplugB Logger, YAML Schema, Native-Image\n* IoTDB is now again available as data logger.  \n* SparkplubB message format for MQTT logger.\n```yaml\n  Logger:\n    - Id: mqtt1\n      Type: Mqtt\n      Enabled: true\n      LogLevel: INFO       \n      Mqtt:\n        Host: linux0.rocworks.local\n        Port: 1883\n        Topic: Enterprise/Site/Area/Line\n        Format: Json\n        BulkMessages: false                   \n      Logging:\n        - Topic: opc/demo1/path/Objects/Variables/#\n        - Topic: opc/demo2/path/Objects/Demo/SimulationMass/#\n```\n* YAML json schema is now availabe in the doc directory. It can be used with the \"YAML Language Support by Red Hat\" Extension. In the settings find the Schema section, open the settings.json and add one line to the yaml.schemas sections: \n```\"\nyaml.schemas\": {        \n        \"your-path-to-gateway/doc/yaml-json-schema.json\": [\"config*.yaml\"]\n}\n```\n* Update to Gradle 8.4 with new build files.\n* Native image build was upgraded to GraalVM 17 and Java 17.\n* Native image works with Mqtt,Kafka,InfluxDB and IoTDB.\n* Logger configurations have now a separate object for the type specific configurations (but the old style yaml format is still supported). This was necessary for the YAML json schema. \n```yaml\n- Id: iotdb1\n    Type: IoTDB\n    Enabled: false\n    IoTDB: # same name as Type\n      Host: linux0.rocworks.local\n      Port: 6667\n      Database: root.gateway\n      Username: \"root\"\n      Password: \"root\" \n      LogLevel: INFO       \n\n```\n\n* For published data we use now a DataPoint type instead of JSON. With the JSON format we have lost the origin datatype of the source. By using the new DataPoint type (Topic+TopicValue) the data type is preserved. \n* Upgrade from Vert.X 4.2.4 to 4.2.7\n\n## 1.20.3 Moved Neo4J to separate branches \nNeo4J is now in a separate branch and is removed from the main branch\n\n## 1.20.2 Modifed JSON Format of Kafka Logger\nAdded times in ms epoch and also added the value as double and as string. \n```json\n{\n\t\"nodeId\": \"ns=2;i=3\",\n\t\"systemName\": \"scadaopcua\",\n\t\"topicName\": \"opc/scadaopcua/path/Objects/Home/#\",\n\t\"browsePath\": \"Objects/Home/Gas/Daily\",\n\t\"sourceTime\": \"2023-07-14T09:53:43.945894Z\",\n\t\"serverTime\": \"1601-01-01T00:00:00Z\",\n\t\"sourceTimeMs\": 1689328423945,\n\t\"serverTimeMs\": -11644473600000,\n\t\"value\": 2,\n\t\"valueAsString\": \"2\",\n\t\"valueAsDouble\": 2,\n\t\"statusCode\": \"0\"\n}\n```\n## 1.20.1 Kafka properties in the config file\nIt can be configured with a bunch of properties as described in the official [Apache Kafka documentation](https://kafka.apache.org/documentation/#producerconfigs).  \n\nPut the properites and values below \"Configs\" - see example below where the \"batch.size\" is set to 10000.  \n\nBut be careful, you will not get an error message if you set an unknow property, so be sure to use right name of the property.\n```yaml\nDatabase:\n  Logger:\n    - Id: kafka1\n      Type: Kafka\n      Enabled: true\n      Servers: nuc1.rocworks.local:9092\n      Configs:  # you can set any valid Kafka producer property here\n        batch.size: 10000\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_SByte/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Byte/+\n```\n## 1.20 Cleanup and GraalVM Native Build\nGraalVM Native Image build is now possible, see directory \"native\". Removed various unused features and upgraded libraries to the latest versions. For the native build it was needed to replace SLF4J logging with the standard Java logging.  \n\nRemoved Features:  \n* Clustering  \n* NATS  \n* DDS\n* IoTDB: it leads to debug(!) log messages of other components, did not invest time to find out how to get rid of this behaviour.  \n* MQTTDriver: it uses Groovy and I could get it running natively compiled, so I had to remove it to get rid of Groovy  \n\nFixed bug: Topic data class now contains two separate fields \"path\" and \"node\". Before there was only one field \"address\". This also fixed a Bug when connecting/disconnecting of topics with wildcards.\n\n## 1.19 Neo4j Logger  \n!! **NOT AVAILABLE ANYMORE** !!  Separate Branch !!  \nAdded Neo4j as an option to log values from OPC UA to the graph database. Additionally the OPC UA node structure can also be replicated to the graph database. This will be done only once at the startup of the Automation Gateway.  \n```yaml\nDatabase:\n  Logger:\n    - Id: neo4j\n      Enabled: true\n      Type: Neo4j\n      Url: bolt://nuc1.rocworks.local:7687\n      Username: \"neo4j\"\n      Password: \"manager\"\n      Schemas:\n        - System: opc1  # Replicate node structure to the graph database\n          RootNodes:\n            - \"ns=2;s=Demo\"  # This node and everything below this node\n        - System: winccoa1  # Replicate the nodes starting from \"i=85\" (Objects) node\n      WriteParameters:\n        BlockSize: 1000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Float/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Double/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Int16/+\n        - Topic: opc/winccoa1/path/Objects/PUMP1/#\n        - Topic: opc/winccoa1/path/Objects/ExampleDP_Int/#\n\n\n```\n\n## 1.18.3 Added MQTT Websocket Option and simple Authentication  \nAdded the option to enable a Websocket listener for the MQTT server.\nThe Websocket listener is listening on the endpoint \"/mqtt\".\nExample MQTT Client Url: \"ws://your-host-or-ip/mqtt\"\n```yaml\nMqttServer:\n  Listeners:\n    - Id: Mqtt  # Tcp listener without authentication\n      Port: 1883\n\n    - Id: MqttWs  # Websocket listener with authentication\n      Port: 1884\n      Websocket: true  \n      Username: system  # If empty, then any username can be used\n      Password: manager  # If empty, then any or no password can be used\n```\n\n## 1.18.2 Raw value to engineering value conversion for PLC4X driver\n!! **NOT AVAILABLE ANYMORE** !!  \nIt is now possible to define Groovy functions to convert raw values to engineering values. Every incoming or outgoing value will be passed through the defined functions. The functions can currently only be defined at the connection level, so every value coming from this PLC connection goes through the conversion functions.  \n\n```yaml\nPlc4x:\n  Drivers:\n    - Id: \"niryo\"\n      Enabled: true\n      Url: \"modbus://192.168.1.9:5020\"\n      Polling:\n        Time: 100\n        Timeout: 90\n        OldNew: true\n      WriteTimeout: 100\n      ReadTimeout: 100\n      LogLevel: ALL\n      Value:\n        Reader: \u003e # Here is the function for incoming values\n          def x = value as int;\n          return x \u003e 32767 ? 32767 - x : x\n        Writer: \u003e # Here is the function for outgoing values\n          def x = value as int;\n          return x \u003c 0 ? 32767 - x : x\n```\nIf you need different functions for different addresses then you have to implement a switch/case statement. The functions have two arguments: \"address\" and \"value\". \n\nHere is an example where a value conversion is done only for some addresses:\n```yaml\n      Value:\n        Reader: \u003e\n          def x = value as int;\n          def xs = [\"input-register:1:UINT\", \"input-register:2:UINT\", \"input-register:2:UINT\"];\n          if (xs.contains(address)) {\n            println \"convert!\"\n            return x \u003e 32767 ? 32767 - x : x;\n          } else {\n            println(\"default\")\n            return x;\n          } \n```\n\n## 1.18.1 Features and fixes in PLC4X driver  \nAuto reconnect to the PLC if the connection is lost.  \nRead/Write/Poll only if the connection to the PLC is up.  \nFixed issue when writing a value fails (stopped the writing thread).  \nFixed issue in unsubscribe with polling option (polling item was not removed).  \nFixed issue when a client disconnects (unsubscribe with a list of topics failed).  \nAdded new configuration settings for the driver: timeout in ms for write, read, polling.  \n```yaml\nPlc4x:\n  Drivers:\n    - Id: \"machine1\"\n      Enabled: true\n      Url: \"modbus://127.0.0.1:502\"\n      Polling:\n        Time: 1000\n        Timeout: 900 # ms\n        OldNew: true\n      WriteTimeout: 100 # ms\n      ReadTimeout: 100 # ms\n      LogLevel: INFO\n````\n\n## 1.18 Removed Apache Ignite\nApache Ignite was removed due to its size and maintenance requirements. We don't know of anyone using the Ignite option, so we decided to remove it.  \n\n## 1.17 Added CrateDB as supported JDBC database for logging  \n[CrateDB](https://crate.io) is now also supported as JDBC database for logging. If the table (default name \"events\") does not exists, it will be created partitioned by the month of the source time with four shareds. But you can create the table manually in advance with the settings of your needs.\n```\nCREATE TABLE IF NOT EXISTS $sqlTableName (\n  \"sys\" TEXT, \n  \"nodeid\" TEXT,          \n  \"sourcetime\" TIMESTAMP WITH TIME ZONE,\n  \"servertime\" TIMESTAMP WITH TIME ZONE,\n  \"sourcetime_month\" TIMESTAMP WITH TIME ZONE GENERATED ALWAYS AS date_trunc('month', \"sourcetime\"),\n  \"numericvalue\" DOUBLE,\n  \"stringvalue\" TEXT,\n  \"status\" TEXT,\n  PRIMARY KEY (sourcetime_month, sourcetime, sys, nodeid)\n) CLUSTERED INTO 4 SHARDS PARTITIONED BY (\"sourcetime_month\");   \n```\n\n\n## 1.16 JDBC Logger to write field values to relational databases  \nAdded the option to log values to a JDBC compliant relational database. You have to add the JDBC driver to your classpath and set the appropriate JDBC URL path in the configuration file. PostgreSQL, MySQL and Microsoft SQL Server JDBC drivers are already included in the build.gradle file (see lib-jdbc/build.gradle) and also appropriate SQL statements are implemented for those relational databases. If you use other JDBC drivers you can add the driver to the lib-jdbc/build.gradle file as runtime only dependency and you may specify SQL statements for insert and select in the configuration file.\n\nYou can specify the table name in the config file with the option \"SqlTableName\", if you do not specify the table name then \"events\" will be used as default name.  \n\n\nSpecify JDBC drivers in the lib-jdbc/build.gradle file:  \n```\n    runtimeOnly group: 'org.postgresql', name: 'postgresql', version: 'x.x.x'\n    runtimeOnly group: 'mysql', name: 'mysql-connector-java', version: 'x.x.x'\n    runtimeOnly group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: 'x.x.x.jre11'\n```\n\nCreate a table with this structure. For PostgreSQL, MySQL and Microsoft SQL Server the table will be created on startup automatically.  \n```\n  CREATE TABLE IF NOT EXISTS public.events\n  (\n      sys character varying(30) NOT NULL,\n      nodeid character varying(30) NOT NULL,\n      sourcetime timestamp without time zone NOT NULL,\n      servertime timestamp without time zone NOT NULL,\n      numericvalue numeric,\n      stringvalue text,\n      status character varying(30) ,\n      CONSTRAINT pk_events PRIMARY KEY (system, nodeid, sourcetime)\n  )\n  TABLESPACE ts_scada;\n```\n\nConfiguration of JDBC database logger:  \n```yaml\nDatabase:\n  Logger:\n    - Id: postgres\n      Type: Jdbc\n      Enabled: true\n      Url: jdbc:postgresql://nuc1:5432/scada\n      Username: system\n      Password: manager\n      SqlTableName: events     \n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_SByte/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Byte/+\n```\n\nBecause the SQL dialect can be slightly different with other databases, you can specify the insert and select SQL statement in the config file:\n```yaml\nDatabase:\n  Logger:\n    - Id: other\n      Type: Jdbc\n      Enabled: true\n      Url: jdbc:other://nuc1:1111/scada\n      Username: system\n      Password: manager\n      SqlTableName: events     \n      SqlInsertStatement: \u003e \n        INSERT INTO events (sys, nodeid, sourcetime, servertime, numericvalue, stringvalue, status)\n         VALUES (?, ?, ?, ?, ?, ?, ?)\n         ON CONFLICT ON CONSTRAINT PK_EVENTS DO NOTHING  \n      SqlQueryStatement: \u003e\n        SELECT sourcetime, servertime, numericvalue, stringvalue, status\n         FROM events\n         WHERE sys = ? AND nodeid = ? AND sourcetime \u003e= ? AND sourcetime \u003c= ? \n```\n\n## 1.15 Nats Logger to write field values to a Nats server\n!! **NOT AVAILABLE ANYMORE** !!  \nAdded a [Nats](https://nats.io) Logger to write field values to a Nats server. It is like a database logger, but it writes the values to a configurable Nats server. Any values which get into Frankenstein (OPC UA, PLC4X, DDS, MQTT) by a Driver can be logged to a Nats server. The values are stored in JSON format.\n```yaml\nDatabase:\n  Logger:\n    - Id: nats1\n      Type: Nats\n      Enabled: true\n      Url: \"nats://nuc1:4222\"\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_SByte/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Byte/+\n````\n\n## 1.14 Fixes and optimizations\n* MQTT Driver: Caching of NodeIds in MQTT Driver (topic to NodeId cache).  \n* MQTT Server and Driver:  max message size can be set in config file (e.g. for Video streaming).\n* GraphQL Define one or more starting/root NodeIds for GraphQL OPC UA schema import.\n* GraphQL: Renamed \"NodeIds\" to \"RootNodes\" in Schema section (for OCP UA )\n* Renamed TopicValueDDS to TopicValueJson. \n\n\n## 1.13 MQTT Logger to write field values to a MQTT Broker\nAdded a **MQTT Logger** to write field values to a MQTT Broker. It is like a database logger, but it writes the values to a configurable MQTT Broker. Any values which get into Frankenstein (OPC UA, PLC4X, DDS, MQTT) by a Driver can be logged to a MQTT Broker. The values are stored in JSON format.\n\n```yaml\nDatabase:\n  Logger:\n    - Id: mqtt1\n      Type: Mqtt\n      Enabled: true\n      Host: 192.168.1.169\n      Port: 1883\n      Ssl: false\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/smarthome/path/Meter_Input/WattAct\n        - Topic: opc/smarthome/path/Meter_Output/WattAct\n        - Topic: opc/smarthome/path/PV/Spot/+\n```\n\n## 1.12 MQTT Driver with Groovy script transformer\n!! **NOT AVAILABLE ANYMORE** !!  \nAdded a inital version of a **MQTT Driver** to get values from a MQTT Broker into Frankenstein. A Groovy script can be used to transform the values to an OPC UA format, so that Frankenstein can be used to log those value to databases. Functionality is currently very limited, only subscribe is implemented.\n\nIn this example we transform values of a MQTT Broker from this format: {\"TimeMS\":1620327963328,\"Value\":10.277357833719135} to our internal TopicValueOpc format by using a Groovy script and then log some topic values to an InfluxDB.\n\n```yaml\nMqttClient:\n  - Id: \"mqtt1\"\n    Enabled: true\n    LogLevel: INFO\n    Host: 192.168.1.6\n    Port: 1883\n    Ssl: false\n    Value:\n      Format: JSON\n      Script: \u003e\n        return [ \n          className: \"TopicValueOpc\",\n          sourceTime: Instant.ofEpochMilli(source.TimeMS).toString(),\n          serverTime: Instant.now().toString(),\n          value: source.Value,\n          dataTypeId: 0,\n          statusCode: 0 ]\n\nDatabase:\n  Logger:\n    - Id: influx1\n      Type: InfluxDB\n      Enabled: true\n      Url: http://192.168.1.13:8086\n      Database: test\n      Username: \"\"\n      Password: \"\"\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: mqtt/mqtt1/path/Meter_Input/WattAct\n        - Topic: mqtt/mqtt1/path/Meter_Output/WattAct\n        - Topic: mqtt/mqtt1/path/PV/Spot/+          \n```\n\n## 1.11 Apache Kafka Database Logger\nAdded **Apache Kafka** as tag logger option, all incoming value changes of the configured topics will be published to an Apache Kafka Broker. How to can be found [here](https://www.rocworks.at/wordpress/?p=1076)\n```yaml\nDatabase:\n  Logger:\n    - Id: kafka1\n      Type: Kafka\n      Servers: server2:9092\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_SByte/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Byte/+\n```\n\nYou can also use KSQL to analyze the tag stream.\nConnect to a ksql cli:\n```\ndocker exec -ti ksqldb-cli ksql http://ksqldb-server:8088\n```\n\nCreate a stream for your logger. Each logger has its own Kafka-Topic. Topic name is equal to the source id of the tag (opc/**opc1**/...)\n```\nCREATE STREAM opc1(\n  browsePath VARCHAR KEY, \n  sourceTime VARCHAR, \n  value DOUBLE, \n  statusCode VARCHAR\n) WITH (\n  KEY_FORMAT='KAFKA',\n  KAFKA_TOPIC='opc1', \n  VALUE_FORMAT='JSON',\n  TIMESTAMP='sourceTime',TIMESTAMP_FORMAT='yyyy-MM-dd''T''HH:mm:ss[.n]X'\n);\n```\n\nExample of a simple quey:\n```\nSELECT node, COUNT(*) \nFROM opc1 \nWINDOW SESSION (10 SECONDS) \nWHERE node like '%_00' \nGROUP BY node \nEMIT CHANGES;\n```\n\n## 1.10 Apache IoTDB Database Logger\n!! **NOT AVAILABLE ANYMORE** !!  Separate Branch !!  \nAdded **Apache IoTDB** as tag logger option.\n```yaml\nDatabase:\n  Logger:\n   - Id: iotdb1\n      Type: IoTDB\n      Host: server2\n      Port: 6667\n      Database: root.scada1\n      Username: \"root\"\n      Password: \"root\"\n      WriteParameters:\n        QueueSize: 20000\n        BlockSize: 10000\n      Logging:\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_SByte/+\n        - Topic: opc/opc1/path/Objects/Demo/SimulationMass/SimulationMass_Byte/+\n```\n\n## 1.9 Apache Ignite as Cluster option and Ignite as Memory-Store\nAdded **Apache Ignite** as an option for clustering and also to use the Apache Ignite Distributed In Memory **Cache** for storing last and history values coming from OPC UA or other sources. With that enabled it is possible to do **SQL** queries on the process values. The cache node stores historical value changes for a defined timerange. So older values are purged on a regular basis (configurable in the configuration file). It is configurable which topics should be stored in the Apache Ignite Cache.\n\n```yaml\nCache:\n  - Id: \"global\"\n    Enabled: true\n    LogLevel: INFO\n    SqlIndexMaxInlineSize: 1000\n    StoreHistoryValues: true\n    Systems:\n      - SystemType: Opc\n        SystemName: \"opcua1\"\n        PurgeEverySeconds: 10\n        KeepLastSeconds: 180\n      - SystemType: Opc\n        SystemName: \"opcua2\"\n        PurgeEverySeconds: 10        \n        KeepLastSeconds: 180                                    \n    Logging:\n      - Topic: opc/opcua1/path/Objects/Demo/SimulationMass/SimulationMass_Double/+\n      - Topic: opc/opcua2/path/Objects/Demo/SimulationMass/SimulationMass_Double/+\n```\n\nUpdates on the tables are currently not possible. It is implemented that updates on the updatevalue columns should write the values to the source OPC UA server, but unfortunately it leads to unexpected behaviour of the Ignite Cluster.  \n\nThere is a command line query tool available in the Apache Ignite Distribution. But you can also download the JDBC driver or ODBC drive from the Apache Ignite Website and use any other JDBC/ODBC client tool.\n\u003e C:\\Tools\\apache-ignite-2.9.1-bin\\bin\\sqlline.bat -u jdbc:ignite:thin://192.168.1.18\n\nExample Queries:\n```\nselect * from global.opcnode where nodeid like '%Mass%'  \n\nselect systemname, count(*) \nfrom global.opcvalue\n group by systemname;   \n\nselect systemname, count(*), min(sourcetime), max(sourcetime) \nfrom global.opcvaluehistory \ngroup by systemname;  \n```\n\nThere are Docker examples available in the docker/examples directory.  \n\u003e C:\\Workspace\\automation-gateway\\source\u003e gradle build  \n\u003e C:\\Workspace\\automation-gateway\\docker\u003e build.bat  \n\u003e C:\\Workspace\\automation-gateway\\docker\\examples\\ignite\u003e docker compose up -d  \n\n## 1.8 Upgrade to VertX 4.0.3\nUpgraded to VertX 4.0.3 and splitted the value type to a base class with subclasses for Opc, Plc and DDS. The app names have been changed, the clustered apps are now named with \"cluster\". DDS values can now be logged to InfluxDB.  \n\n## 1.7 DDS Driver (subscribe and publish)\n!! **NOT AVAILABLE ANYMORE** !!  \nAdded a first version of DDS support. Currently only MQTT subscribe and publish to DDS topics are functional. It is the app-dds application, the app-gateway must also be up and running.\n\nYou need to install OpenDDS and build it with Java support. And you also have to compile your DDS IDL files with Java support. See the ReadMe.txt in the idl directory of app-dds.\n\nExample MQTT Topic: \n\u003e dds/system-id/path/topic-type-name/topic-name  \n\u003e dds/demo/path/shape/Circle  \n\u003e dds/demo/path/shape/Square  \n\nConfiguration\n```yaml\nDDS:\n  Domains:\n    - Id: \"demo\"\n      Enabled: true\n      LogLevel: ALL\n      DCPSConfigFile: rtps.ini\n      Domain: 0\n      TopicTypes:\n        - Id: \"shape\"\n          TopicTypeName: \"org.omg.dds.demo.ShapeType\"\n        - Id: \"message\"\n          TopicTypeName: \"org.omg.dds.demo.Message\"\n```\n\n## 1.6 Added GraphiQL (http://localhost:4000/graphiql/)\nAdded GraphiQL to the Gateway and optionally write the browsed schemas (OPC UA and generated GraphQL scheam) to files.\n```yaml\nGraphQLServer:\n  Listeners:\n    - Port: 4000\n      LogLevel: ALL\n      GraphiQL: true  # Enable GraphiQL\n      WriteSchemaToFile: false  # Write GraphQL Schema to a file\n      Schemas:\n        - System: ignition\n          FieldName: BrowseName # BrowseName | DisplayName\nOpcUaClient:\n  - Id: \"ignition\"\n    Enabled: true\n```\n\n## 1.5 OPC UA Schemas to GraphQL Schema Importer\nSupport multiple OPC UA schemas in GraphQL. Be sure that you have set `BrowseOnStartup: true` for the OPC UA servers which you want to embed in the GraphQL schema. Additionally it can be defined which OPC UA field should be taken as the GraphQL field name: it can be \"BrowseName\" or \"DisplayName\". But be careful, the DisplayName must not be unique below a node, so it can lead to an invalid schema. \n```yaml\nGraphQLServer:\n  Listeners:\n    - Port: 4000\n      LogLevel: ALL\n      Schemas:\n        - System: ignition\n          FieldName: BrowseName # BrowseName | DisplayName\nOpcUaClient:\n  - Id: \"ignition\"\n    Enabled: true\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvogler75%2Fautomation-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvogler75%2Fautomation-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvogler75%2Fautomation-gateway/lists"}