{"id":22753853,"url":"https://github.com/slaclab/k2eg","last_synced_at":"2025-04-13T00:43:58.879Z","repository":{"id":155619431,"uuid":"618974954","full_name":"slaclab/k2eg","owner":"slaclab","description":"Kafka To EPICS Gateway","archived":false,"fork":false,"pushed_at":"2025-04-10T00:06:47.000Z","size":789,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-13T00:43:52.006Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slaclab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-25T22:13:47.000Z","updated_at":"2025-03-31T19:41:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"2af86958-9f2a-4b34-9a0f-94229af94add","html_url":"https://github.com/slaclab/k2eg","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fk2eg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fk2eg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fk2eg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slaclab%2Fk2eg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slaclab","download_url":"https://codeload.github.com/slaclab/k2eg/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650419,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-11T06:13:44.201Z","updated_at":"2025-04-13T00:43:58.872Z","avatar_url":"https://github.com/slaclab.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka to Epics Gateway\nAuthor: Claudio Bisegni\n\nCompany: SLAC National Accelerator Laboratory\n\n[![Build k2eg with CMake](https://github.com/slaclab/k2eg/actions/workflows/build.yml/badge.svg)](https://github.com/slaclab/k2eg/actions/workflows/build.yml)\n[![Build k2eg Docker Image](https://github.com/slaclab/k2eg/actions/workflows/build-docker-image.yml/badge.svg)](https://github.com/slaclab/k2eg/actions/workflows/build-docker-image.yml)\n## Description\nA C++ implementation of a bidirectional gateway between Kafka and the [EPICS](https://epics.anl.gov) Control System that aims to serve as a central processing unit for specialized DAQ requirements involving advanced algorithms.\n\nIt receives commands from a specific Kafka topic and allows the injection of EPICS data towards other topics.\n\n## Implementation Status\nThe current implementation receives and interprets commands and uses only JSON serialization (described later). It lacks cluster management and other advanced features.\n\n```[tasklist]\n### Command Task List\n- [x] Get Command\n- [x] Monitor Command\n- [x] Snapshot Command\n- [X] Put Command\n    - Scalar\n    - ScalarArray\n\n### Functional Task List\n- [x] JSON Serialization\n- [X] MSGPack Binary serialization\n- [X] MSGPack compact serialization\n- [X] Multithreading EPICS Monitor\n- [ ] Cluster implementation\n```\nFor the serialization and message format see documentation [here](doc/message-format.md)\n## Application Architecture\n\u003cp\u003eThe application architecture logic is designed following the scheme below. The dotted line boxes represent elements not yet developed.\u003c/p\u003e\n\u003cp\u003e\nThere are two principal layers:\n\n* Controller:\n    * Node Controller\n    * Command Controller\n    * Cluster Controller (to design)\n* Service:\n    * Publisher and Subscriber:\n    * Data storage\n    * Log\n    * EPICS\n    * Cluster Services (to design)\n\u003c/p\u003e\n\n![K2EG Software Layer Interaction](doc/image/scheme.png)\n\n## Getting started\nThis project aims to realize an [EPICS](https://epics.anl.gov) gateway for interacting with EPICS IOCs using Kafka. It uses an input topic from a Kafka cluster to receive JSON encoded commands that allow IO operations on the IOCs.\n\n## Parameter\nk2eg relies upon boost program options to manage the startup option configuration. Below is the complete list of parameters:\n\n```console\nk2eg --help\nEpics k2eg:\n  --help                                Produce help information\n  --version                             Print the application version\n  --conf-file arg (=0)                  Specify if we need to load\n                                        configuration from file\n  --conf-file-name arg                  Specify the configuration file\n  --log-level arg (=info)               Specify the log level[trace, debug,\n                                        info, error, fatal]\n  --log-on-console                      Specify when the logger prints in\n                                        console\n  --log-on-file                         Specify when the logger prints in file\n  --log-file-name arg                   Specify the log file path\n  --log-file-max-size arg (=1)          Specify the maximum log file size in\n                                        mbyte\n  --log-on-syslog                       Specify when the logger prints in syslog\n                                        server\n  --syslog-server arg                   Specify syslog hostname\n  --syslog-port arg (=514)              Specify syslog server port\n  --cmd-input-topic arg                 Specify the message bus queue where\n                                        the k2eg receives the configuration\n                                        command\n  --cmd-max-fecth-element arg (=10)     The max number of commands fetched per\n                                        consume operation\n  --cmd-max-fecth-time-out arg (=250)   Specify the timeout for waiting for the\n                                        command in microseconds\n  --nc-monitor-expiration-timeout arg (=3600)\n                                        Specify the amount of time with no\n                                        consumer on a queue after which monitor\n                                        can be stopped\n  --nc-purge-queue-on-exp-timeout arg (=1)\n                                        Specify when a queue is purged when\n                                        the monitor that pushes data onto it\n                                        is stopped\n  --nc-monitor-consumer-filterout-regex arg\n                                        Specify regular expression to be used to\n                                        filter out consumer groups from those\n                                        used to calculate the number of active\n                                        consumers of the monitor queue\n  --pub-server-address arg              Publisher server address\n  --pub-impl-kv arg                     The key:value list for publisher\n                                        implementation driver\n  --sub-server-address arg              Subscriber server address\n  --sub-group-id arg (=k2eg-default-group)\n                                        Subscriber group id\n  --sub-impl-kv arg                     The key:value list for subscriber\n                                        implementation driver\n  --storage-path arg (=/opt/app/k2eg.sqlite)\n                                        The path where the storage files are\n                                        saved\n  --monitor-worker-cron-schedule arg    The cron string for configuring the\n                                        monitor checking scheduler\n  --scheduler-check-delay-seconds arg (=60)\n                                        The number of seconds for which the\n                                        scheduler thread is going to sleep\n  --scheduler-thread-number arg (=1)    The number of the scheduler workers\n  --metric-enable arg (=0)              Enable metric management\n  --metric-server-http-port arg (=8080) The port used for publishing the http\n                                        metric server\n```\n\nThere are two other different ways to configure the application other than the common command line option:\n* [configuration file](#configuration-file)\n* [environment variable](#environment-variable)\n\n### Configuration File\nThe use of the configuration file is activated using the command line option:\n```console\nk2eg --conf-file --conf-file-name \u003cpath/to/configuration/file\u003e\n```\nThe content of the file needs to follow the rules:\n```conf\n...\nlog-file-max-size=1234\nlog-on-syslog=true\nsyslog-server=syslog-server\nsyslog-port=5678\nsub-server-address=sub-address\nsub-group-id=sub-group-id\n...\n```\n\n### Environment variable\nThe use of the environment variable is automatically managed, each variable with the prefix *EPICS_k2eg_* is evaluated, for example enabling the use of the config file can be done via ENV variable with:\n``` console\nexport EPICS_k2eg_conf-file\nexport EPICS_k2eg_conf-file-name=\u003cpath/to/configuration/file\u003e\n```\n\n## Commands\n\nGeneraly each command return a reply to the client on a specifc destination where the client is readind data. Each client has his own kafka topic where it receive reply form k2eg. The general structure of the command with the base field is:\n``` json\n{\"error\":\"int32\",\"reply_id\":\"rep-id-snapshot\", \"message\":\"optional\", ...}\n```\n* `error`: Indicates the status of the command execution. A value of `0` means the command was successfully executed. `Negative` values indicate an error occurred, while `positive` values may represent specific types of messages as defined in the documentation for each command.\n* `reply_id`: A unique identifier sent by the client along with the command request. It is included in the reply to allow the client to match the response to the corresponding request. This is particularly useful because commands are processed asynchronously, and replies may not be received in the same order as the requests were sent. This mechanism enables clients to send multiple asynchronous requests and correctly associate each reply with its respective command.\n* `message`: Describes errors or other information sent by the broker as a reply to the client. This provides details about the execution status of the associated command. The presence of this field depends on the specific command's description.\n\n### Get Command\nThis implements the base caget|pvaget function of EPICS command. If possible, it will use a client from a monitor thread; otherwise, a new client is allocated to perform a get operation. The ***reply_id*** field is not mandatory and it is forwarded into the reply message.\n```json\n{\n    \"command\": \"get\",\n    \"serialization\": \"json|msgpack\",\n    \"pv_name\": \"(pva|ca)://PV_NAME\",\n    \"reply_topic\": \"reply-destination-topic\",\n    \"reply_id\": \"reply id\"\n}\n```\n\n### Put command\nPut command permits changing the value field of a PV. The value needs to be a string that will be automatically converted to the type of the PV. The put command can send a reply with the result of the put operation. In this case, the ***dest_topic*** field is mandatory and the ***reply_id*** field is not mandatory and it is forwarded into the reply message.\n```json\n{\n    \"command\": \"monitor\",\n    \"pv_name\": \"(pva|ca)://PV_NAME.attribute\",\n    \"value\": \"pv new value\",\n    \"reply_topic\": \"reply-destination-topic\",\n    \"reply_id\":\"reply id\"\n}\n```\n\n### Monitor Command\nThis implements the base camonitor|pvamonitor function of EPICS client, creating a monitor thread in the gateway that sends the received values over the destination topic. The field **monitor_destination_topic** should be used to specify the monitor event topic. If not specified, the event will be forwarded to the reply topic.\n\nMonitor Activation\n```json\n{\n    \"command\": \"monitor\",\n    \"serialization\": \"json|msgpack\",\n    \"pv_name\": \"(pva|ca)://PV_NAME\",\n    \"reply_topic\": \"reply-destination-topic\",\n    \"reply_id\":\"reply id\",\n    \"monitor_destination_topic\":\"alternate-destination-topic\"\n}\n```\n\nMulti Monitor Activation\n```json\n{\n    \"command\": \"multi-monitor\",\n    \"serialization\": \"json|msgpack\",\n    \"pv_name\": [\"(pva|ca)://PV_NAME\", ....],\n    \"reply_topic\": \"reply-destination-topic\",\n    \"reply_id\":\"reply id\",\n    \"monitor_destination_topic\":\"alternate-destination-topic\"\n}\n```\nNote: The Monitor deactivation is managed by k2eg itself or specific timeout when no other process is consuming data.\n\n### Snapshot Command\nThis implements a one time DAQ snapshot funcationality, the comman has this tempalte below:\n```JSON\n{\n    \"command\": \"snapshot\",\n    \"serialization\": \"json|msgpack\",\n    \"pv_name_list\": [\"(pva|ca)://PV_NAME\", ....],\n    \"reply_topic\": \"reply-destination-topic\",\n    \"reply_id\":\"reply id\",\n}\n```\nThis command allocates a monitor watcher for each PV. Currently, it uses a one-second time window to acquire the values of all specified PVs. The EPICS monitor ensures at least one event is received for each PV. If, at the end of the timeout, some PVs have not been received, a fallback `get` operation is performed to fetch the missing PV values, as last chance.\nThe data received for each PVs is asyncrhonously sent, by the broker,  to the client without waiting for the time window timeout. The client receive a completion message that inform it the the snapshot has been completed.\nThe JSON version of the reply is:\nOne messgae for each PV in the ```pv_name_list``` list:\n```json\n{\"error\":0,\"reply_id\":\"\u003creply id\u003e\",\"\u003cpv name\u003e\":{\"value\":0, ...}}\n```\non completion message\n```json\n{\"error\":1,\"reply_id\":\"rep-id-snapshot\"}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslaclab%2Fk2eg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslaclab%2Fk2eg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslaclab%2Fk2eg/lists"}