{"id":15493277,"url":"https://github.com/dcsolutions/kalinka","last_synced_at":"2025-04-22T19:49:20.144Z","repository":{"id":57719698,"uuid":"82536152","full_name":"dcsolutions/kalinka","owner":"dcsolutions","description":"Kafka-connectivity for MQTT-devices in both directions","archived":false,"fork":false,"pushed_at":"2019-04-26T12:42:39.000Z","size":432,"stargazers_count":3,"open_issues_count":3,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-10-19T11:10:28.430Z","etag":null,"topics":["activemq","amqp","java","jms","kafka","mqtt"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dcsolutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-20T08:49:19.000Z","updated_at":"2024-05-21T05:08:08.000Z","dependencies_parsed_at":"2022-09-13T12:50:57.344Z","dependency_job_id":null,"html_url":"https://github.com/dcsolutions/kalinka","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcsolutions%2Fkalinka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcsolutions%2Fkalinka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcsolutions%2Fkalinka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcsolutions%2Fkalinka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dcsolutions","download_url":"https://codeload.github.com/dcsolutions/kalinka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250313118,"owners_count":21410163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activemq","amqp","java","jms","kafka","mqtt"],"created_at":"2024-10-02T08:05:16.456Z","updated_at":"2025-04-22T19:49:20.121Z","avatar_url":"https://github.com/dcsolutions.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/dcsolutions/kalinka.svg?branch=master)](https://travis-ci.org/dcsolutions/kalinka)\n\n# What is it for?\n\nThere are several solutions to access a Kafka-cluster via MQTT, e.g. with [Kafka-Bridge](https://github.com/jacklund/mqttKafkaBridge).\nIn general these solutions work in a \"one-way\"-manner so that MQTT-clients may publish to Kafka but cannot consume from it.\n*Kalinka* tries to solve this issue. Since the reference-implementation uses ActiveMQ as MQTT-broker there is no restriction to MQTT\nso it should work with any supported protocol.\n\n# Status Quo\n\nThe project is in prototype-state right now and acts as a proof of concept.\n\n# Goals\n\n* No cluster of MQTT-Brokers. The system should scale well and as we found out in our tests \"normal\" broker-clusters are limited in this respect. Instead we want to use 1-n \"standalone\" broker-instances. As we will see, this decision implies some issues we had to deal with.\n* Kafka/Zookeeper is the only \"real\" clustered component.\n* No single-point-of-failure (of course)\n\n# Design\n\n*Kalinka* consists of 2 components:\n\n## kalinka-pub\n\nConsumes from MQTT and publishes the messages to Kafka.\n\nThere are 2 different possibilities to deploy *kalinka-pub*:\n\n* as a plugin inside ActiveMQ-Broker (recommended)\n* as a standalone service\n\nThe way, *kalinka-pub* works, is quite straightforward: If deployed as a standalone service it subscribes several MQTT-topics (more precisely, JMS-queues because we use ActiveMQ as JMS-broker which uses JMS internally). If it runs as an ActiveMQ-plugin it intercepts the broker's *send*-method. In both variants it publishes all received messages to a kafka-cluster. Since JMS-queues and kafka-topics behave in a different way, some kind of mapping is required. See chapter *Topic-Mapping* for details on how to deal with that.\n\nIn standalone-mode each *kalinka-pub*-instance may consume from several brokers and each broker should be consumed by multiple *kalinka-pub*-instances (no s.p.o.f.).\n\n## kalinka-sub\n\nSubscribes to Kafka-topics and forwards messages to ActiveMQ-broker \n\nIn contrast to *kalinka-pub* the situation is slightly different: As mentioned before the ActiveMQ-brokers do not act as one logical broker, they consist of several independent broker-instances. So it is crucial for the service to know, which of the available broker-instances the MQTT-client is connected to. This information is currently stored in Zookeeper. If it turns out that this does scale well, we'll have to find a better solution.\n\nAnother challenge is to limit the number of connections between hosts. If *n* instances of *kalinka-sub* had to send to *m* broker-instances this would result in n\\*m relations between *kalinka-sub*- and broker-instances. We are trying to solve this issue by leveraging Kafka's partitioning-mechanism in combination with an additional grouping-strategy. More details in chapter *2nd-level-partitioning*.\n\n\n# Topic-mapping\n\nT.B.D.\n\n# 2nd-level partitioning\n\nT.B.D.\n\n# Test-environment\n\n## General\n\nThis setup allows to test the different components in combination.\n**Note:** This is just a simple setup for testing and development. Security aspects are not considered so do not use this in a production-like environment without major changes!\n\nThe setup has been tested with Ubuntu 16.04 and Ubuntu 16.10 but it should work on any modern Linux-distribution. Feel free to test it on a Windows-OS but I cannot guarantee success.\n\nThe setup contains:\n\n* Zookeeper\n* Kafka\n* ActiveMQ\n* The MQTT-Kafka-Bridge-components\n\n## VM-setup\n\n### Installation of VirtualBox\n\n\u003chttps://www.virtualbox.org/wiki/Linux_Downloads\u003e\n\n### Installation of Vagrant\n\n\u003chttp://www.vagrantup.com/downloads\u003e\n\n### Start VM's\n\n```\ncd vagrant\nvagrant up\n```\n\nYou will end up with 3 virtual machines:\n\n* `192.168.33.20 (dev1)`\n* `192.168.33.21 (dev2)`\n* `192.168.33.22 (dev3)`\n\n### SSH-access\n\nThe test-setup uses an insecure SSH-keypair (`vagrant/ssh/insecure_rsa` and `vagrant/ssh/insecure_rsa.pub`). Never use this in a non-testing environment.\nSSH-user ist `root`.\n\nAfter you have started the VM's you can access them by entering:\n\n```\nssh root@192.168.33.20 -i ssh/insecure_rsa\nssh root@192.168.33.21 -i ssh/insecure_rsa\nssh root@192.168.33.22 -i ssh/insecure_rsa\n```\n\n## Deployment of components\n\nAll components are deployed as docker-containers. Docker will be installed on the VM's so you don't have to install Docker on your host machine.\n\n### Installation of Python and Virtualenv\n\n```\nsudo apt-get install python python-dev virtualenv build-essential libffi-dev libxml2-dev libxslt1-dev libssl-dev python-pip\n```\n\n### Creation of Virtualenv\n\nCreate your virtualenv in an arbitrary directory.\n\n```\nvirtualenv \u003cVENV_INSTALLATION_PATH\u003e\n```\n\n### Activate Virtualenv\n\nActivates Virtualenv for current shell.\n\n```\nsource \u003cVENV_INSTALLATION_PATH\u003e/bin/activate\n```\n\n### Install dependencies\n\n```\ncd ansible\npip install -r requirements.txt\n```\n\n### Deployment\n\nIt is recommended to deploy the basic infrastructure first and create a snapshot afterwards. So you can reset the environment very quickly.\n**Note:** All deployment-commands must be executed from inside folder *ansible*\n\n* Set up Docker-daemon, a Docker-registry and pull required images. These are the most time-consuming steps.\n```\nansible-playbook infrastruct.yml\n```\n\n* Create a snapshot\n```\ncd ../vagrant\nvagrant snapshot save \u003cNAME\u003e\n```\n\n* You can restore the snapshot by entering:\n```\nvagrant snapshot restore --no-provision \u003cNAME\u003e\n```\nSince we don't want to waste time, the flag `--no-provision` is recommended.\n\n* Now you'll have to decide if you want to run the latest snapshot or build the setup on your own\n* For further deployment-options take a look into the playbooks in folder *ansible*\n\n#### Run latest snapshot \n\n* Run the setup with *kalinka-pub* as seperate process/container (standalone)\n\n```\ncd ../ansible\nansible-playbook  -v -e \"reset_all=True kalinka_pub_plugin_enabled=False\" zookeeper.yml kafka.yml activemq.yml kalinka-pub.yml kalinka-sub.yml\n```\n\n* Run the setup with *kalinka-pub* embedded in activemq-broker\n\n```\ncd ../ansible\nansible-playbook  -v -e \"reset_all=True\" zookeeper.yml kafka.yml activemq.yml kalinka-sub.yml\n```\n\n#### Self-built\n\n* You have not the permission to push to `dcsolutions` so you'll have to create an account on \u003chttps://hub.docker.com\u003e and setup the following repositories first:\n  * kalinka-pub-service-example\n  * kalinka-sub-service-example\n* After creation of account you can build the whole project, including docker images by entering:\n```\ncd ..\ncd \u003cPROJECT_ROOT_DIR\u003e\nmvn clean install -DdockerImageBuild=true -DdockerRegistryPush=true -Ddocker.registry.prefix=\u003cNAME_OF_YOUR_DOCKERHUB_ACCOUNT\u003e\n```\n\nThe deployment is the nearly the same as described before but you'll have to refer to your own docker-account:\n\n* *kalinka-pub*-standalone\n```\nansible-playbook  -v -e \"reset_all=True kalinka_pub_plugin_enabled=False organization=\u003cNAME_OF_YOUR_DOCKER_ACCOUNT\u003e\" zookeeper.yml kafka.yml activemq.yml kalinka-pub.yml kalinka-sub.yml\n```\n\n* *kalinka-pub* as plugin\n```\nansible-playbook  -v -e \"reset_all=True organization=\u003cNAME_OF_YOUR_DOCKER_ACCOUNT\u003e\" zookeeper.yml kafka.yml activemq.yml kalinka-sub.yml\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcsolutions%2Fkalinka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdcsolutions%2Fkalinka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcsolutions%2Fkalinka/lists"}