{"id":42307973,"url":"https://github.com/ertis-research/kafka-ml","last_synced_at":"2026-02-28T04:00:49.985Z","repository":{"id":39128168,"uuid":"257248193","full_name":"ertis-research/kafka-ml","owner":"ertis-research","description":"Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)","archived":false,"fork":false,"pushed_at":"2024-09-27T07:33:33.000Z","size":5709,"stargazers_count":181,"open_issues_count":26,"forks_count":25,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-02-24T02:31:10.285Z","etag":null,"topics":["data-stream","deep-learning","docker","gpu-acceleration","iot","kafka","keras","keras-tensorflow","kubernetes","machine-learning","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ertis-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-20T10:36:57.000Z","updated_at":"2025-02-08T23:22:00.000Z","dependencies_parsed_at":"2024-09-18T15:58:34.837Z","dependency_job_id":"5fbdca81-305a-427c-a891-251907c0764b","html_url":"https://github.com/ertis-research/kafka-ml","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/ertis-research/kafka-ml","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ertis-research%2Fkafka-ml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ertis-research%2Fkafka-ml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ertis-research%2Fkafka-ml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ertis-research%2Fkafka-ml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ertis-research","download_url":"https://codeload.github.com/ertis-research/kafka-ml/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ertis-research%2Fkafka-ml/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29924719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-27T19:37:42.220Z","status":"online","status_checked_at":"2026-02-28T02:00:07.010Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-stream","deep-learning","docker","gpu-acceleration","iot","kafka","keras","keras-tensorflow","kubernetes","machine-learning","pytorch","tensorflow"],"created_at":"2026-01-27T11:12:46.288Z","updated_at":"2026-02-28T04:00:49.975Z","avatar_url":"https://github.com/ertis-research.png","language":"Python","funding_links":[],"categories":["Python","AI/ML Integration"],"sub_categories":["ML Pipelines"],"readme":"# Kafka-ML: connecting the data stream with ML/AI frameworks\n\nKafka-ML is a framework to manage the pipeline of Tensorflow/Keras and PyTorch\n(Ignite) machine learning (ML) models on Kubernetes. The pipeline allows the\ndesign, training, and inference of ML models. The training and inference\ndatasets for the ML models can be fed through Apache Kafka, thus they can be\ndirectly connected to data streams like the ones provided by the IoT.\n\nML models can be easily defined in the Web UI with no need for external\nlibraries and executions, providing an accessible tool for both experts and\nnon-experts on ML/AI.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"images/pipeline_.png\" height=\"300\"\u003e\n\u003c/p\u003e\n\nYou can find more information about Kafka-ML and its architecture in the\nopen-access publication below:\n\n\u003e _C. Martín, P. Langendoerfer, P. Zarrin, M. Díaz and B. Rubio \u003cbr\u003e \u003e\n\u003e **Kafka-ML: connecting the data stream with ML/AI frameworks** \u003cbr\u003e Future\n\u003e Generation Computer Systems, 2022, vol. 126, p. 15-33 \u003cbr\u003e \u003e\n\u003e [10.1016/j.future.2021.07.037](https://www.sciencedirect.com/science/article/pii/S0167739X21002995)_\n\nIf you wish to reuse Kafka-ML, please properly cite the above mentioned paper.\nBelow you can find a BibTex reference:\n\n```\n@article{martin2022kafka,\n  title={Kafka-ML: connecting the data stream with ML/AI frameworks},\n  author={Mart{\\'\\i}n, Cristian and Langendoerfer, Peter and Zarrin, Pouya Soltani and D{\\'\\i}az, Manuel and Rubio, Bartolom{\\'e}},\n  journal={Future Generation Computer Systems},\n  volume={126},\n  pages={15--33},\n  year={2022},\n  publisher={Elsevier}\n}\n```\n\nKafka-ML article has been selected as\n[Spring 2022 Editor’s Choice Paper at Future Generation Computer Systems](https://www.sciencedirect.com/journal/future-generation-computer-systems/about/editors-choice)!\n:blush: :book: :rocket:\n\n## Table of Contents\n\n- [Changelog](#changelog)\n- [Deploy Kafka-ML in a fast way](#Deploy-Kafka-ML-in-a-fast-way)\n  - [Requirements](#Requirements)\n  - [Steps to run Kafka-ML](#Steps-to-run-Kafka-ML)\n  - [Troubleshooting](#Troubleshooting)\n- [Usage](#usage)\n  - [Single models](#Single-models)\n  - [Distributed models](#Distributed-models)\n  - [Semi-supervised learning](#Semi-supervised-learning)\n  - [Incremental training](#Incremental-training)\n  - [Federated learning](#Federated-learning)\n- [Installation and development](#Installation-and-development)\n  - [Requirements to build locally](#Requirements-to-build-locally)\n  - [Steps to build Kafka-ML](#Steps-to-build-Kafka-ML)\n  - [GPU configuration](#GPU-configuration)\n- [Publications](#publications)\n- [License](#license)\n\n## Changelog\n\n- [29/04/2021] Integration of distributed models.\n- [05/11/2021] Automation of data types and reshapes for the training module.\n- [20/01/2022] Added GPU support. ML Code has been taken out of backend.\n- [04/03/2022] Added PyTorch ML Framework support!\n- [08/04/2022] Added support for learning curves visualization, confusion matrix\n  generation and small changes on metrics visualization. Now datasets can be\n  splitted into training, validation and test.\n- [26/05/2022] Included support for visualization of prediction data. Now you\n  can easily prototype and visualize your ML/AI application. You can train\n  models, deploy them for inference, and visualize your prediction data just\n  with data streams.\n- [14/07/2022] Added incremental training support and configuration of training\n  parameters for the deployment of distributed models.\n- [02/09/2022] Added real-time display of training parameters.\n- [26/12/2022] Added indefinite incremental training support.\n- [07/07/2023] Added federated training support (currently only for Tensorflow/Keras models).\n- [28/09/2023] Federated learning enabled for distributed neural networks and incremental training.\n- [05/07/2024] Added semi-supervised learning support.\n\n## Deploy Kafka-ML in a fast way\n\n### Requirements\n\n- [Docker](https://www.docker.com/)\n- [kubernetes\u003e=v1.15.5](https://kubernetes.io/)\n\n### Steps to run Kafka-ML\n\nFor a basic local installation, we recommend using Docker Desktop with\nKubernetes enabled. Please follow the installation guide on\n[Docker's website](https://docs.docker.com/desktop/). To enable Kubernetes,\nrefer to\n[Enable Kubernetes](https://docs.docker.com/desktop/kubernetes/#enable-kubernetes)\n\nOnce Kubernetes is running, open a terminal and run the following command:\n\n```sh\n# Uncomment only if you are running Kafka-ML on Apple Silicon\n# export DOCKER_DEFAULT_PLATFORM=linux/amd64\nkubectl apply -k \"github.com/ertis-research/kafka-ml/kustomize/local?ref=v1.2\"\n```\n\nThis will install all the required components of Kafka-ML, plus Kafka on the\nnamespace `kafkaml`. The UI will be available at http://localhost/ . You can\ncontinue with the [Usage](#usage) section to see how you can use Kafka-ML!\n\nFor a more advanced installation on Kubernetes, please refer to the\n[kustomization guide](kustomize/README.md)\n\n## Usage\n\nTo follow this tutorial, please deploy Kafka-ML as indicated in\n[Deploy Kafka-ML in a fast way](#Deploy-Kafka-ML-in-a-fast-way) or\n[Installation and development](#Installation-and-development).\n\n### Single models\n\n#### 1. Define a ML/AI model in Kafka-ML\n\nCreate a model in the Models tab with just a TF/Keras model source code and some\nimports/functions if needed. Maybe this model for the MNIST dataset is a simple\nway to start:\n\n```py\nmodel = tf.keras.models.Sequential([\n  tf.keras.layers.Flatten(input_shape=(28, 28)),\n  tf.keras.layers.Dense(128, activation='relu'),\n  tf.keras.layers.Dense(10, activation='softmax')\n])\nmodel.compile(\n    optimizer=tf.keras.optimizers.Adam(0.001),\n    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],\n)\n```\n\nSomething similar should be done in case you wish to use PyTorch:\n\n```py\nclass NeuralNetwork(nn.Module):\n    def __init__(self):\n        super(NeuralNetwork, self).__init__()\n        self.flatten = nn.Flatten()\n        self.linear_relu_stack = nn.Sequential(\n            nn.Linear(28*28, 128),\n            nn.ReLU(),\n            nn.Linear(128, 10),\n            nn.Softmax()\n        )\n\n    def forward(self, x):\n        x = self.flatten(x)\n        logits = self.linear_relu_stack(x)\n        return logits\n\n    def loss_fn(self):\n        return nn.CrossEntropyLoss()\n\n    def optimizer(self):\n        return torch.optim.Adam(model.parameters(), lr=0.001)\n\n    def metrics(self):\n        val_metrics = {\n            \"accuracy\": Accuracy(),\n            \"loss\": Loss(self.loss_fn())\n         }\n        return val_metrics\n\nmodel = NeuralNetwork()\n```\n\nNote that functions 'loss_fn', 'optimizer', and 'metrics' must necessarily be\ndefined.\n\nInsert the ML code into the Kafka-ML UI.\n\n\u003cimg src=\"images/create-model-tensorflow.svg\" width=\"500\"\u003e\n\n\u003cbr/\u003e\n\n\u003cimg src=\"images/create-model-pytorch.svg\" width=\"500\"\u003e\n\n#### 2. Define a configuration\n\nA configuration is a set of models that can be grouped for training. This can be\nuseful when you want to evaluate and compare the metrics (e.g, loss and\naccuracy) of a set of models or just to define a group of them that can be\ntrained with the same data stream in parallel. A configuration can also contain\na single ML model.\n\n\u003cimg src=\"images/create-configuration.svg\" width=\"500\"\u003e\n\n#### 3. Deploy a configuration of models for training\n\n\u003cimg src=\"images/deploy-configuration.png\" width=\"500\"\u003e\n\nChange the batch size, training and validation parameters in the Deployment\nform. Use the same format and parameters than TensorFlow methods _fit_ and\n_evaluate_ respectively. Validation parameters are optional (they are only used\nif _validation_rate\u003e0 or test_rate\u003e0_ in the stream data received).\n\nNote: If you do not have the GPU(s) properly tuned, **set the \"GPU Memory usage\nestimation\" parameter to 0**. Otherwise, the training component will be\ndeployed, but in a pending state waiting to allocate GPU memory. If the pod is\ndescribed, it will show a `aliyun.com/gpu-mem` related warning. If you wish, you\ncan mark the last field for the creation of the confusion matrix at the end of\nthe training.\n\n\u003cimg src=\"images/configure-deployment.svg\" width=\"500\"\u003e\n\nOnce the configuration is deployed, you will see one training result per model\nin the configuration. Models are now ready to be trained and receive stream\ndata.\n\n![](./images/training-results.png)\n\n#### 4. Stream data for model training\n\nNow, it is time to ingest the model(s) with your data stream for training and\nmaybe evaluation.\n\nIf you have used the MNIST model you can use the example\n`mnist_dataset_training_example.py`. You only need to configure the\n_deployment_id_ attribute to the one generated in Kafka-ML, maybe it is still 1.\nThis is the way to match data streams with configurations and models during\ntraining. You may need to install the Python libraries listed in\ndatasources/requirements.txt.\n\nIf so, please execute the MNIST example for training:\n\n```\npython examples/MNIST_RAW_format/mnist_dataset_training_example.py\n```\n\nYou can use your own example using the AvroSink (for Apache Avro types) and\nRawSink (for simple types) sink libraries to send training and evaluation data\nto Kafka. Remember, you always have to configure the _deployment_id_ attribute\nto the one generated in Kafka-ML.\n\n#### 5. Model metrics visualization\n\nOnce sent the data stream, and deployed and trained the models, you will see the\nmodels metrics and results in Kafka-ML. You can download now the trained models,\nor just continue the ML pipeline to deploy a model for inference.\n\n![](./images/training-metrics.svg)\n\nIf you wish to visualise the generated confusion matrix (in case it has been\nindicated) or to visualise some training and validation metrics (if any) per\nepoch, you can access for each training result to the following view.\n\n![](./images/plot-view.svg)\n\nIn addition, from this view you can access to this data in a more generic way in\nJSON, allowing you to generate new plots and other information for your reports.\n\n#### 6. Deploy a trained model for inference\n\nWhen deploying a model for inference, the parameters for the input data stream\nwill be automatically configured based on previous data streams received, you\nmight also change this. Mostly you will have to configure the number of replicas\nyou want to deploy for inference and the Kafka topics for input data (values to\npredict) and output data (predictions).\n\nNote: If you do not have the GPU(s) properly tuned, **set the \"GPU Memory usage\nestimation\" parameter to 0**. Otherwise, the inference component will be\ndeployed, but in a pending state waiting to allocate GPU memory. If the pod is\ndescribed, it will show a `aliyun.com/gpu-mem` related warning.\n\n\u003cimg src=\"images/deploy-inference.svg\" width=\"500\"\u003e\n\n#### 7. Stream data for inference\n\nFinally, test the inference deployed using the MNIST example for inference in\nthe topics deployed:\n\n```\npython examples/MNIST_RAW_format/mnist_dataset_inference_example.py\n```\n\n#### 8. Prediction (classification or regression) visualization\n\nIn the visualization tab, you can easily visualize your deployed models. First\nthing, you need to configure how your model prediction data will be visualized.\nHere is the example for the MNIST dataset:\n\n```json\n{\n  \"average_updated\": false,\n  \"average_window\": 10000,\n  \"type\": \"classification\",\n  \"labels\": [\n    {\n      \"id\": 0,\n      \"color\": \"#fff100\",\n      \"label\": \"Zero\"\n    },\n    {\n      \"id\": 1,\n      \"color\": \"#ff8c00\",\n      \"label\": \"One\"\n    },\n    {\n      \"id\": 2,\n      \"color\": \"#e81123\",\n      \"label\": \"Two\"\n    },\n    {\n      \"id\": 3,\n      \"color\": \"#ec008c\",\n      \"label\": \"Three\"\n    },\n    {\n      \"id\": 4,\n      \"color\": \"#68217a\",\n      \"label\": \"Four\"\n    },\n    {\n      \"id\": 5,\n      \"color\": \"#00188f\",\n      \"label\": \"Five\"\n    },\n    {\n      \"id\": 6,\n      \"color\": \"#00bcf2\",\n      \"label\": \"Six\"\n    },\n    {\n      \"id\": 7,\n      \"color\": \"#00b294\",\n      \"label\": \"Seven\"\n    },\n    {\n      \"id\": 8,\n      \"color\": \"#009e49\",\n      \"label\": \"Eight\"\n    },\n    {\n      \"id\": 9,\n      \"color\": \"#bad80a\",\n      \"label\": \"Nine\"\n    }\n  ]\n}\n```\n\nYou can specify the two types of visualization: 'regression' and\n'classification'. In classification mode, 'average_update' determines if you\nwant to have the current status displayed based on the higher average status,\nand 'average_window' determines the windows for calculating the average.\n\nFor each output of your model, you have to define a label. 'id' represents the\nposition of the param in the model output (e.g., suppose you have a temperature\noutput as the second parameter of your model), and with 'color' and 'label' you\ncan set a color and label to display for the param.\n\nOnce you set the configuration, you must also set the output topic where the\nmodel is deployed, 'mnist-out' in our last example. After this, visualization\ndisplays your data.\n\nHere is an example in classification mode:\n\n\u003cimg src=\"images/classification.png\"\u003e\n\nAnd in regression mode:\n\n\u003cimg src=\"images/regression.png\"\u003e\n\n### Distributed models\n\n#### 1. Define a ML/AI distributed model in Kafka-ML\n\nCreate a distributed model with just a TF/Keras model source code and some\nimports/functions if needed. Maybe this distributed model consisting of three\nsub-models for the MNIST dataset is a simple way to start:\n\n```py\nedge_input = keras.Input(shape=(28,28,1), name='input_img')\nx = tf.keras.layers.Conv2D(28, kernel_size=(3,3), name='conv2d')(edge_input)\nx = tf.keras.layers.MaxPooling2D(pool_size=(2,2), name='maxpooling')(x)\nx = tf.keras.layers.Flatten(name='flatten')(x)\noutput_to_fog = tf.keras.layers.Dense(64, activation=tf.nn.relu, name='output_to_fog')(x)\nedge_output = tf.keras.layers.Dense(10, activation=tf.nn.softmax, name='edge_output')(output_to_fog)\nedge_model = keras.Model(inputs=[edge_input], outputs=[output_to_fog, edge_output], name='edge_model')\n\nfog_input = keras.Input(shape=64, name='fog_input')\noutput_to_cloud = tf.keras.layers.Dense(64, activation=tf.nn.relu, name='output_to_cloud')(fog_input)\nfog_output = tf.keras.layers.Dense(10, activation=tf.nn.softmax, name='fog_output')(output_to_cloud)\nfog_model = keras.Model(inputs=[fog_input], outputs=[output_to_cloud, fog_output], name='fog_model')\n\ncloud_input = keras.Input(shape=64, name='cloud_input')\nx = tf.keras.layers.Dense(64, activation=tf.nn.relu, name='relu1')(cloud_input)\nx = tf.keras.layers.Dense(128, activation=tf.nn.relu, name='relu2')(x)\nx = tf.keras.layers.Dropout(0.2)(x)\ncloud_output = tf.keras.layers.Dense(10, activation=tf.nn.softmax, name='cloud_output')(x)\ncloud_model = keras.Model(inputs=cloud_input, outputs=[cloud_output], name='cloud_model')\n```\n\nInsert the ML code of each sub-model into the Kafka-ML UI separately. You will\nhave to specify the hierarchical relationships between the sub-models through\nthe \"Upper model\" field of the form (before you will have to check the\ndistributed box). In the example case proposed it has to be defined the\nfollowing relationships: the upper model of the Edge sub-model is the Fog and\nthe upper model of the Fog sub-model is the Cloud (Cloud sub-model is placed at\nthe top of the distributed chain so it does not have any upper model).\n\n\u003cimg src=\"images/create-distributed-model.png\" width=\"500\"\u003e\n\n#### 2. Define a configuration\n\nKafka-ML will only show those sub-models which are on the top of the distributed\nchain. Choosing one of them will add its corresponding full distributed model to\nthe configuration.\n\n\u003cimg src=\"images/create-distributed-configuration.png\" width=\"500\"\u003e\n\n#### 3. Deploy a configuration of distributed models for training\n\nDeploy the configuration of distributed sub-models in Kubernetes for training.\n\n\u003cimg src=\"images/deploy-distributed-configuration.png\" width=\"500\"\u003e\n\nChange the optimizer, learning rate, loss function, metrics, batch size,\ntraining and validation parameters in the Deployment form. Use the same format\nand parameters than TensorFlow methods _fit_ and _evaluate_ respectively.\nOptimizer, learning rate, loss function and metrics parameters are optional, so\nif not specified, default values are taken, which are: _adam_, _0.001_,\n_sparse_categorical_crossentropy_ and _sparse_categorical_accuracy_,\nrespectively. Validation parameters are also optional (they are only used if\n_validation_rate\u003e0 or test_rate\u003e0_ in the stream data received).\n\n\u003cimg src=\"images/configure-distributed-deployment.png\" width=\"500\"\u003e\n\nOnce the configuration is deployed, you will see one training result per\nsub-model in the configuration. Full distributed model is now ready to be\ntrained and receive stream data.\n\n![](./images/distributed-training-results.png)\n\n#### 4. Stream data for model training\n\nNow, it is time to ingest the distributed model with your data stream for\ntraining and maybe evaluation.\n\nIf you have used the MNIST distributed model you can use the example\n`mnist_dataset_training_example.py`. You only need to configure the\n_deployment_id_ attribute to the one generated in Kafka-ML, maybe it is still 1.\nThis is the way to match data streams with configurations and models during\ntraining. You may need to install the Python libraries listed in\ndatasources/requirements.txt.\n\nIf so, please execute the MNIST example for training:\n\n```\npython examples/MNIST_RAW_format/mnist_dataset_training_example.py\n```\n\n#### 5. Model metrics visualization\n\nOnce sent the data stream, and deployed and trained the full distributed model,\nyou will see the sub-models metrics and results in Kafka-ML. You can download\nnow the trained sub-models, or just continue the ML pipeline to deploy a model\nfor inference.\n\n![](./images/distributed-training-metrics.png)\n\n#### 6. Deploy a trained sub-model for inference\n\nWhen deploying a sub-model for inference, the parameters for the input data\nstream will be automatically configured based on previous data streams received,\nyou might also change this. Mostly you will have to configure the number of\nreplicas you want to deploy for inference and the Kafka topics for input data\n(values to predict) and output data (predictions). Lastly, in case you are\ndeploying a sub-model for inference which is not the last one in the distributed\nchain, you will also have to specify one more topic for upper data (partial\npredictions) and a limit number (between 0 and 1). These two fields work as\nfollows: on the one hand, if your deployed inference gets lower predictions\nvalues than the limit it will send partial predictions to its upper model using\nthe upper data topic in order to continue the data processing there; on the\nother hand, if your deployed inference gets higher predictions values than the\nlimit it will send these final results to the output topic.\n\n\u003cimg src=\"images/distributed-deploy-inference.png\" width=\"500\"\u003e\n\n#### 7. Stream data for inference\n\nFinally, test the inference deployed using the MNIST example for inference in\nthe topics deployed:\n\n```\npython examples/MNIST_RAW_format/mnist_dataset_inference_example.py\n```\n\n### Semi-supervised learning\n\nSemi-supervised learning is a type of machine learning that falls between supervised\nand unsupervised learning. In supervised learning, the model is trained on a labeled\ndataset, where each example is associated with a correct output or label. In unsupervised\nlearning, the model is trained on an unlabeled dataset, and it must learn to identify\npatterns or structure in the data without any explicit guidance. Semi-supervised learning,\non the other hand, involves training a machine learning model on a dataset that contains\nboth labeled and unlabeled examples. The idea behind semi-supervised learning is to use\nthe small amount of labeled data to guide the learning process, while also leveraging\nthe much larger amount of unlabeled data to improve the model's performance.\n\nCurrently, the only framework that supports semi-supervised training is TensorFlow.\nIn this case, the usage example will be the same as the one presented for the\nsingle models, only the configuration deployment form will change and will now\ncontain more fields.\n\nAs before, change the fields as desired. The new semi-supervised fields are:\nunsupervised_rounds and confidence. Unsupervised rounds are used to define the number\nof rounds to iterate through the so far unlabelled data. Confidence is used to specify\nthe minimum reliance that the model has to have in a prediction of an unlabelled data\nin order to subsequently assign that label to it. They are not required, so if not specified,\ndefault values are taken, which are: _5_ and _0.9_, respectively.\n\n\u003cimg src=\"images/deploy-unsupervised-configuration.png\" width=\"500\"\u003e\n\nOnce the configuration is deployed, you will see one training result per model\nin the configuration. Models are now ready to be trained and receive stream\ndata. Now, it is time to ingest the model(s) with your data stream for training.\n\nIf you have used the MNIST model you can use the example\n`mnist_dataset_unsupervised_training_example.py`. You may need to install the Python\nlibraries listed in datasources/requirements.txt.\n\nIf so, please execute the incremental MNIST example for training:\n\n```\npython examples/MNIST_RAW_format/mnist_dataset_unsupervised_training_example.py\n```\n\n### Incremental training\n\nIncremental training is a machine learning method in which input data is\ncontinuously used to extend the existing model's knowledge i.e. to further train\nthe model. It represents a dynamic learning technique that can be applied when\ntraining data becomes available gradually over time or its size is out of system\nmemory limits.\n\nCurrently, the only framework that supports incremental training is TensorFlow.\nIn this case, the usage example will be the same as the one presented for the\nsingle models, only the configuration deployment form will change and will now\ncontain more fields.\n\nAs before, change the fields as desired. For this case, there are two types of\ndeployments: time-limited and indefinite. The new time-limited incremental field\nis: stream timeout. The stream timeout parameter is used to configure the duration\nfor which the dataset will block for new messages before timing out. It is not\nrequired, so if not specified, default value is taken, which is: _60000_.\n\n\u003cimg src=\"images/deploy-incremental-configuration-1.png\" width=\"500\"\u003e\n\nThe new indefinite incremental fields are: monitoring metric, direction, and improvement.\nThe monitoring metric is used to keep track of a specific parameter (of the user's choice)\nwithin the validation phase of the model's training. The direction is used to let Kafka-ML\nknow in which direction the monitoring metric is improving (as it is configurable).\nFinally, the improvement serves to establish a range from which an automatic deployment of\nthe model for inference should be carried out, since this training is indefinite in time.\nMonitoring metric and direction must be specified. Improvement is not required, so if not\nspecified, default value is taken, which is: _0.05_.\n\n\u003cimg src=\"images/deploy-incremental-configuration-2.png\" width=\"500\"\u003e\n\nOnce the configuration is deployed, you will see one training result per model\nin the configuration. Models are now ready to be trained and receive stream\ndata. Now, it is time to ingest the model(s) with your data stream for training.\n\nIf you have used the MNIST model you can use the example\n`mnist_dataset_online_training_example.py`. You may need to install the Python\nlibraries listed in datasources/requirements.txt.\n\nIf so, please execute the incremental MNIST example for training:\n\n```\npython examples/MNIST_RAW_format/mnist_dataset_online_training_example.py\n```\n\n### Federated learning\n\nFederated learning is a privacy-preserving machine learning approach that enables\ncollaborative model training across multiple decentralized devices without the\nneed to transfer sensitive data to a central location. In federated learning,\ninstead of sending raw data to a central server for training, local devices\nperform the training on their own data and only share model updates or gradients\nwith the central server. These updates are then aggregated to create an improved\nglobal model, which is sent back to the devices for further training. \nThis distributed learning approach allows for the benefits of collective intelligence\nwhile ensuring data privacy and reducing the need for large-scale data transfers.\nFederated learning has gained popularity in scenarios where data is sensitive or\nresides in diverse locations, such as mobile devices, healthcare systems, and IoT networks.\n\nCurrently, the only framework that supports federated learning is TensorFlow.\nIn this case, the usage example will be the same as the one presented for the\nsingle models, only the configuration deployment form will change and will now\ncontain more fields.\n\nAs before, change the fields as desired. The new incremental fields are: aggregation_rounds,\nminimun_data, data_restriction and aggregation strategy. The aggregation_rounds parameter\nis used to configure the number of rounds that the model will be aggregated (an aggregation\nround is a round in which the model is trained with the data of the devices and then aggregated\nwith the other models). The minimun_data parameter is the minimum number of data that a device\nmust have to be able to participate in the training. The data_restriction parameter is the\ndata pattern (such as input shape, labels, etc.) that the data must have to be able to participate.\nFinally, the aggregation strategy parameter is the strategy that will be used to aggregate the\nmodels. Currently, the only strategy available is the average strategy, which consists of averaging\nthe weights of the models.\n\n\u003cimg src=\"images/deploy-federated-configuration.png\" width=\"500\"\u003e\n\nOnce the configuration is deployed, you will see one training result per model\nin the configuration. Models are now ready to be aggregated and they are sent\nto the devices for training. Now, if the devices have data that meets the\nrequirements, they will train the model and send the weights to the server for\naggregation. Once the aggregation is finished, the new model will be sent to\nthe devices for training again. This process will be repeated until the number\nof rounds specified in the configuration is reached.\n\nIf you have used the MNIST model you can use the example\n`mnist_dataset_federated_training_example.py`. You only need to configure the\n_deployment_id_ attribute to the one generated in Kafka-ML, maybe it is still 1.\nThis is the way to match data streams with configurations and models during\ntraining. You may need to install the Python libraries listed in\ndatasources/requirements.txt.\n\nIf so, please execute the incremental MNIST example for training:\n\n```\npython examples/FEDERATED_MNIST_RAW_format/mnist_dataset_federated_training_example.py\n```\n\n## Installation and development\n\n### Requirements to build locally\n\n- [Python supported by Tensorflow 3.5–3.7 and PyTorch 1.10](https://www.python.org/)\n- [Node.js](https://nodejs.org/)\n- [Docker](https://www.docker.com/)\n- [kubernetes\u003e=v1.15.5](https://kubernetes.io/)\n\n### Steps to build Kafka-ML\n\nIn this repository you can find files to build Kafka-ML in case you want to\ncontribute.\n\nIn case you want to build Kafka-ML in a fast way, you should set the variable\n`LOCAL_BUILD` to `true` in build scripts and modify the deployments files to use\nthe local images. Once that is done, you can run the build scripts.\n\nBy default, Kafka-ML will be built using CPU-only images. If you desire to build\nKafka-ML with images enabled for GPU acceleration, the `Dockerfile` and\n`requirements.txt` files of `mlcode_executor`, `model_inference` and\n`model_training` modules must be modified as indicated in those files.\n\nIn case you want to build Kafka-ML step-by-step, then follow the following\nsteps:\n\n1. You may need to deploy a local register to upload your Docker images. You can\n   deploy it in the port 5000:\n\n   ```bash\n   docker run -d -p 5000:5000 --restart=always --name registry registry:2\n   ```\n\n2. Build the backend and push the image into the local register:\n\n   ```bash\n   cd backend\n   docker build --tag localhost:5000/backend .\n   docker push localhost:5000/backend\n   ```\n\n3. Build ML Code Executors and push images into the local register:\n\n   3.1. Build the TensorFlow Code Executor and push the image into the local\n   register:\n\n   ```bash\n   cd mlcode_executor/tfexecutor\n   docker build --tag localhost:5000/tfexecutor .\n   docker push localhost:5000/tfexecutor\n   ```\n\n   3.2. Build the PyTorch Code Executor and push the image into the local\n   register:\n\n   ```bash\n   cd mlcode_executor/pthexecutor\n   docker build --tag localhost:5000/pthexecutor .\n   docker push localhost:5000/pthexecutor\n   ```\n\n4. Build the model_training components and push the images into the local\n   register:\n\n   ```bash\n   cd model_training/tensorflow\n   docker build --tag localhost:5000/tensorflow_model_training .\n   docker push localhost:5000/tensorflow_model_training\n\n   cd ../pytorch\n   docker build --tag localhost:5000/pytorch_model_training .\n   docker push localhost:5000/pytorch_model_training\n   ```\n\n5. Build the kafka_control_logger component and push the image into the local\n   register:\n\n   ```bash\n   cd kafka_control_logger\n   docker build --tag localhost:5000/kafka_control_logger .\n   docker push localhost:5000/kafka_control_logger\n   ```\n\n6. Build the model_inference component and push the image into the local\n   register:\n\n   ```bash\n   cd model_inference/tensorflow\n   docker build --tag localhost:5000/tensorflow_model_inference .\n   docker push localhost:5000/tensorflow_model_inference\n\n   cd ../pytorch\n   docker build --tag localhost:5000/pytorch_model_inference .\n   docker push localhost:5000/pytorch_model_inference\n   ```\n\n7. Install the libraries and execute the frontend:\n   ```bash\n   cd frontend\n   npm install # nvm install 10 \u0026 nvm use 10.24.1\n   npm i -g @angular/cli@9.1.15\n   ng build -c production\n   docker build --tag localhost:5000/frontend .\n   docker push localhost:5000/frontend\n   ```\n\n### Deploying Kafka-ML in a single node Kubernetes cluster (e.g., minikube, Docker desktop)\n\nOnce built the images, you can deploy the system components in Kubernetes\nfollowing this order:\n\n    kubectl apply -f zookeeper-pod.yaml\n    kubectl apply -f zookeeper-service.yaml\n\n    kubectl apply -f kafka-pod.yaml\n    kubectl apply -f kafka-service.yaml\n\n    kubectl apply -f backend-deployment.yaml\n    kubectl apply -f backend-service.yaml\n\n    kubectl apply -f frontend-deployment.yaml\n    kubectl apply -f frontend-service.yaml\n\n    kubectl apply -f tf-executor-deployment.yaml\n    kubectl apply -f tf-executor-service.yaml\n\n    kubectl apply -f pth-executor-deployment.yaml\n    kubectl apply -f pth-executor-service.yaml\n\n    kubectl apply -f kafka-control-logger-deployment.yaml\n\nFinally, you will be able to access the Kafka-ML Web UI: http://localhost/\n\n### Deploying Kafka-ML in a distributed Kubernetes cluster\n\n#### Configuring the back-end\n\nThe first thing to keep in mind is that the images we compiled earlier were\nintended for a single node cluster (localhost) and will not be able to be\ndownloaded from a distributed Kubernetes cluster. Therefore, assuming that we\nare going to upload them into a registry as before and on a node with IP\nx.x.x.x.x, we would have to do the same for all the images as for the following\nbackend example:\n\n```bash\ncd backend\ndocker build --tag x.x.x.x:5000/backend .\ndocker push x.x.x.x:5000/backend\n```\n\nNow, we have to update the location of these images (tr) in the\n`backend-deployment.yaml` file:\n\n```yaml\n containers:\n -   - image: localhost:5000/backend\n +   - image: x.x.x.x:5000/backend\n\n    - name: BOOTSTRAP_SERVERS\n      value: kafka-cluster:9092 # You can specify all the Kafka Bootstrap Servers that you have. e.g.: kafka-cluster-2:9092,kafka-cluster-3:9092,kafka-cluster-4:9092,kafka-cluster-5:9092,kafka-cluster-6:9092,kafka-cluster-7:9092\n\n    - name: TRAINING_MODEL_IMAGE\n-     value: localhost:5000/model_training\n+     value: x.x.x.x:5000/model_training\n    - name: INFERENCE_MODEL_IMAGE\n-     value: localhost:5000/model_inference\n+     value: x.x.x.x:5000/model_inference\n    - name: FRONTEND_URL\n-     value: http://localhost\n+     value: http://x.x.x.x\n```\n\nThe same should be done at `frontend-deployment.yaml` file:\n\n```yaml\n containers:\n -   - image: localhost:5000/backend\n +   - image: x.x.x.x:5000/backend\n\n    - name: BACKEND_URL\n-     value: http://localhost:8000\n+     value: http://x.x.x.x:8000\n```\n\nTo be able to deploy components in a Kubernetes cluster, we need to create a\nservice account, give access to that account and generate a token:\n\n```bash\n$ sudo kubectl create serviceaccount k8sadmin -n kube-system\n\n$ sudo kubectl create clusterrolebinding k8sadmin --clusterrole=cluster-admin --serviceaccount=kube-system:k8sadmin\n\n$ sudo kubectl -n kube-system describe secret $(sudo kubectl -n kube-system get secret | (grep k8sadmin || echo \"$_\") | awk '{print $1}') | grep token: | awk '{print $2}'\n```\n\nWith the obtained token in the last step, we have to change the **KUBE_TOKEN**\nenv var to include it, and the **KUBE_HOST** var to include the URL of the\nKubernetes master (e.g., https://IP_MASTER:6443) in the\n`backend-deployment.yaml` file:\n\n```\n    - name: KUBE_TOKEN\n      value: # include token here (and remove #)\n    - name: KUBE_HOST\n      value: # include kubernetes master URL here\n```\n\nFinally, to allow access to the back-end from outside Kubernetes, we can do this\nby assigning a node cluster IP available to the back-end service in Kubernetes.\nFor example, given the IP y.y.y.y.y of a node in the cluster, we could include\nit in the `backend-service.yaml` file:\n\n```\n  type: LoadBalancer\n+ externalIPs:\n+ - y.y.y.y.y.y\n```\n\nAdd this IP also to the **ALLOWED_HOSTS** env var in the\n`backend-deployment.yaml` file:\n\n```\n    - name: ALLOWED_HOSTS\n      value: y.y.y.y, localhost\n```\n\n### GPU configuration\n\nThe following steps are required in order to use GPU acceleration in Kafka-ML\nand Kubernetes. These steps are required to be performed in all the Kubernetes\nnodes.\n\n1. GPU Driver installation\n\n```bash\n# SSH into the worker machine with GPU\n$ ssh USERNAME@EXTERNAL_IP\n\n# Verify ubuntu driver\n$ sudo apt install ubuntu-drivers-common\n$ ubuntu-drivers devices\n\n# Install the recommended driver\n$ sudo ubuntu-drivers autoinstall\n\n# Reboot the machine\n$ sudo reboot\n\n# After the reboot, test if the driver is installed correctly\n$ nvidia-smi\n```\n\n2. Nvidia Docker installation\n\n```bash\n# SSH into the worker machine with GPU\n$ ssh USERNAME@EXTERNAL_IP\n\n# Add the package repositories\n$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)\n$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -\n$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list\n\n$ sudo apt-get update \u0026\u0026 sudo apt-get install -y nvidia-docker2\n$ sudo systemctl restart docker\n```\n\n3. Modify the following file\n\n```bash\n# SSH into the worker machine with GPU\n$ ssh USERNAME@EXTERNAL_IP\n$ sudo tee /etc/docker/daemon.json \u003c\u003cEOF\n{\n    \"default-runtime\": \"nvidia\",\n    \"runtimes\": {\n        \"nvidia\": {\n            \"path\": \"/usr/bin/nvidia-container-runtime\",\n            \"runtimeArgs\": []\n        }\n    }\n}\nEOF\n$ sudo pkill -SIGHUP docker\n$ sudo reboot\n```\n\n4. Kubernetes GPU Sharing extension installation\n\n```bash\n# From your local machine that has access to the Kubernetes API\n$ curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/gpushare-schd-extender.yaml\n$ kubectl create -f gpushare-schd-extender.yaml\n\n$ wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml\n$ kubectl create -f device-plugin-rbac.yaml\n\n$ wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml\n# update the local file so the first line is 'apiVersion: apps/v1'\n$ kubectl create -f device-plugin-ds.yaml\n\n# From your local machine that has access to the Kubernetes API\n$ kubectl label node worker-gpu-0 gpushare=true\n```\n\nThanks to Sven Degroote from ML6team for the GPU and Kubernetes setup\n[documentation](https://blog.ml6.eu/a-guide-to-gpu-sharing-on-top-of-kubernetes-6097935ababf).\n\n## Publications\n\n1. Carnero, A., Martín, C., Torres, D. R., Garrido, D., Díaz, M., \u0026 Rubio, B.\n   (2021).\n   [Managing and Deploying Distributed and Deep Neural Models through Kafka-ML in the Cloud-to-Things Continuum](https://ieeexplore.ieee.org/abstract/document/9529202).\n   IEEE Access, 9, 125478-125495.\n\n2. Martín, C., Langendoerfer, P., Zarrin, P. S., Díaz, M., \u0026 Rubio, B. (2022).\n   [Kafka-ML: connecting the data stream with ML/AI frameworks](https://www.sciencedirect.com/science/article/pii/S0167739X21002995).\n   Future Generation Computer Systems, 126, 15-33.\n\n3. Torres, D. R., Martín, C., Rubio, B., \u0026 Díaz, M. (2021).\n   [An open source framework based on Kafka-ML for DDNN inference over the Cloud-to-Things continuum](https://www.sciencedirect.com/science/article/pii/S138376212100151X).\n   Journal of Systems Architecture, 102214.\n\n4. Chaves, A. J., Martín, C., \u0026 Díaz, M. (2023).\n   [The orchestration of Machine Learning frameworks with data streams and GPU acceleration in Kafka‐ML: A deep‐learning performance comparative](https://onlinelibrary.wiley.com/doi/10.1111/exsy.13287).\n   Expert Systems, e13287.\n\n5. Chaves, A. J., Martín, C., \u0026 Díaz, M. (2024).\n   [Towards flexible data stream collaboration: Federated Learning in Kafka-ML](https://doi.org/10.1016/j.iot.2023.101036).\n   Internet of Things, 101036.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fertis-research%2Fkafka-ml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fertis-research%2Fkafka-ml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fertis-research%2Fkafka-ml/lists"}