{"id":23445933,"url":"https://github.com/fission/fission-kafka-sample","last_synced_at":"2025-04-13T15:11:34.343Z","repository":{"id":42554147,"uuid":"142045437","full_name":"fission/fission-kafka-sample","owner":"fission","description":"Kafka with fission to process events from a vehicle fleet.","archived":false,"fork":false,"pushed_at":"2022-11-16T09:26:13.000Z","size":1254,"stargazers_count":15,"open_issues_count":11,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-27T06:05:17.717Z","etag":null,"topics":["events","fission","iot","kafka","kubernetes","serverless"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fission.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-23T17:14:42.000Z","updated_at":"2024-11-26T18:48:02.000Z","dependencies_parsed_at":"2023-01-22T00:03:23.003Z","dependency_job_id":null,"html_url":"https://github.com/fission/fission-kafka-sample","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fission%2Ffission-kafka-sample","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fission%2Ffission-kafka-sample/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fission%2Ffission-kafka-sample/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fission%2Ffission-kafka-sample/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fission","download_url":"https://codeload.github.com/fission/fission-kafka-sample/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248732484,"owners_count":21152852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["events","fission","iot","kafka","kubernetes","serverless"],"created_at":"2024-12-23T20:28:28.260Z","updated_at":"2025-04-13T15:11:34.303Z","avatar_url":"https://github.com/fission.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Table of Contents\n=================\n\n   * [Table of Contents](#table-of-contents)\n   * [Fission Kafka - IoT Demo](#fission-kafka---iot-demo)\n      * [Architecture](#architecture)\n         * [Functions](#functions)\n      * [Result](#result)\n      * [Setup](#setup)\n         * [Prerequisite](#prerequisite)\n         * [Setup Kafka](#setup-kafka)\n         * [Setup Redis](#setup-redis)\n         * [Configuration Check/Change](#configuration-checkchange)\n         * [Build \u0026amp; Deploy functions](#build--deploy-functions)\n         * [Testing](#testing)\n\n\nCreated by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)\n\n\n# Fission Kafka - IoT Demo\n\nThis sample project is inspired from [Yugabyte's IOT fleet management demo](https://github.com/YugaByte/yb-iot-fleet-management). The original example was built using  microservice, this demo will transform that into an example which uses Fission functions.\n\n## Architecture\n\n![Architecture of the Fission Kafka - IOT Demo](/static_assets/architecture-diagram.png)\n\n### Functions\n\n1. IOT Data Producer: This function generates data about a fleet of vehicles and their lattitude, longitude, speed, fuel level etc. This function can be trigerred every N seconds resembling a sensor sending the data. This function sends this data to the a Kafka topic: iotdata (Configured in environment variable of IOT data Producer function).\n\n2. IOT Data Consumer: Second function retrieves data from Kafka topics, runs some transformations and persists into Redis. This function is trigerred for every message in the Kafka topic.\n\n3. There are 4 more functions which read the data stored in Redis and expose them at REST endpoints: count, fuelavg, data \u0026 speedavg\n\n4. The Kafka consumer \u0026 4 rest endpoint functions are all in same Spring project and form a single jar archive. The entrypoint for each function is different.\n\n5. IOT Web: The last queries the data from rest endpoints and renders using chart.js, HTML \u0026 jQuery. This dashboard page itself is a function rendered using python flask \u0026 HTML.\n\n\n## Result\n\nThe resulting dashboard shows various trends. It also provides a way to invoke the IOT Data producer function which sends data to Kafka. Sample screen:\n\n![Dashboard: Fission Kafka - IOT Demo](/static_assets/iot-demo-screen.png)\n\n## Setup\n\n### Prerequisite\n\n- A Kubernetes cluster with latest version of Fission\n- Fission installation should have Kafka MQT installed with address of Kafka brokers.\n- Docker and Fission-CLI on local machine\n\n### Setup Kafka\n\nEasiest way to install Kafka is to start with Quickstart of [Strimzi Kafka Operator](https://strimzi.io/quickstarts/). \n\n```\n$ kubectl create namespace kafka\n\n$ kubectl apply -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka\n\n$ kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-persistent-single.yaml -n kafka \n\n```\n\n### Setup Redis\n\nWe will setup redis without persistence and as a single instance (i.e. no master \u0026 slave). You can customize the command as per your needs based on [instrcutions from here](https://github.com/helm/charts/tree/master/stable/redis#configuration). Also note that for simplicity - Redis has been setup without a password.\n\n```\n$ kubectl create namespace redis\n\n$ helm repo add bitnami https://charts.bitnami.com/bitnami\n\n$ helm install redis-single --namespace redis \\\n  --set usePassword=false \\\n  --set cluster.enabled=false \\\n  --set master.persistence.enabled=false \\\n  --set master.disableCommands={\"FLUSHALL\"} \\\n    bitnami/redis\n```\n\n**Note**: The parameter `--set master.disableCommands={\"FLUSHALL\"}` enables the [FLUSHDB](https://redis.io/commands/flushdb) command in Redis, which allows to delete everything in a database. This is needed for the `Flush Redis Data` action in this example, but should be used carefully. To disbale the FLUSHDB command, just remove the line `--set master.disableCommands={\"FLUSHALL\"} \\`\n\n### Configuration Check/Change\n\nSome key configuration need to be checked for having proper values:\n\n\n- You will also need to update the Fission installation if not already to enable the Kafka MQT trigger with appropriate configuration of broker address etc.\n\n```\n## Kafka: enable and configure the details\nkafka:\n  enabled: true\n  # note: below link is only for reference. \n  # Please use the brokers link for your kafka here. \n  brokers: 'my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9092' # or your-bootstrap-server.kafka:9092/9093\n  authentication:\n```\n\n- In specs/env-java.yaml check \n\n  - KAFKA_ADDR is pointing to appropriate address of Kafka cluster created in earlier section (Current default to my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9092)\n\n  - REDIS_ADDR is pointing to correct Redis address within cluster (Default \"redis-single-master.redis\")\n\n  - CONSUMER_SLEEP_DELAY - is right now set to 300 second. This is delay which consumer function introduces before consuming a message.\n\n  - PRODUCER_MSG_PER_SECOND - is set to 300 and is number of Kafka messages that producer produces every second.\n\n\n\n### Build \u0026 Deploy functions\n\nThe functions are built inside the cluster after the `fission spec apply` command.\n\n- All the environment, package and function definitions are in specs directory. To create all of them run command:\n\n```\n$ fission spec apply \n```\n\n- You can check details of deployed functions:\n\n```\n$ fission fn list \n```\n\n\n### Testing\n\n- To generate some sample data, you can run the Kafka producer (kp) function a few times or create a trigger which invokes the function every N minutes:\n\n```\n# Creating trigger to produce data every 5 minutes\n$ fission tt create --name kptrigger --function kp --cron '@every 5m'\ntrigger 'kptrigger' created\nCurrent Server Time: \t2018-08-09T10:58:38Z\nNext 1 invocation: \t2018-08-09T11:03:38Z\n$ fission tt list\nNAME      CRON      FUNCTION_NAME\nkptrigger @every 5m kp\n\n# Calling the producer function directly.\n$ fission fn test --name kp\n```\n\n- After the Kafka producer function has trigerred at least once, you can open the dashboard function in browser:\n\n```http://$FISSION_ROUTER/iot/index.html```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffission%2Ffission-kafka-sample","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffission%2Ffission-kafka-sample","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffission%2Ffission-kafka-sample/lists"}