{"id":21877641,"url":"https://github.com/rbiedrawa/kafka-ksqldb-workshop","last_synced_at":"2026-05-13T07:07:17.968Z","repository":{"id":162193102,"uuid":"380293453","full_name":"rbiedrawa/kafka-ksqldb-workshop","owner":"rbiedrawa","description":"Starter code for the ksqlDB workshop. This workshop explains how to integrate ksqlDB with an external data source (Elasticsearch) to build Driver Location Tracking \u0026 Monitoring app and use streaming SQL engine for Apache Kafka.","archived":false,"fork":false,"pushed_at":"2022-01-31T02:03:23.000Z","size":26036,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-26T18:17:20.618Z","etag":null,"topics":["apache-kafka","docker","docker-compose","elasticsearch","kafka","kafka-connect","kibana","ksql","ksqldb","ksqldb-cli","streaming","workshop","workshop-materials"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rbiedrawa.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-25T16:17:32.000Z","updated_at":"2021-12-27T12:19:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"ed296914-ab96-4c03-a383-6c433a10743d","html_url":"https://github.com/rbiedrawa/kafka-ksqldb-workshop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-ksqldb-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-ksqldb-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-ksqldb-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-ksqldb-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rbiedrawa","download_url":"https://codeload.github.com/rbiedrawa/kafka-ksqldb-workshop/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244885526,"owners_count":20526293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","docker","docker-compose","elasticsearch","kafka","kafka-connect","kibana","ksql","ksqldb","ksqldb-cli","streaming","workshop","workshop-materials"],"created_at":"2024-11-28T08:09:51.694Z","updated_at":"2026-05-13T07:07:17.922Z","avatar_url":"https://github.com/rbiedrawa.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ksqlDB Workshop\n\nThis repository contains the starter code for the **ksqlDB** workshop.\n\n## Overview\n\nThis workshop explains how to integrate ksqlDB with an external data source (Elasticsearch) to build **Driver\nLocation Tracking \u0026 Monitoring** app and use streaming SQL engine for Apache Kafka.\n\n![driver-live-location-dashboard.png](./_docs/img/driver-live-location-dashboard.png)\n\n###  Architecture\n\n![architecture](./_docs/img/architecture.png)\n\n## Getting Started\n\nUse Docker Compose ([docker-compose.yml](docker/docker-compose.yml)) for setting up and running the infrastructure,\nwhich includes ksqlDB and Kafka Connect in embedded mode.\n\nThe following tools are available when you run the whole infrastructure:\n\n* [Apache Kafka](https://kafka.apache.org/)\n* [Zookeeper](https://zookeeper.apache.org/)\n* [ksqlDB](https://docs.ksqldb.io/en/latest/reference/)\n* [Kafka Connect](https://kafka.apache.org/documentation/#connect)\n* [Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html)\n* [Kowl UI](https://github.com/cloudhut/kowl)\n* [Kafka Connect UI](https://github.com/lensesio/kafka-connect-ui)\n* [Elasticsearch](https://www.elastic.co/elasticsearch/)\n* [Kibana](https://www.elastic.co/kibana)\n\n### Prerequisites\n\n* Docker\n\n### Usage\n\n#### 1. Start Docker containers\n\n```shell\ncd docker\ndocker compose up -d\n```\n\n#### 2. Check if all components are up and running\n\n```shell\ndocker compose ps\n\n# NAME                SERVICE             STATUS              PORTS\n# elasticsearch       elasticsearch       running             0.0.0.0:9300-\u003e9300/tcp, :::9300-\u003e9300/tcp, 0.0.0.0:9200-\u003e9200/tcp, :::9200-\u003e9200/tcp\n# kafka               kafka               running             0.0.0.0:9092-\u003e9092/tcp, :::9092-\u003e9092/tcp, 0.0.0.0:9101-\u003e9101/tcp, :::9101-\u003e9101/tcp\n# kafka-connect-ui    kafka-connect-ui    running             0.0.0.0:8000-\u003e8000/tcp, :::8000-\u003e8000/tcp\n# kibana              kibana              running             0.0.0.0:5601-\u003e5601/tcp, :::5601-\u003e5601/tcp\n# kowl                kowl                running             0.0.0.0:8080-\u003e8080/tcp, :::8080-\u003e8080/tcp\n# ksqldb-cli          ksqldb-cli          running             \n# ksqldb-server       ksqldb-server       running             0.0.0.0:8083-\u003e8083/tcp, :::8083-\u003e8083/tcp, 0.0.0.0:8088-\u003e8088/tcp, :::8088-\u003e8088/tcp\n# schema-registry     schema-registry     running             0.0.0.0:8081-\u003e8081/tcp, :::8081-\u003e8081/tcp\n# zookeeper           zookeeper           running             0.0.0.0:2181-\u003e2181/tcp, :::2181-\u003e2181/tcp, 2888/tcp, 3888/tcp\n```\n\n#### 3. Start ksqlDB's interactive CLI\n\n```shell\ndocker compose exec ksqldb-cli bash ksql http://ksqldb-server:8088\n```\n\n#### 4. Create driver profile table in ksqlDB\n\n```shell\nCREATE TABLE driver_table (driver_id BIGINT PRIMARY KEY, name VARCHAR)\n  WITH (kafka_topic='driver.info.snapshots', value_format='AVRO', partitions=6);\n```\n\n#### 5. Create stream for driver location events\n\n```shell\nCREATE STREAM driver_location_stream (driver_id BIGINT KEY, location STRUCT\u003clat DOUBLE, lon DOUBLE\u003e)\n  WITH (kafka_topic='driver.location.events', value_format='AVRO', partitions=6);\n```\n\n#### 6. Enrich driver_location stream by joining with driver table\n\n```shell\nCREATE STREAM enriched_driver_location_stream  WITH (\n  kafka_topic='driver.location.enriched.events',\n  value_format='AVRO') AS\n  SELECT\n    driver.driver_id       AS driver_id,\n    driver.name            AS driver_name,\n    CAST(location-\u003elat AS STRING) + ',' + CAST(location-\u003elon AS STRING) AS location\n  FROM driver_location_stream driver_location JOIN driver_table driver\n    ON driver_location.driver_id = driver.driver_id\n  EMIT CHANGES;\n```\n\n#### 7. List defined streams\n\n```shell\nshow streams;\n\n#  Stream Name                     | Kafka Topic                        | Key Format | Value Format | Windowed \n# -------------------------------------------------------------------------------------------------------------\n#  DRIVER_LOCATION_STREAM          | driver.location.events             | KAFKA      | AVRO         | false    \n#  ENRICHED_DRIVER_LOCATION_STREAM | driver.location.enriched.events    | KAFKA      | AVRO         | false    \n#  KSQL_PROCESSING_LOG             | ksql_connect_01ksql_processing_log | KAFKA      | JSON         | false    \n# -------------------------------------------------------------------------------------------------------------\n```\n\n#### 8. List defined tables\n\n```shell\nshow tables;\n\n#  Table Name           | Kafka Topic           | Key Format | Value Format | Windowed \n# -------------------------------------------------------------------------------------\n#  DRIVER_TABLE | driver.info.snapshots | KAFKA      | AVRO         | false    \n# -------------------------------------------------------------------------------------\n```\n\n#### 9. Write data to driver profile table\n\n```shell\nINSERT INTO driver_table (driver_id, name) VALUES (1, 'Driver 1');\nINSERT INTO driver_table (driver_id, name) VALUES (2, 'Driver 2');\nINSERT INTO driver_table (driver_id, name) VALUES (3, 'Driver 3');\nINSERT INTO driver_table (driver_id, name) VALUES (4, 'Driver 4');\nINSERT INTO driver_table (driver_id, name) VALUES (5, 'Driver 5');\nINSERT INTO driver_table (driver_id, name) VALUES (6, 'Driver 6');\n```\n\n#### 10. Write data to driver location stream\n\n```shell\nINSERT INTO driver_location_stream (driver_id, location) VALUES (1, STRUCT(lat := 50.055355899352136, lon := 19.935877982646776));\nINSERT INTO driver_location_stream (driver_id, location) VALUES (2, STRUCT(lat := 50.061660189205384, lon := 19.92373795331377));\nINSERT INTO driver_location_stream (driver_id, location) VALUES (3, STRUCT(lat := 50.06767789514652, lon := 19.913912678521136));\nINSERT INTO driver_location_stream (driver_id, location) VALUES (4, STRUCT(lat := 49.984300646662135, lon := 20.053662685583486));\nINSERT INTO driver_location_stream (driver_id, location) VALUES (5, STRUCT(lat := 50.05768283673512, lon := 19.85473522517211));\nINSERT INTO driver_location_stream (driver_id, location) VALUES (6, STRUCT(lat := 50.06721095767255, lon := 19.87215179908983));\n```\n\n#### 11. Print the raw data from `driver.info.snapshots` topic\n\n```shell\nPRINT 'driver.info.snapshots' FROM BEGINNING;\n```\n\n#### 12. Tell ksqlDB to start all queries from `earliest` point in each topic\n\n```shell\nSET 'auto.offset.reset' = 'earliest';\n```\n\n#### 13. Run a continuous query\n\n```shell\nSELECT * from enriched_driver_location_stream EMIT CHANGES;\n```\n\n#### 14. Exit ksqlDB interactive CLI\n\n```shell\nexit\n\n# ksql\u003e exit\n# Exiting ksqlDB.\n```\n\n#### 15. Check if ElasticSearch is up and running\n\n```shell\ncurl localhost:9200\n\n# {\n#   \"name\" : \"74c66e90824c\",\n#   \"cluster_name\" : \"docker-cluster\",\n#   \"cluster_uuid\" : \"CPyD4K3QT4KOqs-ut9HPCA\",\n#   \"version\" : {\n#     \"number\" : \"7.8.0\",\n#     \"build_flavor\" : \"default\",\n#     \"build_type\" : \"docker\",\n#     \"build_hash\" : \"757314695644ea9a1dc2fecd26d1a43856725e65\",\n#     \"build_date\" : \"2020-06-14T19:35:50.234439Z\",\n#     \"build_snapshot\" : false,\n#     \"lucene_version\" : \"8.5.1\",\n#     \"minimum_wire_compatibility_version\" : \"6.8.0\",\n#     \"minimum_index_compatibility_version\" : \"6.0.0-beta1\"\n#   },\n#   \"tagline\" : \"You Know, for Search\"\n# }\n```\n\n#### 16. Add dynamic template for `driver.location.enriched.events`\n\n```shell\ncurl -X PUT \"localhost:9200/driver.location.enriched.events/?pretty\" -H 'Content-Type: application/json' -d'\n{\n\"mappings\": {\n\"dynamic_templates\": [{\n\"locations\": {\n\"mapping\": {\n\"type\": \"geo_point\"\n},\n\"match\": \"*LOCATION\"\n}}]}}\n'\n\n# {\n#   \"acknowledged\" : true,\n#   \"shards_acknowledged\" : true,\n#   \"index\" : \"driver.location.enriched.events\"\n# }\n```\n\n#### 17. Import Kibana 'Driver live-location' dashboard\n\nOpen your web browser and go to Kibana\n[Saved Objects page](http://localhost:5601/app/kibana#/management/kibana/objects).\n\nImport [driver_live_location.ndjson](kibana/driver_live_location.ndjson) file.\n\n#### 18. Create elasticsearch sink connector\n\nStart ksqlDB's interactive CLI\n\n```shell\ndocker compose exec ksqldb-cli bash ksql http://ksqldb-server:8088\n```\n\nIn the ksqlDB CLI, run the following command to create sink connector.\n\n```shell\nCREATE SOURCE CONNECTOR elasticsearch_sink_demo WITH (\n'connector.class'                       = 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector',\n'connection.url'                        = 'http://elasticsearch:9200',\n'topics'                                = 'driver.location.enriched.events',\n'type.name'                             = '_doc',\n'errors.log.enable'                     = 'true',\n'schema.registry.url'                   = 'http://schema-registry:8081',\n'key.converter'                         = 'org.apache.kafka.connect.storage.StringConverter',\n'value.converter'                       = 'io.confluent.connect.avro.AvroConverter',\n'value.converter.schema.registry.url'   = 'http://schema-registry:8081'\n); \n```\n\nWhen the sink connector is created, it exports any data matching the specified `topics`.\n\n#### 19. Describe created connector\n\n```shell\nDESCRIBE connector elasticsearch_sink_demo;\n```\n\n#### 20. Check Kibana dashboard\n\nOpen your web browser and go to Kibana [Maps page](http://localhost:5601/app/maps) then open `Driver live-location`.\n\nUpdate `Driver 1` position, by running following command.\n\n```shell\nINSERT INTO driver_location_stream (driver_id, location) VALUES (1, STRUCT(lat := 50.078682625477754, lon := 19.7877404489687));\n```\n\nUse `Driver live location` Layer -\u003e **Fit to Data** to move to current location of driver in Kibana dashboard.\n\n#### 21. Experiment with ksqlDB queries and build awesome stream apps.\n\nThis workshop tutorial was just the beginning and only touched some basic concepts and useful commands that can be used during ksqlDB development.\n\nBelow few useful links that can help improve Your knowledge about ksqlDB.\n\n* [Introducing ksqlDB](https://www.confluent.io/blog/intro-to-ksqldb-sql-database-streaming/)\n* [An introduction to ksqlDB](https://www.youtube.com/watch?v=7mGBxG2NhVQ)\n* [ksqlDB HOWTO: Joins](https://www.youtube.com/watch?v=_0Ktp2eB-as)\n* [ksqlDB HOWTO: Handling Time](https://www.youtube.com/watch?v=scpbbl71CD8)\n* [ksqlDB HOWTO: Split and Merge Kafka Topics](https://www.youtube.com/watch?v=5NoU7D4OGA0)\n* [4 Incredible ksqlDB Techniques (#2 Will Make You Cry)](https://www.confluent.io/blog/ksqldb-techniques-that-make-stream-processing-easier-than-ever/)\n\n#### 22. Destroy demo infrastructure\n\nWhen you're done, stop Docker containers by running.\n\n```shell\ndocker compose down -v\n```\n\n## Important Endpoints\n\n| Name | Endpoint | \n| -------------:|:--------:|\n| `Ksql` | [http://localhost:8083/](http://localhost:8083/) |\n| `Ksql - Embedded Kafka Connect` | [http://localhost:8088/](http://localhost:8088/) |\n| `Kafka Connect UI` | [http://localhost:8000/](http://localhost:8000/) |\n| `Kowl UI` | [http://localhost:8080/](http://localhost:8080/) |\n| `Schema-registry` | [http://localhost:8081/](http://localhost:8081/) |\n| `Elasticsearch` | [http://localhost:9200/](http://localhost:9200/) |\n| `Kibana` | [http://localhost:5601/](http://localhost:5601/) |\n\n## References\n\n* [ksqlDB](https://docs.ksqldb.io/en/latest/reference/)\n* [ksqlDB Headless Server Settings](https://docs.ksqldb.io/en/latest/operate-and-deploy/installation/install-ksqldb-with-docker/#ksqldb-headless-server-settings)  \n* [Kafka Docker Images](https://github.com/confluentinc/kafka-images)\n* [Elasticsearch Sink Connector](https://docs.confluent.io/kafka-connect-elasticsearch/current/index.html)\n* [Confluent Hub](https://www.confluent.io/hub/)\n* [Kafka Connect UI](https://github.com/lensesio/kafka-connect-ui)\n* [cloudhut/kowl](https://github.com/cloudhut/kowl)\n* [Elasticsearch](https://www.elastic.co/elasticsearch/)\n* [Kibana](https://www.elastic.co/kibana)\n* [How to Visualize Geo Data on a Map with Kibana - Version 7.10](https://www.youtube.com/watch?v=slwlQQLesKA)\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbiedrawa%2Fkafka-ksqldb-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frbiedrawa%2Fkafka-ksqldb-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbiedrawa%2Fkafka-ksqldb-workshop/lists"}