{"id":24576575,"url":"https://github.com/dmrhimali/kafkaintro","last_synced_at":"2025-07-22T03:06:11.130Z","repository":{"id":270609801,"uuid":"910691854","full_name":"dmrhimali/kafkaIntro","owner":"dmrhimali","description":"Introduction to kafka, producers, consumers and connectors","archived":false,"fork":false,"pushed_at":"2025-01-01T19:24:10.000Z","size":19,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-17T13:14:28.457Z","etag":null,"topics":["docker-compose","kafka"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmrhimali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-01T04:53:37.000Z","updated_at":"2025-01-01T20:43:21.000Z","dependencies_parsed_at":"2025-01-01T19:37:53.852Z","dependency_job_id":"344e339e-a788-4f95-9b93-ca77dba3a8d9","html_url":"https://github.com/dmrhimali/kafkaIntro","commit_stats":null,"previous_names":["dmrhimali/kafkaintro"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmrhimali/kafkaIntro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FkafkaIntro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FkafkaIntro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FkafkaIntro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FkafkaIntro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmrhimali","download_url":"https://codeload.github.com/dmrhimali/kafkaIntro/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FkafkaIntro/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266417111,"owners_count":23925301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker-compose","kafka"],"created_at":"2025-01-23T22:40:20.961Z","updated_at":"2025-07-22T03:06:11.104Z","avatar_url":"https://github.com/dmrhimali.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafkaDemo\n\n# Install Kafka\n\nSee [installing kafka in local machine](https://github.com/dmrhimali/debeziumDemo/tree/main/postgres-kafka#debezium-kafka-on-local-mac) for details.\n\n## With docker-compose.yml\n\nRun docker-compose:\n\n`docker-compose up zookeeper`\n\n`docker-compose up kafka`\n\n`docker-compose up postgres`\n\n`docker-compose up connect` \n\n`docker-compose ps`\n\nOUTPUT:\n```sh\n    Name                  Command              State                                Ports\n----------------------------------------------------------------------------------------------------------------------\nconnect        /docker-entrypoint.sh start     Up      0.0.0.0:8083-\u003e8083/tcp, 8778/tcp, 9092/tcp, 9779/tcp\ndeb_postgres   docker-entrypoint.sh postgres   Up      0.0.0.0:5432-\u003e5432/tcp\nkafka          /docker-entrypoint.sh start     Up      8778/tcp, 0.0.0.0:9092-\u003e9092/tcp, 9779/tcp\nzookeeper      /docker-entrypoint.sh start     Up      0.0.0.0:2181-\u003e2181/tcp, 0.0.0.0:2888-\u003e2888/tcp,\n```\n                                                     0.0.0.0:3888-\u003e3888/tcp, 8778/tcp, 9779/tcp\n\n                                                     \ncheck kafka-connect:\n`url -H \"Accept:application/json\" localhost:8083/`\n{\"version\":\"2.4.0\",\"commit\":\"77a89fcf8d7fa018\",\"kafka_cluster_id\":\"4zDsU771QMS_rZQr8LHTkw\"}%\n\n\n**Without docker-cpmpose.yml**\nDebizium docker images:\nhttps://github.com/debezium/docker-images/tree/master/postgres\n\nhttps://medium.com/@tilakpatidar/streaming-data-from-postgresql-to-kafka-using-debezium-a14a2644906d\n\n\nStart a PostgreSQL instance:\n`docker run -it --rm --name postgres -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres debezium/example-postgres:1.0`\n\nStart a Zookeeper instance:\n`docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.0`\n\n\nStart a Kafka instance:\n`docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.0`\n\nStart a Debezium instance (connect):\n`docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME=$(echo $DOCKER_HOST | cut -f3 -d'/' | cut -f1 -d':') --link zookeeper:zookeeper --link postgres:postgres --link kafka:kafka debezium/connect:1.0`\n\nConnect to postgres container and create inventory db:\n\n`docker exec -it postgres psql -U postgres`\n\nin `postgres=#` prompt execute following:\n\n```sql\n\\l\n\nCREATE DATABASE inventory;\n\n\\c inventory\n\nCREATE TABLE customers\n(\n  id bigint NOT NULL,\n  first_name VARCHAR (50),\n  last_name VARCHAR (50) NOT NULL,\n  email VARCHAR (355) UNIQUE NOT NULL,\n  CONSTRAINT pk_customers PRIMARY KEY (id)\n);\n\n\\d customers\n\ninsert into customers values(1001, 'sally', 'thomas', 's.t@acme.com');\ninsert into customers values(1002, 'george', 'bailey', 'g.b@acme.com');\ninsert into customers values(1003, 'edward', 'walker', 'e.w@acme.com');\n\nselect * from public.customers;\n```\n\nOUPTPUT:\n\ncheck database:\n`postgres\u003e \\list`\n\\c inventory\n```sql\n                                 List of databases\n   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges\n-----------+----------+----------+------------+------------+-----------------------\n postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |\n template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +\n           |          |          |            |            | postgres=CTc/postgres\n template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +\n           |          |          |            |            | postgres=CTc/postgres\n(3 rows)\n\n\ncreate database inventory;\n```\n\nuse inventory:\n\n`postgres-# \\c inventory`\nFATAL:  database \"inventory\" does not exist\n\n```sql\npostgres=# create database inventory;\nCREATE DATABASE\n\npostgres=# \\c inventory;\nYou are now connected to database \"inventory\" as user \"postgres\".\n\ninventory=# \\dt;\nDid not find any relations.\n\npostgres=# \nCREATE TABLE customers(\n   id INT PRIMARY KEY,\n   first_name VARCHAR (50),\n   last_name VARCHAR (50) NOT NULL,\n   email VARCHAR (355) UNIQUE NOT NULL\n);\n\nCREATE TABLE\n\ninventory=# \\dt;\n           List of relations\n Schema |   Name    | Type  |  Owner\n--------+-----------+-------+----------\n public | customers | table | postgres\n(1 row)\n\ninventory=# insert into customers values(1001, 'sally', 'thomas', 's.t@acme.com');\nINSERT 0 1\ninventory=# insert into customers values(1002, 'george', 'bailey', 'g.b@acme.com');\nINSERT 0 1\ninventory=# insert into customers values(1003, 'edward', 'walker', 'e.w@acme.com');\nINSERT 0 1\n\ninventory=# select * from customers;\n    id |    first_name |    last_name |      email\n-------+---------------+--------------+--------------\n  1001 | sally         | thomas       | s.t@acme.com\n  1002 | george        | bailey       | g.b@acme.com\n  1003 | edward        | walker       | e.w@acme.com\n(3 rows)\n```\n\nhttps://hub.docker.com/r/debezium/postgres\n\n\n\n## install kafka on local mac:\n\n`$ brew cask install java` - i skipped this. i had openjdk installed.\n\n`$ brew install kafka`\n\nThis will install kafka on : `/usr/local/Cellar/kafka/`\n\ncontents of bin folder(these are the scripts/commands avaialbe):\n\n```sh\n(base)  rdissanayakam@RBH12855  /usr/local/Cellar/kafka/2.4.1/bin  ls\nconnect-distributed              kafka-configs                    kafka-delegation-tokens          kafka-mirror-maker               kafka-run-class                  kafka-verifiable-consumer        zookeeper-server-stop\nconnect-mirror-maker             kafka-console-consumer           kafka-delete-records             kafka-preferred-replica-election kafka-server-start               kafka-verifiable-producer        zookeeper-shell\nconnect-standalone               kafka-console-producer           kafka-dump-log                   kafka-producer-perf-test         kafka-server-stop                trogdor\nkafka-acls                       kafka-consumer-groups            kafka-leader-election            kafka-reassign-partitions        kafka-streams-application-reset  zookeeper-security-migration\nkafka-broker-api-versions        kafka-consumer-perf-test         kafka-log-dirs                   kafka-replica-verification       kafka-topics                     zookeeper-server-start\n```\n\nthe poroperies files are located at `/usr/local/Cellar/kafka/2.4.1/libexec/config`: \n\n```sh\n(base)  rdissanayakam@RBH12855  /usr/local/Cellar/kafka/2.4.1  ls libexec\nbin    config libs   logs\n(base)  rdissanayakam@RBH12855  /usr/local/Cellar/kafka/2.4.1  ls libexec/config\nconnect-console-sink.properties   connect-file-sink.properties      connect-mirror-maker.properties   log4j.properties                  tools-log4j.properties\nconnect-console-source.properties connect-file-source.properties    connect-standalone.properties     producer.properties               trogdor.conf\nconnect-distributed.properties    connect-log4j.properties          consumer.properties               server.properties                 zookeeper.properties\n```\n\n### Start Zookeeper:\n`zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties`\n\n### Start Kafka server:\n`kafka-server-start /usr/local/etc/kafka/server.properties`\n\nDuring server start, you might be facing connection broken issue.\n\n```sh\n[2018-08-28 16:24:41,166] WARN [Controller id=0, targetBrokerId=0] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)\n[2018-08-28 16:24:41,268] WARN [Controller id=0, targetBrokerId=0] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)\n```\n\nTo fix this issue, we need to change the server.properties file.\n`$ vim /usr/local/etc/kafka/server.properties`\nHere uncomment the server settings and update the value from\n`listeners=PLAINTEXT://:9092`\nto\n```sh\n############################# Socket Server Settings #############################\n# The address the socket server listens on. It will get the value returned from \n# java.net.InetAddress.getCanonicalHostName() if not configured.\n#   FORMAT:\n#     listeners = listener_name://host_name:port\n#   EXAMPLE:\n#     listeners = PLAINTEXT://your.host.name:9092\nlisteners=PLAINTEXT://localhost:9092\n```\nand restart the server and it will work great.\n\nIf you get the error `InconsistentClusterIdException: The Cluster ID z7KWdJyvQauQHEjzR7JErw doesn't match stored clusterId Some(ABmFXLNBTB-i9xatRD-Faw) in meta.properties. The broker is trying to join the wrong cluster`\n\n- Just Delete all the log/Data file created (or generated) into zookeeper and kafka.\n    **kafka :**\n    logs dir is mentioned in `/usr/local/etc/kafka/server.properties`:\n    ```sh\n    # A comma separated list of directories under which to store log files\n    log.dirs=/usr/local/var/lib/kafka-logs\n    ```\n    delete logs dir content:\n    ```sh\n    base)  rdissanayakam@RBH12855  ~  `rm -rf /usr/local/var/lib/kafka-logs/*`\n    zsh: sure you want to delete all 6 files in /usr/local/var/lib/kafka-logs [yn]? y\n    ```\n    **zookeeper**\n    data dir in `/usr/local/etc/kafka/zookeeper.properties`\n    ```sh\n    # the directory where the snapshot is stored.\n    dataDir=/usr/local/var/lib/zookeeper\n    ```\n    delete:\n    ```sh\n    base)  ✘ rdissanayakam@RBH12855  /usr/local/var/lib/kafka-logs  rm -rf /usr/local/var/lib/zookeeper/*\n    zsh: sure you want to delete the only file in /usr/local/var/lib/zookeeper [yn]? y\n    (base)  rdissanayakam@RBH12855  /usr/local/var/lib/zookeeper \n    ```\n\n\n- Run Zookeper\n\n- Run Kafka\n\n## Create Kafka Topic:\n\ntopic members\n\n`kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic  members`\n\n## List kafka topics:\n`kafka-topics --list --bootstrap-server localhost:9092`\n\nooutput:\n```sh\n__consumer_offsets\nmember\n```\n\n## Initialize Producer console:\nNow we will initialize the Kafka producer console, which will listen to localhost at port 9092 at topic members :\n\n`kafka-console-producer --broker-list localhost:9092 --topic members`\n\noutput:\n\n```sh\n$ kafka-console-producer --broker-list localhost:9092 --topic members\n\u003esend first message\n\u003esend second message\n\u003ewow it is working\n```\n\n## Initialize Consumer console:\nNow we will initialize the Kafka consumer console, which will listen to bootstrap server localhost at port 9092 at topic members from beginning:\n\n`kafka-console-consumer --bootstrap-server localhost:9092 --topic members --from-beginning`\n\n```sh\n$ kafka-console-consumer --bootstrap-server localhost:9092 --topic members --from-beginning\nsend first message\nsend second message\nwow it is working\n```\n\nTo stop kafka broker:\n`kafka-server-stop`\n`zookeeper-server-stop`\n\n## Troubleshoot \n\n### kafka broker will not stop\n\nfix:\nfind process id of kafka borker\n`lsof -t -i :9092`\nthen `kill -9 \u003cPID\u003e`\n\n```sh\n(base)  ✘ rdissanayakam@RBH12855  ~  lsof -t -i :9092\n48370\n(base)  rdissanayakam@RBH12855  ~  kill -9 48370\n(base)  rdissanayakam@RBH12855  ~ \n```\n\n## add plugin for postgres\n\nPlace PostgreSQL JDBC driver placed into /kafka/libs directory:\n`(base)  rdissanayakam@RBH12855  /usr/local/Cellar/kafka/2.4.1/libexec/libs  wget https://jdbc.postgresql.org/download/postgresql-42.2.11.jar`\n\nCreate directory for debezium plugins:\n`mkdir -p debezium_plugins`\n\n`cd debezium_plugins`\n\nPlace debezium postgres plugin in debezium_plugins:\n`wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.0.0.Final/debezium-connector-postgres-1.0.0.Final-plugin.tar.gz`\n\n`tar -xzf debezium-connector-postgres-1.0.0.Final-plugin.tar.gz`\n\nPlace \n\ndownload kafka-connect confluent jdbc connector: https://docs.confluent.io/current/connect/connect-jdbc/docs/index.html\nand move to debezium_plugins\n\n\n`vi /usr/local/Cellar/kafka/2.4.1/libexec/config/connect-standalone.properties`\n\nadd and save:\n`plugin.path=/Users/rdissanayakam/debezium_plugins`\n\n`vi /usr/local/Cellar/kafka/2.4.1/libexec/config/connect-distributed.properties`\n\nadd and save:\n`plugin.path=/Users/rdissanayakam/debezium_plugins`\n\n## run kafka connector\n`connect-standalone /usr/local/Cellar/kafka/2.4.1/libexec/config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties`\n\n\nTODO....\n\n\n## create connector for rds\nhttps://debezium.io/blog/2017/09/25/streaming-to-another-database/\n\n```json\ncurl -X POST -H \"Accept:application/json\" -H \"Content-Type:application/json\" localhost:8083/connectors/ -d '\n{\n \"name\": \"inventory-connector\",\n \"config\": {\n \"connector.class\": \"io.debezium.connector.postgresql.PostgresConnector\",\n \"tasks.max\": \"1\",\n \"database.hostname\": \"ras-db.c4dsc34i9waw.us-east-1.rds.amazonaws.com\",\n \"database.port\": \"5432\",\n \"database.user\": \"testUser\",\n \"database.password\": \"testPassword\",\n \"database.dbname\" : \"genesis\",\n \"database.server.name\": \"dbserver1\",\n \"database.whitelist\": \"genesis\",\n \"database.history.kafka.bootstrap.servers\": \"kafka:9092\",\n \"database.history.kafka.topic\": \"schema-changes.genesis\"\n }\n}'\n```\n\n\n# Debizium with Postgres RDS\n\n## create ec2 instance \n\nI used :\n- Amazon Linux 2 AMI (HVM), SSD Volume Type - ami-0fc61db8544a617ed (64-bit x86) / ami-0f90a34c9df977efb (64-bit Arm)\n- t2.small\n- securtiy group : allow allincoming tcp from anybody ::0\n\n## log into ec2 instance\n\n`ssh -i \"dmrhimali_keypair.pem\" ec2-user@ec2-52-4-164-216.compute-1.amazonaws.com`\n\n## install java\n`sudo yum -y install java-1.8.0-openjdk`\n\n## install kafka\n\nget a mirror link for latest dowmload binary tar.gz from https://kafka.apache.org/downloads\n\nfor latest version 2.4.1 the site https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka_2.11-2.4.1.tgz\nrecommends mirror: http://mirror.metrocast.net/apache/kafka/2.4.1/kafka_2.11-2.4.1.tgz\n\n`wget http://mirror.metrocast.net/apache/kafka/2.4.1/kafka_2.11-2.4.1.tgz`\n\nThis downloads kafka_2.11-2.4.1.tgz\n\nExtract:\n\n`tar -xzf kafka_2.11-2.4.1.tgz`\n\nthis will extract content to a folder kafka_2.11-2.4.1.\n\n## install zookeeper\n\ncd into the kafka directory\n\n`cd kafka_2.11-2.4.1`\n\nWe now need a single node zookeeper server (the script for which is provided by the good folks at Apache Kafka). So, you don’t have to worry about installing it separately.\n\nIf you have zookeeper version \u003e 3.5.3,  four Letter Words need to be explicitly white listed before using:\n\n`vi config/zookeeper.properties`\nadd line and save:\n`4lw.commands.whitelist=*`\n\n## Run zookeeper\nkickoff zookeeper in background:\n`bin/zookeeper-server-start.sh -daemon config/zookeeper.properties`\n\nInstall netcat:\n`sudo yum install nmap`\n\nThis installs `ncat`.\n\nRunning the following ruok command, should return an **imok**.\n`echo \"ruok\" | ncat localhost 2181`\n\noutput:\n```sh\n[ec2-user@ip-172-31-45-35 kafka_2.11-2.4.1]$ echo \"ruok\" |  ncat localhost 2181\nimok\n```\n\nThis tells you that zookeeper is alive and well! See [other four letter words](https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_zkCommands)\n\ncheck zookeeper is runnig :\n\n`ps aux | grep zoo`\n\n\n## Run kafka\n\nNow, we are ready to start up the Kafka server. \n\n###  Set up the advertised.listeners config parameter\nIf you are **NOT** connecting to the topic from the same machine set up the advertised.listeners config parameter in the server.properties file (/home/ec2-user/kafka_2.11-2.4.1/config/server.properties).\n\nadvertised.listeners=PLAINTEXT://\u003ckafkahostname\u003e:9092\n\nIf you are connecting to the topic from the same machine, you don’t need this step.\n\nTo start the Kafka server, run: \n`bin/kafka-server-start.sh -daemon config/server.properties`\n\nOnce the Kafka server is up, proceed to create a topic.\n`bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test`\n\nList topics:\n\n`bin/kafka-topics.sh --list --zookeeper localhost:2181`\n\noutput\n```sh\n__consumer_offsets\nnumtest\ntest\n```\n\nNow that the topic is ready to receive messages, test it using the producer and consumer scripts that are packaged within Kafka.\n\nIf you want to connect to the topic programmatically, skip this section and move on to the next one.\n\n### simple test\n\nUse the producer script to publish messages to your new topic.\n\n`bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test`\n\nEnter some messages at prompt:\n\n```sh\n\u003eMessage One\n\u003eMessage Two\n\u003eMessage Three\n```\n\noutput:\n```sh\n[ec2-user@ip-172-31-45-35 kafka_2.11-2.4.1]$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test\nOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N\n\u003eMessage one\n\u003eMessage two\n\u003eMessage three\n```\n\nYou then run the consumer to see those messages.\n\nssh to ec2 instance in a different terminal and run:\n\n`cd cd kafka_2.11-2.4.1/`\n\n`bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning`\n\nprints :\n  Message One\n  Message Two\n  Message Three\n\noutput:\n\n```sh\n[ec2-user@ip-172-31-45-35 kafka_2.11-2.4.1]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning\nOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N\nMessage one\nMessage two\nMessage three\n```\n\nAny additional messages sent via the producer will automatically show up on the consumer console window.\n\n### Publishing messages using kafka-python\n\nkafka-python is one of the python clients for Apache Kafka.\nYou can install it using pip.\n\ninstall pip:\n`sudo yum install python-pip`\n\ninstall kafka-python:\n`sudo pip install kafka-python`\n\nTo publish a message on to the topic and test, create a KafkaProducer and use the send method to publish a sample message.\n\n**producer.py**\n\n```python\nfrom time import sleep\nfrom json import dumps\nfrom kafka import KafkaProducer\n\nproducer = KafkaProducer(bootstrap_servers=['localhost:9092'],\n                         value_serializer=lambda x: \n                         dumps(x).encode('utf-8'))\n\nfor e in range(1000):\n    data = {'number' : e}\n    producer.send('numtest', value=data)\n    sleep(5)                         \n```\nRun producer:\n`python ./producer.py`\n\n\n**consumer.py**\n\n```python\nfrom kafka import KafkaConsumer\nfrom json import loads\n\nconsumer = KafkaConsumer(\n    'numtest',\n     bootstrap_servers=['localhost:9092'],\n     auto_offset_reset='earliest',\n     enable_auto_commit=True,\n     group_id='my-group',\n     value_deserializer=lambda x: loads(x.decode('utf-8')))\n\nfor message in consumer:\n    message = message.value\n    print('read message: {}'.format(message))\n```\nIn a new ec2 terminal run consumer:\n`python ./consumer.py`\n\nThe new message should now appear on the consumer console that you have running from the previous section.\n\n```sh\n[ec2-user@ip-172-31-45-35 ~]$ python ./consumer.py\nread message: {u'number': 0}\nread message: {u'number': 1}\nread message: {u'number': 2}\n...\n```\n\nIf you prefer a UI to look at the messages in the topic, there are a few options such as Landoop, Cloudkarafka, KafkaTool and others.\n\n## Install kafka connect\n\nhttp://kafka.apache.org/documentation.html#connect\n\nKafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. An export job can deliver data from Kafka topics into secondary storage and query systems or into batch systems for offline analysis\n\nKafka Connect currently supports two modes of execution: \n - standalone (single process) and \n - distributed.\n\nIn standalone mode all work is performed in a single process. This configuration is simpler to setup and get started with and may be useful in situations where only one worker makes sense (e.g. collecting log files), but it does not benefit from some of the features of Kafka Connect such as fault tolerance. \n\nTo start a standalone Kafka Connector, we need following three configuration files.\n- connect-standalone.properties\n- connect-file-source.properties\n- connect-file-sink.properties\n\n**config/connect-file-source.properties**\n```sh\nname=local-file-source\nconnector.class=FileStreamSource\ntasks.max=1\nfile=test.txt\ntopic=connect-test\n```\n\n**config/connect-file-sink.properties**\n```sh\nname=local-file-sink\nconnector.class=FileStreamSink\ntasks.max=1\nfile=test.sink.txt\ntopics=connect-test\n```\n\nhttps://www.tutorialkart.com/apache-kafka/apache-kafka-connector/ \n\nKafka by default provides these configuration files in config folder. We shall use those config files as is. If you go through those config files, you may find in connect-file-source.properties, that the file is test.txt, which we have created in our first step. Run the following command from the kafka directory to start a Kafka Standalone Connecto\n\nLet us test if kafka-connect works:\n\ncreate connect-test topic:\n`bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic connect-test`\n\ncreate test.txt needed for config/connect-file-source.properties in kafka folder with content:\n```sh\nHello!\nWelcome to TutorialKart\nLearn Apache Kafka\n```\n\nNow start a standalone process with the following command:\n`bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties`\n\nObserve `test.sync.txt` created next to `test.txt`\n\nOnce the Connector is started, initially the data in test.txt would be synced to test.sync.txt and the data is published to the Kafka Topic named, connect-test. Then any changes to the test.txt file would be synced to test.sync.txt and published to connect-test topic. Add a new line, ” Learn Connector with Example” to test.txt.\n\n```sh\necho \"Learn Connector\" \u003e\u003e test.txt\n~/kafka_2.12-1.0.0$ cat test.sink.txt\nHello!\nWelcome to TutorialKart\nLearn Apache Kafka\nLearn Connector\n```\n\nNow Consume the messages posted to connect-test topic by a Consumer.\n\nWe shall start a Consumer and consume the messages (test.txt and additions to test.txt). Following is a Kafka Console Consumer. You may create [Kafka Consumer of your application choice](https://www.tutorialkart.com/apache-kafka/kafka-consumer-with-example-java-application/).\n\n`bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning`\n\noutput:\n```sh\n[ec2-user@ip-172-31-45-35 kafka_2.11-2.4.1]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning\nOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N\n{\"schema\":{\"type\":\"string\",\"optional\":false},\"payload\":\"Hello!\"}\n{\"schema\":{\"type\":\"string\",\"optional\":false},\"payload\":\"Welcome to TutorialKart\"}\n{\"schema\":{\"type\":\"string\",\"optional\":false},\"payload\":\"Learn Apache Kafka\"}\n{\"schema\":{\"type\":\"string\",\"optional\":false},\"payload\":\"\"}\n{\"schema\":{\"type\":\"string\",\"optional\":false},\"payload\":\"Learn Connector\"}\n\n```\n\n## install debizium\n\n### download and extract postgres connector to ec2\nhttps://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/\n\n`mkdir -p /home/ec2-user/plugins`\n`cd /home/ec2-user/plugins`\n\n`wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.0.0.Final/debezium-connector-postgres-1.0.0.Final-plugin.tar.gz`\n\n`tar -xzf debezium-connector-postgres-1.0.0.Final-plugin.tar.gz`\n\nThis extract to folder `debezium-connector-postgres`\n\nmove to plugins\n`mv debezium-connector-postgres plugins`\n\nupdate plugin path in connect-distributed properties:\n`vi kafka_2.11-2.4.1/config/connect-distributed.properties`\nadd and save:\n`plugin.path=/home/ec2-user/plugins`\n\nupdate plugin path in connect-standalone properties:\n`vi kafka_2.11-2.4.1/config/connect-standalone.properties`\nadd and save:\n`plugin.path=/home/ec2-user/plugins`\n\nRestart your Kafka Connect process to pick up the new JARs [http://kafka.apache.org/documentation.html#connect](see here).\n\nOption1 : restart standalone connector\n`bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties`\n\nOption2 : restart distributed connector [i just went with rerunning standalone connector. not this.]\n`bin/connect-distributed.sh config/connect-distributed.properties`\n\nyou may need to restart kafka service if kafka connect throws errors.\n\ncurl -X POST -H \"Accept:application/json\" -H \"Content-Type:application/json\" localhost:8083/connectors/ -d ' { \"name\": \"genesis-member-connector\", \"config\": { \"connector.class\": \"io.debezium.connector.postgresql.PostgresConnector\", \"tasks.max\": \"1\", \"database.hostname\": \"postgres\", \"database.port\": \"5432\", \"database.user\": \"testUser\", \"database.password\": \"testPassword\", \"database.dbname\" : \"genesis\", \"database.server.name\": \"dbserver1\", \"database.whitelist\": \"genesis\", \"database.history.kafka.bootstrap.servers\": \"kafka:9092\", \"database.history.kafka.topic\": \"schema-changes.inventory\" } }'\n\n\n\n### Using a Debezium Connector\n\nhttps://debezium.io/documentation/reference/install.html\n\nTo use a connector to produce change events for a particular source server/cluster, simply create a configuration file for the [Postgres Connector]( https://debezium.io/documentation/reference/connectors/postgresql.html#deploying-a-connector) and use the Kafka Connect REST API to add that connector configuration to your Kafka Connect cluster. When the connector starts, it will connect to the source and produce events for each inserted, updated, and deleted row or document.\n\nSee the Debezium Connectors documentation for more information.\n\n## Topics and Partitions\n\nMessages in Kafka are categorized into **topics**.\n\nTopics are additionally broken down into a number of **partitions**.\n\nGoing back to the “commit log” description, a partition is a sin‐ gle log. Messages are written to it in an append-only fashion, and are read in order from beginning to end. Note that as a topic typically has multiple partitions, there is no guarantee of message time-ordering across the entire topic, just within a single partition\n\n# Kafka-Springboot\n\nhttps://www.confluent.io/blog/apache-kafka-spring-boot-application/\n\nhttps://dzone.com/articles/magic-of-kafka-with-spring-boot\n\nhttps://www.baeldung.com/spring-kafka\n\n\nhttps://www.javainuse.com/spring/spring-boot-apache-kafka-hello-world\n\n\nDevelopers,\n\nAs we continue to advance our Kafka development strategies, with developer experience in mind, we come to the first Avro and Avro Schema Migration guidelines to help keep everyone on a similar page. \n\nEvent Avro Schema Will be defined in Kotlin using the [Avro4k](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Favro-kotlin%2Favro4k\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487419089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=rldNMlyRF8onEelB7FdYAVJc0ET0gTnnl5%2FtrHKx8Ls%3D\u0026reserved=0) libraries. \n\nA Gradle plugin will generate the Avro Schema from code [Avro4k Build Plugin](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMagusDevOps%2Favro4k-build-plugins\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487419089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=GKz3UraS1OHE2EN0sqWPq9KOn7nvrK7U%2FReOJteayDg%3D\u0026reserved=0).\n\nSchema Will be validated on pull requests (IN PROGRES) for comparability for Stage and Production using [ImFlog Schema Registry Plugin()]. \nSchema Will be validated on pull requests (IN PROGRES) for comparability for Stage and Production using [ImFlog Schema Registry Plugin](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FImFlog%2Fschema-registry-plugin\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487429084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=J6rj636sFgsWng9HjE2sWnk6XbzUJxFvBPdxlVtv5WE%3D\u0026reserved=0). \n\nThe schema will use compatibility of \"FORWARD\" on stage and production environments as that level matches the closest to the way things currently work with SNS and SQS.\n\nDeployment of Schemas is currently manual using the [ImFlog Schema Registry Plugin](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FImFlog%2Fschema-registry-plugin\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487429084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=J6rj636sFgsWng9HjE2sWnk6XbzUJxFvBPdxlVtv5WE%3D\u0026reserved=0), but the CI pipeline method is a work-in-progress.\n\nGeneric Documentation: [Kafka Schema Registry Quick Start](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconfluence.virginpulse.com%2Fdisplay%2FBUL%2FKafka%2BSchema%2BRegistry%2BDeveloper%2BQuick%2BStart\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487439080%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=kp8asngNM7k76h4dNwYloi63BzCMxjzJcSxuoNO99Gg%3D\u0026reserved=0)\n\nProject Example: [Data Ingestion Example](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.virginpulse.com%2Fengineering%2Fgenesis-data-ingestion%2Fdata-ingestion%2F-%2Fmerge_requests%2F667\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487449071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=QrNTq8XiseuVGp4EkaviiswF6Hz1TLWGxu%2F0fLRtRuQ%3D\u0026reserved=0)\n\nExtras: [ImFlog Schema Registry Plugin ](https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FImFlog%2Fschema-registry-plugin\u0026data=04%7C01%7CRasanjalee.Dissanayaka%40virginpulse.com%7Cc7dcfa713fba494fedc708d8cd136479%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C637484830487449071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026sdata=%2BYa2sWxFec0x4nCjF76jUh2G0mYAgWcTmhYe0UDsybo%3D\u0026reserved=0)\n\n\nnote: you don't need to check in the Avro files as they will automatically be generated during first compile\n\nPlease reach out with any feedback or concerns.\n\nThanks,\nJacob","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmrhimali%2Fkafkaintro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmrhimali%2Fkafkaintro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmrhimali%2Fkafkaintro/lists"}