{"id":19308782,"url":"https://github.com/ciphertron/kafka-streamer","last_synced_at":"2026-04-16T10:01:24.858Z","repository":{"id":174088517,"uuid":"651752261","full_name":"CIPHERTron/kafka-streamer","owner":"CIPHERTron","description":"A microservice that uses Kafka to listens to an event involving order creation and then insert the product orders to a PostgreSQL database.","archived":false,"fork":false,"pushed_at":"2023-06-13T08:58:28.000Z","size":598,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-24T03:17:55.432Z","etag":null,"topics":["docker","docker-c","flask","kafka","kafka-consumer","kafka-producer","postgresql","python-kafka","zook"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CIPHERTron.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-10T02:01:54.000Z","updated_at":"2023-09-15T10:54:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"8f1655fb-1c60-44cf-83d5-4dc749654b6e","html_url":"https://github.com/CIPHERTron/kafka-streamer","commit_stats":null,"previous_names":["ciphertron/kafka-streamer"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/CIPHERTron/kafka-streamer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIPHERTron%2Fkafka-streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIPHERTron%2Fkafka-streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIPHERTron%2Fkafka-streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIPHERTron%2Fkafka-streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CIPHERTron","download_url":"https://codeload.github.com/CIPHERTron/kafka-streamer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CIPHERTron%2Fkafka-streamer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31880882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T09:23:21.276Z","status":"ssl_error","status_checked_at":"2026-04-16T09:23:15.028Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-c","flask","kafka","kafka-consumer","kafka-producer","postgresql","python-kafka","zook"],"created_at":"2024-11-10T00:16:26.806Z","updated_at":"2026-04-16T10:01:24.834Z","avatar_url":"https://github.com/CIPHERTron.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/CIPHERTron/kafka-streamer/\"\u003e\n    \u003cimg src=\"https://dyltqmyl993wv.cloudfront.net/assets/stacks/kafka/img/kafka-stack-220x234.png\"\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003eKafka Streamer\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    A microservice that uses Kafka to listens to an event involving order creation and then insert the product orders to a PostgreSQL database.\n  \u003c/p\u003e\n\u003c/p\u003e\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\"\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n      \u003cul\u003e\n      \u003c/ul\u003e\n        \u003cli\u003e\u003ca href=\"#built-with\"\u003eBuilt With\u003c/a\u003e\u003c/li\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#prerequisites\"\u003ePrerequisites\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#running-the-project\"\u003eRunning the project\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## ℹ️ About The Project\n\n**Kafka Streamer** is a microservice that uses Kafka to listens to an event involving order creation and then insert the product orders to a Postgres database. The database operations are carried out using a DB sink connector which reads _product_orders_ topic and writes the data to a table called **orders** in Postgres DB.\n\nThe overall goal of the assignment is to build a system that accepts product orders, stores them in a PostgreSQL database, forwards them to city-specific topics in Kafka, and sends an email to customers for high-priority orders.\n\n\u003cimg src=\"images/architecture_diagram.png\" alt=\"Architecture diagram for this microservice\" /\u003e\n\n### 🛠️ Built With\n\nFollowing technologies and libraries are used for the development of this\nproject.\n\n- [Python](https://www.python.org/)\n- [Flask](https://flask.palletsprojects.com/en/2.3.x/)\n- [Kafka](https://hub.docker.com/r/bitnami/kafka/)\n- [PostgreSQL](https://www.postgresql.org/)\n- [Docker](https://www.docker.com/)\n- [Docker Compose](https://docs.docker.com/compose/)\n- Shell/CLI\n\n\u003c!-- GETTING STARTED --\u003e\n\n## 📌 Getting Started\n\nTo setup the project locally follow the steps below\n\n### 💻 Prerequisites\n\n- [Docker](https://docs.docker.com/get-docker/)\n- [Docker Compose](https://docs.docker.com/compose/install/)\n- Terminal\n\n### 🤖 Running the project.\n\n1. **Fork** and **clone** the project to your local system\n2. cd into the project and run\n\n```shell\ndocker-compose build --no-cache\ndocker-compose up\n```\n\n3. Then we have to create a table in postgre by running the following commands:\n\n```shell\ndocker ps -a\n\n\u003c!-- copy the container id of db container --\u003e\n\ndocker exec -it \u003ccontainer_id\u003e /bin/bash\n\npsql -U pritishsamal -p 5432 -h localhost -d order\n\n\u003c!-- Now run the following query: --\u003e\n\nCREATE TABLE orders (\n  order_id SERIAL PRIMARY KEY,\n  name VARCHAR(255) NOT NULL,\n  email VARCHAR(255) NOT NULL,\n  street VARCHAR(255) NOT NULL,\n  city VARCHAR(255) NOT NULL,\n  state VARCHAR(255) NOT NULL,\n  postal_code VARCHAR(10) NOT NULL,\n  product_name VARCHAR(255) NOT NULL,\n  quantity INTEGER NOT NULL,\n  order_date DATE NOT NULL,\n  priority VARCHAR(255) NOT NULL\n);\n```\n\nThen if you run `\\dt`, you'll be able to see a table named **orders**\n\n4. Now, let's exec into kafka container \u0026 create a topic named `product_orders`. Run the following commands:\n\n```shell\ndocker ps -a\n\n\u003c!-- copy the container id of kafka container --\u003e\n\ndocker exec -it \u003ccontainer_id\u003e /bin/bash\n\ncd /opt/bitnami/kafka/bin\n\n./kafka-topics.sh --create --topic product_orders --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1\n\n\u003c!-- The following command will list the topics --\u003e\n\n./kafka-topics.sh --list --bootstrap-server localhost:9092\n\n\u003c!-- To run the consumer and see what events it is listening to in real-time, run --\u003e\nkafka-console-consumer.sh --topic \"product_orders\" --from-beginning --bootstrap-server localhost:9092\n\n\u003c!-- To run the producer for testing purposes and publish dummy orders, run --\u003e\nkafka-console-producer.sh --topic \"product_orders\" --bootstrap-server localhost:9092\n```\n\n5. Now we have the server, broker as well as the database up and running.\n\n\u003cimg src=\"images/containers.png\" alt=\"Containers running in docker\" /\u003e\n\n6. If visit [127.0.0.1:5000](http://127.0.0.1:5000), you'll find the flask server running in this port. Now open postman and send a `POST` request to _127.0.0.1:5000/orders_ with the following payload:\n\n```json\n{\n\t\"order_id\": 90123,\n\t\"customer\": {\n\t\t\"name\": \"James Clark\",\n\t\t\"email\": \"james@gmail.com\",\n\t\t\"address\": {\n\t\t\t\"street\": \"Avenue Street\",\n\t\t\t\"city\": \"Illinois\",\n\t\t\t\"state\": \"Chicago\",\n\t\t\t\"postal_code\": \"768987\"\n\t\t}\n\t},\n\t\"product_name\": \"Protein Powder\",\n\t\"quantity\": 5,\n\t\"order_date\": \"2023-06-09\",\n\t\"priority\": \"high\"\n}\n```\n\n7. When we send a `POST` request to `/orders` endpoint, the following set of events are carried out:\n\n- the json is extracted from request body\n- then, it will call `publish_to_kafka_topic` function to initialize a producer and send data to the kafka topic _product_orders_\n- Since the consumer is always running in a separate thread, it'll be able to listen whenever any new order is added to the _product_orders_ topic via the `consume_and_send_emails` function\n- In this function, first the consumer is initialized and then we iterate through the `consumer` variable.\n- We then check the `priority` of the order. If it is high then we call the `send_email` function and then call the `save_order_to_postgres(order)` to write to db else directly write to db if priority is medium or low.\n- Once it is added to database via consumer, we then extract the `city` from the `order` object and invoke `create_topic_if_not_exists` function. This will create a new topic with the name of the city if not already present.\n- Once the respective city topic is created, `publish_to_kafka_topic` function will be called which will publish the order to it's respective city topic.\n- At last, the api will return a success message\n\n8. If we send a `GET` request to `/orders` endpoint, it will simply establish connection with the db and then get all the orders and send it in the form of a json.\n\n### 📉 Relevant Screenshots:\n\n1. _Sending a POST request to **/orders** via Postman_\n   \u003cimg src=\"images/screenshot1.png\" alt=\"Postman Request\" /\u003e\n\n2. _Logs of Docker Compose(Kafka, zookeeper and database) and Flask server_\n   \u003cimg src=\"images/screenshot2.png\" alt=\"Docker Compose and Flask Logs\" /\u003e\n\n3. _Exec into db and query the data added by consumer_\n   \u003cimg src=\"images/screenshot3.png\" alt=\"Postgres DB\" /\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fciphertron%2Fkafka-streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fciphertron%2Fkafka-streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fciphertron%2Fkafka-streamer/lists"}