{"id":19373741,"url":"https://github.com/waelson/stream-processing-with-faust","last_synced_at":"2025-02-24T15:20:44.483Z","repository":{"id":120617191,"uuid":"309186460","full_name":"Waelson/Stream-Processing-With-Faust","owner":"Waelson","description":"This repository is a demo how to process data streaming using Faust Library and Kafka.","archived":false,"fork":false,"pushed_at":"2020-11-24T17:53:28.000Z","size":323,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-07T02:04:56.163Z","etag":null,"topics":["apache-kafka","bash","docker","docker-compose","faust","kafdrop","kafka","kafka-cluster","python","python-application","streaming","topic","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Waelson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-01T20:54:25.000Z","updated_at":"2020-11-25T17:24:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"74882eb5-01ad-4dbb-97aa-9b449dc2a752","html_url":"https://github.com/Waelson/Stream-Processing-With-Faust","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Waelson%2FStream-Processing-With-Faust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Waelson%2FStream-Processing-With-Faust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Waelson%2FStream-Processing-With-Faust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Waelson%2FStream-Processing-With-Faust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Waelson","download_url":"https://codeload.github.com/Waelson/Stream-Processing-With-Faust/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240501242,"owners_count":19811574,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","bash","docker","docker-compose","faust","kafdrop","kafka","kafka-cluster","python","python-application","streaming","topic","zookeeper"],"created_at":"2024-11-10T08:31:25.294Z","updated_at":"2025-02-24T15:20:44.460Z","avatar_url":"https://github.com/Waelson.png","language":"Python","readme":"## Sending and Reading Messages from Apache Kafka\n\nThis repository is a simple application that connect to Apache Kafka, create a topic, send and read messages.\n\n---\n\n### 1. Dependencies\n\n#### 1.1. Python\n\n[Download and install the Python](https://www.python.org/downloads/).\n\n#### 1.2 Pip\n\n[Download and install the Pip](https://pip.pypa.io/en/stable/installing/)\n\n#### 1.3. Docker Desktop\n\nYou would require you to install Docker Desktop to create containers for individual microservices. Refer the following links for instructions\n\n- [macOS](https://docs.docker.com/docker-for-mac/install/),\n- [Windows 10 64-bit: Pro, Enterprise, or Education](https://docs.docker.com/docker-for-windows/install/),\n- [Windows 10 64-bit Home](https://docs.docker.com/toolbox/toolbox_install_windows/).\n- You can find installation instructions for other operating systems at: https://docs.docker.com/install/\n\n---\n\n### 2. Instructions\n\nTo run any project code, you will have to set up a virtual environment with project dependencies. All of following instructions are to be completed via a terminal/command line prompt.\n\n#### 2.1. Clone the Repository\n\nFirst you need to clone this repository\n\n```bash\ngit clone https://github.com/Waelson/Simple-Kafka-Python-Application.git\n```\n\n#### 2.2. Install Python Dependencies\n\nThis project has two dependencies. It is setting in 'requirements.txt' file. To install dependency use follow command:\n\n```bash\npip install r- requirements.txt\n```\n\n#### 2.3. Start Kafka (run on Docker)\n\nInto 'scripts' folder run the following command:\n\n```bash\n./start_kafka.sh\n```\n\nor\n\n```bash\ndocker-componse up -d\n```\n\n#### 2.3. Running Python Application\n\nThe script bellow create a topic if it don't exists. Then, it send and read messages.\n\n```bash\npython app_topic.py\n```\n\nIn this script you can to simulate message producers sync and async\n\n```bash\npython app_producer.py\n```\n\nThe script bellow set a producer with several parameters that impact the producer's performance.\n\n```bash\npython app_param_producer.py\n```\n\n#### 2.5. Output console\n\nIf you did all correctly, you will get an output similarly to image below.\n\n```bash\npython app_topic.py\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/Waelson/kafka-python/main/images/output_topic.png\"\u003e\n\n```bash\npython app_producer.py\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/Waelson/kafka-python/main/images/output_producer.png\"\u003e\n\n#### 2.5. Accessing KafkaDrop\n\n[Kafdrop](https://github.com/obsidiandynamics/kafdrop) is an open source project that allow you visualize informations about your Kafka Cluster. When you start Kafka (step 2.3), you will be running Kafdrop as well. To access it use the URL bellow. Pay attention: There are others tools that allow you manager Kafka Cluster. Please, consider to visit [CMAK - Cluster Manager for Apache Kafka](https://github.com/yahoo/CMAK) for know more about this fantastic tool.\n\n[http://localhost:9000](http://localhost:9000)\n\n\u003cimg src=\"https://raw.githubusercontent.com/Waelson/kafka-python/main/images/output_kafkadrop.png\"\u003e\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaelson%2Fstream-processing-with-faust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwaelson%2Fstream-processing-with-faust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaelson%2Fstream-processing-with-faust/lists"}