{"id":19282972,"url":"https://github.com/jorkdev/python-kafka-reactive-streaming","last_synced_at":"2025-02-23T22:25:50.413Z","repository":{"id":225542558,"uuid":"766249930","full_name":"JorkDev/Python-Kafka-Reactive-Streaming","owner":"JorkDev","description":"Python-based Kafka application for reactive data streaming, enabling real-time processing and scalability.","archived":false,"fork":false,"pushed_at":"2024-03-13T17:09:04.000Z","size":19,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-05T17:25:40.796Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JorkDev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-02T18:38:07.000Z","updated_at":"2024-03-02T20:31:27.000Z","dependencies_parsed_at":"2024-03-13T17:51:34.991Z","dependency_job_id":"240976d0-da12-419e-9553-68a9e4e3ef8b","html_url":"https://github.com/JorkDev/Python-Kafka-Reactive-Streaming","commit_stats":null,"previous_names":["jorkdev/python-kafka-reactive-streaming"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorkDev%2FPython-Kafka-Reactive-Streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorkDev%2FPython-Kafka-Reactive-Streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorkDev%2FPython-Kafka-Reactive-Streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorkDev%2FPython-Kafka-Reactive-Streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JorkDev","download_url":"https://codeload.github.com/JorkDev/Python-Kafka-Reactive-Streaming/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240387032,"owners_count":19793269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T21:29:10.783Z","updated_at":"2025-02-23T22:25:50.369Z","avatar_url":"https://github.com/JorkDev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reactive Data Streaming App with Python and Apache Kafka\n\nEver wanted to receive real-time updates from services that don't offer live notifications? Wondered if you could monitor an online store for price drops or keep track of comments on YouTube videos? This project demonstrates how to transform static data sources into dynamic, reactive systems.\n\n## Overview\n\nThis project showcases a solution for transforming static data from YouTube's REST API into a reactive system using Python and Apache Kafka. With this solution, you can:\n\n- Utilize Python to retrieve and process data from static web APIs\n- Stream the processed data in real-time using Apache Kafka\n- Employ ksqlDB to analyze incoming data streams and detect significant changes\n- Deliver customized, live notifications via Telegram to keep users informed about updates and changes.\n\n## How It Works\n\n1. **Fetching and Processing Data**: Python scripts fetch data from the YouTube REST API and preprocess it for streaming.\n\n2. **Streaming Data with Apache Kafka**: Processed data is streamed into Kafka topics to enable real-time data streaming.\n\n3. **Processing Data with ksqlDB**: ksqlDB is used to analyze the incoming data stream, identifying important events or changes.\n\n4. **Sending Custom Notifications**: Customized notifications are sent via Telegram to notify users of relevant updates or changes.\n\n## Getting Started\n\nTo start using this project:\n\n1. **Clone the Repository**: Clone this repository to your local machine.\n\n2. **Set Up Dependencies**: Install the required dependencies by running `pip install -r requirements.txt`.\n\n3. **Set Up Apache Kafka**: Install and configure Apache Kafka on your machine. Refer to the Kafka documentation for instructions.\n\n4. **Get a Google API Key**: To access YouTube's REST API, you'll need a Google API key. Follow these steps to get one:\n   - Go to the [Google Developers Console](https://console.developers.google.com/)\n   - Create a new project (or select an existing one)\n   - Enable the YouTube Data API v3 for your project\n   - Create credentials and select \"API Key\"\n   - Copy the generated API key and paste it into config.py\n\n5. **Run the Application**: Execute the necessary Python scripts to start the application. Ensure to configure API keys and other settings as required.\n\n6. **Explore and Customize**: Feel free to explore the codebase and customize it to suit your needs. You can modify data sources, processing logic, and notification methods as necessary.\n\n## Setting Up Kafka Cluster and ksqlDB Cluster with Confluent Cloud\n\nTo get started with this project using Confluent Cloud, you'll need to set up a Kafka cluster for real-time data streaming and a ksqlDB cluster for stream processing. Follow the steps below to create both clusters:\n\n### Kafka Cluster Setup\n\n1. **Sign Up for Confluent Cloud**: If you haven't already, sign up for a Confluent Cloud account at [Confluent Cloud](https://www.confluent.io/confluent-cloud/).\n\n2. **Create a Kafka Cluster**: Log in to your Confluent Cloud account and navigate to the Clusters section. Click on \"Create Cluster\" and follow the prompts to create a new Kafka cluster. Choose your desired cloud provider, region, and cluster settings.\n\n3. **Generate API Key and Secret**: Once your Kafka cluster is created, generate an API key and secret for accessing the cluster. Navigate to \"Cluster Settings\" \u003e \"API Access\" and click on \"Create Key\" to generate the credentials.\n\n4. **Download Configuration File**: Download the configuration file for your Kafka cluster from the Confluent Cloud UI. This file contains the necessary connection information for connecting to your Kafka cluster programmatically.\n\n### ksqlDB Cluster Setup\n\n1. **Create a ksqlDB App**: Navigate to the \"ksqlDB\" section in your Confluent Cloud account and click on \"Add ksqlDB App\". Follow the prompts to create a new ksqlDB cluster. Choose your desired cloud provider, region, and cluster settings.\n\n2. **Enable Schema Registry**: If you plan to use Avro serialization with your ksqlDB cluster, enable Schema Registry for your ksqlDB app. Navigate to \"ksqlDB\" \u003e \"Cluster Details\" \u003e \"Schema Registry\" and enable the feature.\n\n3. **Access ksqlDB UI**: Once your ksqlDB cluster is created, access the ksqlDB UI by clicking on \"ksqlDB UI\" in the Confluent Cloud UI. This allows you to interactively query and process streams of data using ksqlDB.\n\n4. **Connect ksqlDB to Kafka**: Use the configuration file downloaded earlier to configure the connection between your ksqlDB cluster and Kafka cluster. Follow the instructions provided in the Confluent Cloud documentation to set up the connection.\n\nBy following these steps, you'll have a Kafka cluster for data streaming and a ksqlDB cluster for stream processing set up and ready to use with this project on Confluent Cloud.\n\n## Contributions\n\nWe welcome contributions from the community to improve this project! If you're interested in contributing, here's how you can get started:\n\n1. **Fork the Repository**: Start by forking this repository to your own GitHub account.\n2. **Clone the Repository**: Clone the forked repository to your local machine using the `git clone` command.\n3. **Create a Branch**: Create a new branch for your contributions using `git checkout -b feature/new-feature` (replace `new-feature` with a descriptive branch name).\n4. **Make Changes**: Make your desired changes to the codebase.\n5. **Test Your Changes**: Ensure that your changes don't break existing functionality and add any necessary tests.\n6. **Commit Your Changes**: Commit your changes with descriptive commit messages.\n7. **Push Changes**: Push your changes to your forked repository with `git push origin feature/new-feature`.\n8. **Create a Pull Request**: Submit a pull request from your branch to the main repository. Be sure to provide a detailed description of your changes and any related issues.\n9. **Review and Collaborate**: Collaborate with maintainers and reviewers to address any feedback and ensure your changes meet the project's standards.\n10. **Celebrate**: Your contribution is valuable! Thank you for your effort in improving this project.\n\nIf you have any questions or need assistance with the contribution process, feel free to reach out to us or refer to the [GitHub documentation](https://docs.github.com/en/get-started) for more guidance.\n\nThis project is maintained by me, for any questions or inquiries, please contact [jorkaefdev@gmail.com](mailto:jorkaefdev@gmail.com).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjorkdev%2Fpython-kafka-reactive-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjorkdev%2Fpython-kafka-reactive-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjorkdev%2Fpython-kafka-reactive-streaming/lists"}