{"id":16283444,"url":"https://github.com/eli64s/pyflink-poc","last_synced_at":"2025-04-08T18:25:39.514Z","repository":{"id":150159162,"uuid":"618273898","full_name":"eli64s/pyflink-poc","owner":"eli64s","description":"PyFlink data stream processing utilities 🐿","archived":false,"fork":false,"pushed_at":"2023-06-16T08:53:42.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-14T14:42:30.788Z","etag":null,"topics":["apache-flink","data-stream-processing","data-streaming","data-streams","pyflink","real-time-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eli64s.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-24T05:40:09.000Z","updated_at":"2023-12-07T15:20:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"38e9196a-5e60-486e-a96f-d9297eaedac5","html_url":"https://github.com/eli64s/pyflink-poc","commit_stats":null,"previous_names":["eli64s/pyflink-poc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eli64s%2Fpyflink-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eli64s%2Fpyflink-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eli64s%2Fpyflink-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eli64s%2Fpyflink-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eli64s","download_url":"https://codeload.github.com/eli64s/pyflink-poc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247900174,"owners_count":21015027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-flink","data-stream-processing","data-streaming","data-streams","pyflink","real-time-data"],"created_at":"2024-10-10T19:13:28.331Z","updated_at":"2025-04-08T18:25:39.489Z","avatar_url":"https://github.com/eli64s.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n\u003ch1 align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-markdown-open.svg\" width=\"100\" /\u003e\n\u003cbr\u003e\nFlinkFlow\n\u003c/h1\u003e\n\u003ch3 align=\"center\"\u003e📍 Real-time stream processing wiht PyFlink.\u003c/h3\u003e\n\u003ch3 align=\"center\"\u003e🚀 Developed with the software and tools below.\u003c/h3\u003e\n\u003cp align=\"center\"\u003e\n\n\u003cimg src=\"https://img.shields.io/badge/Apache%20Flink-E6526F.svg?style=for-the-badge\u0026logo=Apache-Flink\u0026logoColor=white\" alt=\"py\" /\u003e\n\u003cimg src=\"https://img.shields.io/badge/Apache%20Kafka-231F20.svg?style=for-the-badge\u0026logo=Apache-Kafka\u0026logoColor=white\" alt=\"pyflink\" /\u003e\n\u003cimg src=\"https://img.shields.io/badge/AIOHTTP-2C5BB4.svg?style=for-the-badge\u0026logo=AIOHTTP\u0026logoColor=white\" alt=\"\" /\u003e\n\u003cimg src=\"https://img.shields.io/badge/Python-3776AB.svg?style=for-the-badge\u0026logo=Python\u0026logoColor=white\" alt=\"aioresponses\" /\u003e\n\u003cimg src=\"https://img.shields.io/badge/pandas-150458.svg?style=for-the-badge\u0026logo=pandas\u0026logoColor=white\" alt=\"aiohttp\" /\u003e\n\u003cimg src=\"https://img.shields.io/badge/GNU%20Bash-4EAA25.svg?style=for-the-badge\u0026logo=GNU-Bash\u0026logoColor=white\" alt=\"asyncio\" /\u003e\n\u003cimg src=\"https://img.shields.io/badge/Markdown-000000.svg?style=for-the-badge\u0026logo=Markdown\u0026logoColor=white\" alt=\"pack\" /\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n---\n## 📚 Table of Contents\n- [📚 Table of Contents](#-table-of-contents)\n- [📍Overview](#overview)\n- [🔮 Feautres](#-feautres)\n- [⚙️ Project Structure](#️-project-structure)\n- [💻 Modules](#-modules)\n- [🚀 Getting Started](#-getting-started)\n  - [✅ Prerequisites](#-prerequisites)\n  - [💻 Installation](#-installation)\n  - [🤖 Using FlinkFlow](#-using-flinkflow)\n  - [🧪 Running Tests](#-running-tests)\n- [🛠 Future Development](#-future-development)\n- [🤝 Contributing](#-contributing)\n- [🪪 License](#-license)\n- [🙏 Acknowledgments](#-acknowledgments)\n\n---\n\n## 📍Overview\n\nFlinkFlow is a repository for building real-time data processing apps with PyFlink.\n\n## 🔮 Feautres\n\n\u003e `[📌  INSERT-PROJECT-FEATURES]`\n\n---\n\n\u003cimg src=\"https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-github-open.svg\" width=\"80\" /\u003e\n\n## ⚙️ Project Structure\n\n```bash\n.\n├── README.md\n├── conf\n│   ├── conf.toml\n│   └── flink-config.yaml\n├── data\n│   └── data.csv\n├── requirements.txt\n├── scripts\n│   ├── clean.sh\n│   └── run.sh\n├── setup\n│   └── setup.sh\n├── setup.py\n└── src\n    ├── alerts_handler.py\n    ├── consumer.py\n    └── logger.py\n\n6 directories, 12 files\n```\n---\n\n\u003cimg src=\"https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-src-open.svg\" width=\"80\" /\u003e\n\n## 💻 Modules\n\u003cdetails closed\u003e\u003csummary\u003eScripts\u003c/summary\u003e\n\n| File     | Summary                                                                                                                                                                                                        |\n|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| run.sh   | This code is a Bash script that starts a Flink cluster, submits a PyFlink job, and then stops the Flink cluster.                                                                                               |\n| clean.sh | This code is a Bash script that cleans up files and directories related to Python, Jupyter Notebooks, and pytest. It deletes Python cache files, build artifacts, Jupyter notebook checkpoints, and log files. |\n\n\u003c/details\u003e\n\n\u003cdetails closed\u003e\u003csummary\u003eSrc\u003c/summary\u003e\n\n| File              | Summary                                                                                                                                                                                                                          |\n|:------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| alerts_handler.py | This code is a REST API alert handler for the Flink consumer. It buffers alerts and sends them to the API in batches using aiohttp, and serializes them using Apache Avro.                                                       |\n| logger.py         | Logger is a class for the project that provides logging capabilities with colored output and different log levels.                                                                                                               |\n| consumer.py       | This code is a Python script that uses Apache Flink to process streaming data. It creates a StreamExecutionEnvironment, sets the parallelism, time characteristic, and checkpointing mode, and creates a StreamTableEnvironment. |\n\n\u003c/details\u003e\n\u003chr /\u003e\n\n## 🚀 Getting Started\n\n### ✅ Prerequisites\n\nBefore you begin, ensure that you have the following prerequisites installed:\n\u003e `[📌  INSERT-PROJECT-PREREQUISITES]`\n\n### 💻 Installation\n\n1. Clone the FlinkFlow repository:\n```sh\ngit clone https://github.com/eli64s/FlinkFlow\n```\n\n2. Change to the project directory:\n```sh\ncd FlinkFlow\n```\n\n3. Install the dependencies:\n```sh\npip install -r requirements.txt\n```\n\n### 🤖 Using FlinkFlow\n\n```sh\npython main.py\n```\n\n### 🧪 Running Tests\n```sh\n#run tests\n```\n\n\u003chr /\u003e\n\n## 🛠 Future Development\n- [X] [📌  COMPLETED-TASK]\n- [ ] [📌  INSERT-TASK]\n- [ ] [📌  INSERT-TASK]\n\n\n---\n\n## 🤝 Contributing\nContributions are always welcome! Please follow these steps:\n1. Fork the project repository. This creates a copy of the project on your account that you can modify without affecting the original project.\n2. Clone the forked repository to your local machine using a Git client like Git or GitHub Desktop.\n3. Create a new branch with a descriptive name (e.g., `new-feature-branch` or `bugfix-issue-123`).\n```sh\ngit checkout -b new-feature-branch\n```\n4. Make changes to the project's codebase.\n5. Commit your changes to your local branch with a clear commit message that explains the changes you've made.\n```sh\ngit commit -m 'Implemented new feature.'\n```\n6. Push your changes to your forked repository on GitHub using the following command\n```sh\ngit push origin new-feature-branch\n```\n7. Create a pull request to the original repository.\nOpen a new pull request to the original project repository. In the pull request, describe the changes you've made and why they're necessary. \nThe project maintainers will review your changes and provide feedback or merge them into the main branch.\n\n---\n\n## 🪪 License\n\nThis project is licensed under the `[📌  INSERT-LICENSE-TYPE]` License. See the [LICENSE](https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-license-to-a-repository) file for additional info.\n\n---\n\n## 🙏 Acknowledgments\n\n[📌  INSERT-DESCRIPTION]\n\n\n---\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feli64s%2Fpyflink-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feli64s%2Fpyflink-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feli64s%2Fpyflink-poc/lists"}