Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/eli64s/pyflink-poc
PyFlink data stream processing utilities 🐿
https://github.com/eli64s/pyflink-poc
apache-flink data-stream-processing data-streaming data-streams pyflink real-time-data
Last synced: 4 days ago
JSON representation
PyFlink data stream processing utilities 🐿
- Host: GitHub
- URL: https://github.com/eli64s/pyflink-poc
- Owner: eli64s
- Created: 2023-03-24T05:40:09.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-06-16T08:53:42.000Z (over 1 year ago)
- Last Synced: 2024-11-12T09:09:06.887Z (about 1 month ago)
- Topics: apache-flink, data-stream-processing, data-streaming, data-streams, pyflink, real-time-data
- Language: Python
- Homepage:
- Size: 32.2 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
FlinkFlow
📍 Real-time stream processing wiht PyFlink.
🚀 Developed with the software and tools below.
---
## 📚 Table of Contents
- [📚 Table of Contents](#-table-of-contents)
- [📍Overview](#overview)
- [🔮 Feautres](#-feautres)
- [⚙️ Project Structure](#️-project-structure)
- [💻 Modules](#-modules)
- [🚀 Getting Started](#-getting-started)
- [✅ Prerequisites](#-prerequisites)
- [💻 Installation](#-installation)
- [🤖 Using FlinkFlow](#-using-flinkflow)
- [🧪 Running Tests](#-running-tests)
- [🛠 Future Development](#-future-development)
- [🤝 Contributing](#-contributing)
- [🪪 License](#-license)
- [🙏 Acknowledgments](#-acknowledgments)
---
## 📍Overview
FlinkFlow is a repository for building real-time data processing apps with PyFlink.
## 🔮 Feautres
> `[📌 INSERT-PROJECT-FEATURES]`
---
## ⚙️ Project Structure
```bash
.
├── README.md
├── conf
│ ├── conf.toml
│ └── flink-config.yaml
├── data
│ └── data.csv
├── requirements.txt
├── scripts
│ ├── clean.sh
│ └── run.sh
├── setup
│ └── setup.sh
├── setup.py
└── src
├── alerts_handler.py
├── consumer.py
└── logger.py
6 directories, 12 files
```
---
## 💻 Modules
Scripts
| File | Summary |
|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| run.sh | This code is a Bash script that starts a Flink cluster, submits a PyFlink job, and then stops the Flink cluster. |
| clean.sh | This code is a Bash script that cleans up files and directories related to Python, Jupyter Notebooks, and pytest. It deletes Python cache files, build artifacts, Jupyter notebook checkpoints, and log files. |
Src
| File | Summary |
|:------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| alerts_handler.py | This code is a REST API alert handler for the Flink consumer. It buffers alerts and sends them to the API in batches using aiohttp, and serializes them using Apache Avro. |
| logger.py | Logger is a class for the project that provides logging capabilities with colored output and different log levels. |
| consumer.py | This code is a Python script that uses Apache Flink to process streaming data. It creates a StreamExecutionEnvironment, sets the parallelism, time characteristic, and checkpointing mode, and creates a StreamTableEnvironment. |
## 🚀 Getting Started
### ✅ Prerequisites
Before you begin, ensure that you have the following prerequisites installed:
> `[📌 INSERT-PROJECT-PREREQUISITES]`
### 💻 Installation
1. Clone the FlinkFlow repository:
```sh
git clone https://github.com/eli64s/FlinkFlow
```
2. Change to the project directory:
```sh
cd FlinkFlow
```
3. Install the dependencies:
```sh
pip install -r requirements.txt
```
### 🤖 Using FlinkFlow
```sh
python main.py
```
### 🧪 Running Tests
```sh
#run tests
```
## 🛠 Future Development
- [X] [📌 COMPLETED-TASK]
- [ ] [📌 INSERT-TASK]
- [ ] [📌 INSERT-TASK]
---
## 🤝 Contributing
Contributions are always welcome! Please follow these steps:
1. Fork the project repository. This creates a copy of the project on your account that you can modify without affecting the original project.
2. Clone the forked repository to your local machine using a Git client like Git or GitHub Desktop.
3. Create a new branch with a descriptive name (e.g., `new-feature-branch` or `bugfix-issue-123`).
```sh
git checkout -b new-feature-branch
```
4. Make changes to the project's codebase.
5. Commit your changes to your local branch with a clear commit message that explains the changes you've made.
```sh
git commit -m 'Implemented new feature.'
```
6. Push your changes to your forked repository on GitHub using the following command
```sh
git push origin new-feature-branch
```
7. Create a pull request to the original repository.
Open a new pull request to the original project repository. In the pull request, describe the changes you've made and why they're necessary.
The project maintainers will review your changes and provide feedback or merge them into the main branch.
---
## 🪪 License
This project is licensed under the `[📌 INSERT-LICENSE-TYPE]` License. See the [LICENSE](https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-license-to-a-repository) file for additional info.
---
## 🙏 Acknowledgments
[📌 INSERT-DESCRIPTION]
---