https://github.com/mehanix/dhrw
🎢 IaaS visual editor to create & deploy data processing pipelines - python, rmq, react, meteorjs
https://github.com/mehanix/dhrw
computational-graph computational-graphs data-analysis data-engineering data-pipeline data-pipelines data-processing data-processing-and-analysis data-processing-pipelines data-processing-system data-science data-visualization docker-compose good-first-issue help-wanted meteorjs-application rabbitmq react-flow
Last synced: 3 months ago
JSON representation
🎢 IaaS visual editor to create & deploy data processing pipelines - python, rmq, react, meteorjs
- Host: GitHub
- URL: https://github.com/mehanix/dhrw
- Owner: mehanix
- Created: 2024-07-06T21:34:47.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-01-28T22:11:16.000Z (5 months ago)
- Last Synced: 2025-02-09T23:41:12.857Z (4 months ago)
- Topics: computational-graph, computational-graphs, data-analysis, data-engineering, data-pipeline, data-pipelines, data-processing, data-processing-and-analysis, data-processing-pipelines, data-processing-system, data-science, data-visualization, docker-compose, good-first-issue, help-wanted, meteorjs-application, rabbitmq, react-flow
- Language: JavaScript
- Homepage:
- Size: 1.88 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 🎢 DHRW ~ Visual Data Pipelines
DHRW is a proof-of-concept for a low-code, IaaS platform to let you visually create data processing pipelines. [See it in action!](https://www.youtube.com/watch?v=6WjEC7rRUQU)
## ✅ Features
#### ⌨️ **Write** your Python functions (or use functions written by others and imported in the app)
#### 🔗 **Add** them to a GitHub repository [(example repo)](https://github.com/mehanix/dhrw-functions)
#### 📥 **Import** them in the app
#### 💫 **Connect** them to create _execution graphs_ - a sequence of functions that you can pipe data through
#### 🚀 Deploy in the cloud
With just the press of one button, you can **auto-magically ✨ turn a series of functions into a network of Docker containers** that are provisioned with your code and wait to receive and process your data.
#### 📑 Process data
You can then **upload** CSV files with your data and **receive** the processing results right in your browser. Supports both **text** as well as **image** outputs (for plots).
#### 👨🏻💻 Live edit & redeploy
🌐 Pairs really well with GitHub Codespaces to completely **develop right in the browser**. Edit your function code using VSCode in the browser, reupload it in the graph, and redeploy in two clicks. 🙌
## 🐳 Get it running
Setup is a bit tricky since this is a proof-of-concept. Here are the steps to get it running locally:
##### Main setup
1. Clone repository
2. `docker compose up`
3. Manually install python dependencies (found in `worker/requirements.txt` on the meteor container) - known issue, they need to be synchronised 😅
4. Get an access token for your function repository and add it in settings.json, replacing the one that is there (expired, used to point to [this repo](https://github.com/mehanix/dhrw-functions)).
##### RabbitMQ setup
1. enter localhost:15672 and add workers exchange of type `topic`
2. also add a routing key named `worker_reply.*` to `server_responses` queue on this exchange. - known issue, they need to be added manually for now.
##### Get in the app
Enter the app via `localhost:3000`.
## 🤖 Tech stack
## 📜 Docs
Read the docs [here](https://github.com/mehanix/dhrw/blob/32abfdf1329d02b66804a914cff42167a04d6be6/docs%20(in%20romanian).pdf) (in Romanian, this was my master's thesis project).
# 🎬 Demo video
[](https://www.youtube.com/watch?v=6WjEC7rRUQU)