{"id":25193982,"url":"https://github.com/mehanix/dhrw","last_synced_at":"2026-05-02T19:33:10.925Z","repository":{"id":274710754,"uuid":"825128891","full_name":"mehanix/dhrw","owner":"mehanix","description":"🎢 IaaS visual editor to create \u0026 deploy data processing pipelines - python, rmq, react, meteorjs","archived":false,"fork":false,"pushed_at":"2025-01-28T22:11:16.000Z","size":1973,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T14:12:06.458Z","etag":null,"topics":["computational-graph","computational-graphs","data-analysis","data-engineering","data-pipeline","data-pipelines","data-processing","data-processing-and-analysis","data-processing-pipelines","data-processing-system","data-science","data-visualization","docker-compose","good-first-issue","help-wanted","meteorjs-application","rabbitmq","react-flow"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mehanix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-06T21:34:47.000Z","updated_at":"2025-01-28T22:12:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"c07bb9b0-fe67-40b6-8e86-aaecfdd20c23","html_url":"https://github.com/mehanix/dhrw","commit_stats":null,"previous_names":["mehanix/dhrw"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mehanix/dhrw","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehanix%2Fdhrw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehanix%2Fdhrw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehanix%2Fdhrw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehanix%2Fdhrw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mehanix","download_url":"https://codeload.github.com/mehanix/dhrw/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehanix%2Fdhrw/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32547645,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-graph","computational-graphs","data-analysis","data-engineering","data-pipeline","data-pipelines","data-processing","data-processing-and-analysis","data-processing-pipelines","data-processing-system","data-science","data-visualization","docker-compose","good-first-issue","help-wanted","meteorjs-application","rabbitmq","react-flow"],"created_at":"2025-02-09T23:40:59.412Z","updated_at":"2026-05-02T19:33:10.898Z","avatar_url":"https://github.com/mehanix.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# 🎢 DHRW ~ Visual Data Pipelines\nDHRW is a proof-of-concept for a low-code, IaaS platform to let you visually create data processing pipelines. [See it in action!](https://www.youtube.com/watch?v=6WjEC7rRUQU)\n\n![image](https://github.com/user-attachments/assets/b942acab-afef-4d76-96da-d33efb5b2cb2)\n\n## ✅ Features\n\n#### ⌨️ **Write** your Python functions (or use functions written by others and imported in the app)\n![image](https://github.com/user-attachments/assets/caa09e1f-7a80-49c4-9395-70cc496ada41)\n\n#### 🔗 **Add** them to a GitHub repository [(example repo)](https://github.com/mehanix/dhrw-functions)\n\n![image](https://github.com/user-attachments/assets/c37eae98-9ce9-40c1-b301-5563c8f4baec)\n\n#### 📥 **Import** them in the app\n\n![image](https://github.com/user-attachments/assets/3c8c895b-52d3-4753-b34a-f1bc5df9dc9f)\n  \n#### 💫 **Connect** them to create _execution graphs_ - a sequence of functions that you can pipe data through\n\n![image](https://github.com/user-attachments/assets/d409a55e-83ac-481d-99ba-6983098ec79d)\n\n\n#### 🚀 Deploy in the cloud\nWith just the press of one button, you can **auto-magically ✨ turn a series of functions into a network of Docker containers** that are provisioned with your code and wait to receive and process your data.\n\n![image](https://github.com/user-attachments/assets/c86ea1bc-0042-41be-8ccf-2d043cc2a843)\n\n#### 📑 Process data\n\nYou can then **upload** CSV files with your data and **receive** the processing results right in your browser. Supports both **text** as well as **image** outputs (for plots).\n\n![image](https://github.com/user-attachments/assets/22baa3ae-cc8f-4de9-82b3-d6aeb1d655e0)\n\n![image](https://github.com/user-attachments/assets/9d841bc7-bc48-4aa6-8aca-05bd37a80fb7)\n\n\n#### 👨🏻‍💻 Live edit \u0026 redeploy\n\n🌐  Pairs really well with GitHub Codespaces to completely **develop right in the browser**. Edit your function code using VSCode in the browser, reupload it in the graph, and redeploy in two clicks. 🙌 \n\n![image](https://github.com/user-attachments/assets/a6cbb4f1-dd37-457c-8c7a-de05ee37b6a8)\n\n## 🐳 Get it running \n\nSetup is a bit tricky since this is a proof-of-concept. Here are the steps to get it running locally:\n\n##### Main setup\n1. Clone repository\n2. `docker compose up`\n3. Manually install python dependencies (found in `worker/requirements.txt` on the meteor container) - known issue, they need to be synchronised 😅\n4. Get an access token for your function repository and add it in settings.json, replacing the one that is there (expired, used to point to [this repo](https://github.com/mehanix/dhrw-functions)).\n\n##### RabbitMQ setup\n1. enter localhost:15672 and add workers exchange of type `topic`\n2. also add a routing key named `worker_reply.*` to `server_responses` queue on this exchange. - known issue, they need to be added manually for now.\n\n##### Get in the app\nEnter the app via `localhost:3000`.\n\n\n\n## 🤖 Tech stack\n\n![image](https://github.com/user-attachments/assets/86bec09c-18d3-4248-84e8-bea5fd271525)\n\n## 📜 Docs \n\nRead the docs [here](https://github.com/mehanix/dhrw/blob/32abfdf1329d02b66804a914cff42167a04d6be6/docs%20(in%20romanian).pdf) (in Romanian, this was my master's thesis project).\n\n# 🎬 Demo video\n\n[![dhrw video](https://img.youtube.com/vi/6WjEC7rRUQU/0.jpg)](https://www.youtube.com/watch?v=6WjEC7rRUQU)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehanix%2Fdhrw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmehanix%2Fdhrw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehanix%2Fdhrw/lists"}