https://github.com/bytemedirk/distributed-computing-data-engineer

https://github.com/bytemedirk/distributed-computing-data-engineer

Last synced: 10 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/bytemedirk/distributed-computing-data-engineer
• Owner: ByteMeDirk
• License: mit
• Created: 2024-08-08T17:40:15.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-08-21T06:49:42.000Z (over 1 year ago)
• Last Synced: 2025-01-13T10:49:58.509Z (11 months ago)
• Language: Jupyter Notebook
• Size: 9.25 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# distributed-computing-data-engineer

This documentation outlines a Docker-based infrastructure for developing a Python framework for distributed computing.

The setup consists of one main node and two worker nodes, all running Ubuntu and capable of communicating with each

other.

```mermaid

graph TD

    subgraph Docker Network

        M[Main Node] <--> W1[Worker Node 1]

        M <--> W2[Worker Node 2]

        W1 <--> W2

    end

    

    subgraph Python Framework

        PM[Python Main Process] --> PM1[Task Distributor]

        PM --> PM2[Result Aggregator]

        PW1[Python Worker Process 1] --> PW11[Task Executor]

        PW2[Python Worker Process 2] --> PW21[Task Executor]

    end

    

    M --> PM

    W1 --> PW1

    W2 --> PW2

```