https://github.com/lorenzofelletti/edgecloudapproximate

https://github.com/lorenzofelletti/edgecloudapproximate

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/lorenzofelletti/edgecloudapproximate
• Owner: lorenzofelletti
• Created: 2022-11-11T10:36:51.000Z (about 2 years ago)
• Default Branch: master
• Last Pushed: 2024-03-25T23:01:49.000Z (9 months ago)
• Last Synced: 2024-03-26T23:43:41.548Z (9 months ago)
• Language: Rust
• Size: 32.8 MB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# EdgeApproximate

## Environment Setup

- Install Docker Desktop

- Run `docker-compose up -d` in the root directory of the project

- Enjoy!

## Containers Configuration

- There are 5 containers in total

  - One Spark Master

  - One Spark Worker

  - One Zookeeper node

  - Two Kafka nodes

    - Kafka (Main Kafka broker)

    - Edge1 (Edge node).

On the Spark master node, runs a pyspark script that continuously reads from the DATAOUT topic(s) of the Kafka broker and writes the processed data to a local directory. Also on the Spark master node, runs a Flask server that serves the processed data to the frontend. The Flask server is accessible at `localhost:5000`.

On the Zookeeper node, runs a Zookeeper server (required by Kafka).

On the Kafka node, two programs runs:

- `kafka-producer-rs` -- A Rust program that reads from a local csv the data to distribute to the Kafka nodes via a DATAIN topic; simulating the data arriving to the edge nodes.

- `kafka-edge-rs` -- A Rust program that reads from the DATAIN topic and writes to one or more DATAOUT topics (depending on the configuration); simulating being an edge node.

On the Edge1 node, runs the same `kafka-edge-rs` program as the Kafka node, but withouth the `kafka-producer-rs` program (required only on one node).