{"id":23123239,"url":"https://github.com/lupusruber/rnmp_homework3","last_synced_at":"2026-04-30T10:32:34.525Z","repository":{"id":268145881,"uuid":"903452370","full_name":"lupusruber/RNMP_homework3","owner":"lupusruber","description":"A Spark Streaming and Kafka-based project for processing health data in real-time. Includes a machine learning pipeline for predictions, Dockerized infrastructure, and scripts for data ingestion, model training, and streaming pipelines.","archived":false,"fork":false,"pushed_at":"2024-12-21T18:44:26.000Z","size":41,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-09T15:43:47.431Z","etag":null,"topics":["docker-compose","kafka","model-selection-and-evaluation","spark-streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lupusruber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-14T16:36:24.000Z","updated_at":"2024-12-23T18:43:09.000Z","dependencies_parsed_at":"2024-12-21T19:24:48.083Z","dependency_job_id":null,"html_url":"https://github.com/lupusruber/RNMP_homework3","commit_stats":null,"previous_names":["lupusruber/rnmp_homework3"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2FRNMP_homework3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2FRNMP_homework3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2FRNMP_homework3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2FRNMP_homework3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lupusruber","download_url":"https://codeload.github.com/lupusruber/RNMP_homework3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247119352,"owners_count":20886735,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker-compose","kafka","model-selection-and-evaluation","spark-streaming"],"created_at":"2024-12-17T07:33:14.209Z","updated_at":"2026-04-30T10:32:29.490Z","avatar_url":"https://github.com/lupusruber.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Рударење на масивни податоци: Домашна работа 3\n\nThis project is designed to process health data using Spark Streaming, Kafka, and machine learning. \nThe main pipeline reads health indicators from Kafka, makes predictions using a trained model, and sends the results back to Kafka.\n\n## Project Structure\n\n```\n.\n├── checkpoints                           # Directory for storing Spark checkpoint data\n├── data                                  # Data-related files and scripts\n│   ├── get_data_script.sh                # Script to fetch data\n│   └── split_data.py                     # Script to split data into train/test sets\n├── infrastructure                        # Infrastructure configuration files\n│   ├── kafka-docker-compose.yml          # Docker Compose for Kafka\n│   └── spark-docker-compose.yml          # Docker Compose for Spark\n├── model                                 # Model-related files and scripts\n│   ├── best_model_script.py              # Script to find the best model\n├── poetry.lock                           # Poetry lockfile for dependency management\n├── pyproject.toml                        # Poetry configuration file\n├── README.md                             # Main project documentation\n├── main_shell_script.sh                  # Main project script, run this to build everything needed\n└── scripts                               # Main scripts\n    ├── consume_kafka_msgs.py             # Script to consume messages from Kafka\n    ├── make_predictions_spark_script.py  # Main Spark script for predictions\n    ├── produce_kafka_msgs.py             # Script to produce messages to Kafka\n    └── start_stream.sh                   # Shell script to start the streaming pipeline\n```\n\n## Usage Instructions\n\n### 1. Setting up the Environment\n- Use the provided Docker Compose files to set up Kafka and Spark:\n  ```bash\n  docker-compose -f infrastructure/kafka-docker-compose.yml up -d\n  docker-compose -f infrastructure/spark-docker-compose.yml up -d\n  ```\n\n### 2. Preparing the Data\n- Use `data/get_data_script.sh` to fetch the raw data.\n- Split the data into training and testing data for the model (offline data) and data for making the predictions (online data) using `data/split_data.py`.\n\n### 3. Training the Model\n- Run `model/best_model_script.py` to train models and save the best one as `model/model.pkl`.\n- This script trains 3 models (Logistic Regression, XGBoost and Random Forest) using 3 fold CV for finding the best model.\n\n\n### 4. Making the predictions using Spark Structured Streaming\n- Use `scripts/make_predictions_spark_script.py` script as a spark streaming job\n- This script creates a Spark Session, reads the steaming data, adds a prediction column to each row and sends the resultant rows to another Kafka topic.\n\n\n### 5. The whole streaming pipeline\n- Produce messages to Kafka using `scripts/produce_kafka_msgs.py`.\n- Run the Spark streaming script `scripts/make_predictions_spark_script.py` to process the data and produce predictions.\n- Consume the prediction results from Kafka using `scripts/consume_kafka_msgs.py`.\n\n\n## Dependencies\n- Python 3.10 or higher\n- Poetry for dependency management\n- Docker and Docker Compose\n  - Apache Spark\n  - Apache Kafka\n\n## Commands\n- Generate the offline.csv and online.csv files:\n```bash\n  source data/get_data_script.sh\n```\n\n- Find the best model and save it as a pkl file:\n```bash\n  python model/best_model_script.py\n```\n\n- Produce the needed Kafka messages for making predictions:\n```bash\n  python scripts/consume_kafka_msgs.py\n  ```\n\n- Start the pipeline inside the spark container:\n```bash\n  source scripts/start_stream.sh\n  ```\n\n- View Kafka messages:\n```bash\n  python scripts/consume_kafka_msgs.py\n  ```\n\n## Notes\n- There is one script that runs everything needed to get the data, find the best model and make the streaming pipelines, as well as start the docker containers.\n```bash\n  source main_shell_script.sh\n  ```\n- Install the needed python packages using poetry\n- Make sure all containers are running on the same Docker network for communication.\n- Update Kafka broker addresses in the scripts as necessary.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupusruber%2Frnmp_homework3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flupusruber%2Frnmp_homework3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupusruber%2Frnmp_homework3/lists"}