{"id":15056648,"url":"https://github.com/devonwintz/weather-stream","last_synced_at":"2026-01-27T18:06:39.815Z","repository":{"id":245881618,"uuid":"819017426","full_name":"devonwintz/weather-stream","owner":"devonwintz","description":"Real-time weather data pipeline using Flask, Kafka, Spark, Cassandra, and Grafana for generation, ingestion, processing, storage, and visualization respectively.","archived":false,"fork":false,"pushed_at":"2024-07-01T15:37:49.000Z","size":386,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T21:26:54.533Z","etag":null,"topics":["cassandra","grafana","kafka","python","spark-streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devonwintz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-23T14:40:57.000Z","updated_at":"2024-07-16T12:04:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"ea140595-be29-4507-95f9-76af7e992e9c","html_url":"https://github.com/devonwintz/weather-stream","commit_stats":{"total_commits":22,"total_committers":1,"mean_commits":22.0,"dds":0.0,"last_synced_commit":"99f9f28c2cdc2de3b0c5e8ce9640549a96595a66"},"previous_names":["devonwintz/weather-stream"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/devonwintz/weather-stream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devonwintz%2Fweather-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devonwintz%2Fweather-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devonwintz%2Fweather-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devonwintz%2Fweather-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devonwintz","download_url":"https://codeload.github.com/devonwintz/weather-stream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devonwintz%2Fweather-stream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28817796,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T18:01:38.485Z","status":"ssl_error","status_checked_at":"2026-01-27T18:01:27.499Z","response_time":168,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","grafana","kafka","python","spark-streaming"],"created_at":"2024-09-24T21:54:39.212Z","updated_at":"2026-01-27T18:06:39.799Z","avatar_url":"https://github.com/devonwintz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Weather Data Pipeline\n\n\u003cdiv align=\"center\"\u003e\n\n![](./assets/images/WeatherStream.png)\n\n\u003c/div\u003e\n\n## Overview\n\nThis project implements a weather data pipeline that generates, ingests, processes, stores, and visualizes weather data in real-time. It uses Flask for data generation, Kafka for data ingestion, Spark for processing, Cassandra for storage, and Grafana for visualization.\n\n## Directory Structure\n\u003cdiv align=\"center\"\u003e\n\n![](./assets/images/Project-Directory-Structure.PNG)\n\n\u003c/div\u003e\n\n## Components\n\n### WeatherMocker\n- **Purpose**: Generates random weather data simulating a real weather API.\n- **Main File**: `weather-mocker/app.py`\n- **Dependencies**: `weather-mocker/requirements.txt`\n\n### WeatherData\n- **Purpose**: Exposes an API to access weather data stored in Cassandra.\n- **Main File**: `weather-data/app.py`\n- **Dependencies**: `weather-data/requirements.txt`\n\n### Kafka\n- **Setup Script**: `kafka/kafka-setup.sh`\n- **Description**: Creates a Kafka topic if it does not exist.\n- **Producer**: `kafka/producer.py`\n\n### Cassandra\n- **Setup Script**: `cassandra/cassandra-init.sh`\n- **Description**: Initializes Cassandra by creating a keyspace and table if they do not exist.\n\n### Spark-Streaming\n- **Setup Script**: `spark-streaming/spark-setup.sh`\n- **Description**: Configures Spark for streaming and initializes necessary settings.\n- **Consumer**: `spark-streaming/consumer.py`\n\n### Grafana\n- **Setup Script**: `grafana/grafana-setup.sh`\n- **Description**: Installs the Cassandra plugin, adds the data source, and creates a dashboard for real-time data visualization.\n\n## Integration Workflow\n\n1. **Data Generation and Publishing**:\n   - WeatherMocker generates random weather data.\n   - Kafka producer publishes the data to a Kafka topic.\n\n2. **Data Consumption and Processing**:\n   - Spark-Streaming consumes data from Kafka, processes it, and writes it to Cassandra.\n\n3. **Data Visualization**:\n   - Grafana visualizes the real-time data stored in Cassandra.\n\n## Deployment\n\nThe entire stack is managed using Docker Compose, defined in `docker-compose.yml`. Each service runs in its own Docker container.\n\n## Running the Pipeline\n\nTo run the Docker pipeline, run the following two commands from the root directory:\n\n1. Make the `run.sh` script executable:\n   ```sh\n   $ chmod +x run.sh\n   $ ./run.sh","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevonwintz%2Fweather-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevonwintz%2Fweather-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevonwintz%2Fweather-stream/lists"}