{"id":18399144,"url":"https://github.com/bayoadejare/lightning-streams","last_synced_at":"2025-08-14T11:55:31.412Z","repository":{"id":194840420,"uuid":"691778145","full_name":"BayoAdejare/lightning-streams","owner":"BayoAdejare","description":"Batch/stream ETL pipeline of NOAA GLM dataset, using Python frameworks: Dagster, PySpark and Parquet storage. ","archived":false,"fork":false,"pushed_at":"2023-09-18T16:58:47.000Z","size":66470,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-11T04:53:24.833Z","etag":null,"topics":["clustering","csv","data-engineering","data-pipeline","data-warehousing","database","etl-pipeline","jupyter-notebook","k-means-clustering","machine-learning","noaa-data","orchestration","parquet","pyspark","python","spark-sql","spark-streaming","sql","streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BayoAdejare.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-14T21:47:20.000Z","updated_at":"2025-07-31T17:02:01.000Z","dependencies_parsed_at":"2023-09-15T10:53:22.603Z","dependency_job_id":null,"html_url":"https://github.com/BayoAdejare/lightning-streams","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"56cdc6a7b11c10fcee5909f153221721f2fe0ef8"},"previous_names":["bayoadejare/lightning-streams"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BayoAdejare/lightning-streams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Flightning-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Flightning-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Flightning-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Flightning-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BayoAdejare","download_url":"https://codeload.github.com/BayoAdejare/lightning-streams/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Flightning-streams/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270415105,"owners_count":24579556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-14T02:00:10.309Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","csv","data-engineering","data-pipeline","data-warehousing","database","etl-pipeline","jupyter-notebook","k-means-clustering","machine-learning","noaa-data","orchestration","parquet","pyspark","python","spark-sql","spark-streaming","sql","streaming"],"created_at":"2024-11-06T02:25:54.668Z","updated_at":"2025-08-14T11:55:31.348Z","avatar_url":"https://github.com/BayoAdejare.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lightning Streams\r\n\r\nAn example of a simple `stream and batch query` made by implementing PySpark, python API of [Apache Spark™](https://spark.apache.org/), queries on a Lightning flash dataset collected from [NOAA's GLM](https://www.goes-r.gov/spacesegment/glm.html).\r\nUses [Apache Parquet](https://parquet.apache.org/) file format as the storage backend and [Dagster Software-Defined Assets](https://docs.dagster.io/concepts/assets/software-defined-assets) to orchestrate the batch/stream processing pipeline.\r\n\r\nBlog post: [Lightning Streams: PySpark Batch \u0026 Streaming Queries](https://bayoadejare.medium.com/lightning-streams-pyspark-batch-streaming-queries-f13fc6a68cb6)\r\n\r\n|\u003ca href=\"img/main_tech_stack.png\" align=\"center\"\u003e\u003cimg src=\"img/main_tech_stack.png\" alt=\"Technologies used and respective logos\" width=\"800px\"/\u003e\u003c/a\u003e\r\n|:--:|\r\n|Dagster + PySpark + Parquet|\r\n\r\n## Installation\r\n\r\nFirst make sure, you have the requirements installed, this can be installed from the project directory via pip's setup command:\r\n\r\n`pip install . # =\u003c python3.11 `\r\n\r\n## Quick Start\r\n\r\nRun the command to start the dagster orchestration framework: \r\n\r\n`dagster dev # Start dagster daemon and dagit ui`\r\n\r\nThe dagster daemon is required to start the scheduling, from the dagit ui, you can run and monitor the data assets.\r\n\r\n## ETL Pipeline\r\n\r\nETL pipe data assets:\r\n\r\n+ `Source`: **extracts** NOAA GOES-R GLM file datasets from AWS s3 bucket. \r\n+ `Transformations`: **transforms** dataset into time series csv.\r\n+ `Sink`: **loads** dataset to persistant storage.\r\n\r\nSink loading process refactored to use `pyspark` (batch and structured streaming queries) and parquet as the storage backend.\r\n\r\n|\u003ca href=\"img/pipeline/etl_pipe.png\" align=\"center\"\u003e\u003cimg src=\"img/pipeline/etl_pipe.png\" alt=\"ETL Data assets\" width=\"300px\"/\u003e\u003c/a\u003e\r\n|:--:|\r\n|ETL Data asset group|\r\n\r\n## Clustering Pipeline\r\n\r\nBlog post: [Exploratory Data Analysis with Lightning Streaming Pipeline](https://medium.com/@adebayoadejare/exploratory-data-analysis-with-lightning-clustering-pipeline-6a2bca17d0d3)\r\n\r\n\r\n|\u003ca href=\"img/pipeline/eda_sda_job.png\" align=\"center\"\u003e\u003cimg src=\"img/pipeline/eda_sda_job.png\" alt=\"Lightning clustering pipeline Illustration\" width=\"400px\"/\u003e\u003c/a\u003e\r\n|:--:|\r\n|Materializing Lightning clustering pipeline|\r\n\r\n### Data Ingestion\r\n\r\nIngests the data needed based on specified time window: start and end dates.\r\n\r\n#### Data Assets\r\n\r\n+ `ingestor`: Composed of `extract`, `transform`, and `load` data assets.\r\n+ `extract`: downloads [NOAA GOES-R GLM](https://www.goes-r.gov/spacesegment/glm.html) netCDF files from AWS s3 bucket\r\n+ `transform`: converts GLM netCDF into time and geo series CSVs \r\n+ `load`: loads CSVs to a local backend, persistant duckdb\r\n\r\n### Cluster Analysis\r\n\r\nPerforms grouping of the ingested data by implementing K-Means clustering algorithm.\r\n\r\n|\u003ca href=\"img/cluster_process.png\" align=\"center\"\u003e\u003cimg src=\"img/cluster_process.png\" alt=\"An example clustering of flash data points\" width=\"800px\"/\u003e\u003c/a\u003e\r\n|:--:|\r\n|Visual of clustering process|\r\n\r\n#### Data Assets\r\n\r\n+ `preprocessor`: prepares the data for cluster model, clean and normalize the data.\r\n+ `kmeans_cluster`: fits the data to an implementation of k-means cluster algorithm.\r\n+ `silhouette_evaluator`: evaluates the choice of 'k' clusters by calculating the silhouette coefficient for each k in defined range.\r\n+ `elbow_evaluator`: evaluates the choice of 'k' clusters by calculating the sum of the squared distance for each k in defined range.\r\n\r\n|\u003ca href=\"img/pipeline/eda_sda_pipe.png\" align=\"center\"\u003e\u003cimg src=\"img/pipeline/eda_sda_pipe.png\" alt=\"Display of clustering materialized assets\" width=\"400px\"/\u003e\u003c/a\u003e\r\n|:--:|\r\n|Displaying Clusering analysis data assets|\r\n\r\n\r\n\r\n|\u003ca href=\"./img/sample_lightning_clusters.gif\" align=\"center\"\u003e\u003cimg src=\"./img/sample_lightning_clusters.gif\" alt=\"An example clustering of flash data points\" width=\"600px\"/\u003e\u003c/a\u003e\r\n|:--:|\r\n|Lightning clustering map|\r\n\r\n\r\n## Testing\r\n\r\nUse the following command to run tests:\r\n\r\n`pytest`\r\n\r\n## License\r\n\r\n[Apache 2.0 License](LICENSE)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbayoadejare%2Flightning-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbayoadejare%2Flightning-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbayoadejare%2Flightning-streams/lists"}