Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/quantumudit/daily-covid19-india-statewide-analysis

This project automates the Python ETL process through GitHub Actions. The updated data is then analyzed and visualized with Jupyter Notebook and Power BI, respectively, to generate powerful insights
https://github.com/quantumudit/daily-covid19-india-statewide-analysis

data-analytics github-actions power-bi python

Last synced: 11 days ago
JSON representation

This project automates the Python ETL process through GitHub Actions. The updated data is then analyzed and visualized with Jupyter Notebook and Power BI, respectively, to generate powerful insights

Awesome Lists containing this project

README 

        
![Project Logo][project_logo]



---



Analyzing daily statewide COVID-19 cases in India by extracting data from Ministry of Health & Family Welfare website with Python and Power BI

















  Overview •

  Prerequisites •

  Architecture •

  Demo •

  Support •

  License




## Overview



This project focuses on getting the daily India statewide COVID-19 data from [Ministry of Health & Family Welfare][website_link], performing exploratory data analysis to generate insights and visualize them with the help of Power BI.



The repository directory structure is as follows:



├───Daily-COVID19-India-Statewide-Analysis


│ ├───01_ETL


│ ├───02_DATA


│ └───05_RESOURCES



The type of content present in the directories is as follows:



**01_ETL**



This directory contains the ETL script that takes the JSON response from the target website as input, transforms it and exports an analysis-ready dataset into the _02_DATA_ directory.



**02_DATA**



This directory contains the data that can be directly used for exploratory data analysis and data visualization purposes.



**03_ANALYSIS**



This directory contains the python notebooks that analyzes the clean dataset to generate insights



**04_DASHBOARD**



This directory contains the python notebook with an embedded Power BI report that visualizes the data. The Power BI dashboard contains slicers, cross-filtering and other advance capabilities that end user can play with to visualize a specific facet of the data or, to get additional insights.



**05_RESOURCES**



This directory contains images, icons, layouts, etc. that are used in this project



## Prerequisites



The major skills that are required as prerequisite to fully understand this project are as follows:



- Basics of Python

- Python libraries: Requests, Pandas, DateTime

- Basics of Python Notebooks

- Basics of Power BI



In order to complete the project, I've used the following applications and libraries



- Python

- Python libraries mentioned in requirements.txt file

- Jupyter Notebook

- Visual Studio Code

- Microsoft Power BI



> The choice of applications & their installation might vary based on individual preferences & system settings.



## Architecture



The project architecture is quite straight forward and can be explained through the below image:



![Process Architecture][process_workflow]



As per the above workflow; we are first requesting the JSON data from the website which is then processed and cleaned with another ETL specific Python script.



Finally; we leverage the clean & analysis-ready dataset for some exploratory data analysis (EDA) using Jupyter Notebook and creating an insightful report using Power BI



The ETL job has been also scheduled to run everyday with the help of GitHub Actions so that, we can get updated data on daily basis.



## Demo



To be updated.



## Support



If you have any doubts, queries or, suggestions then, please connect with me in any of the following platforms:



[![Linkedin Badge][linkedinbadge]][linkedin] [![Twitter Badge][twitterbadge]][twitter]



If you like my work then, you may support me at Patreon:





git




## License





    




This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.



[project_logo]: 05_RESOURCES/project_cover_image.png

[process_workflow]: 05_RESOURCES/process_architecture.png

[scraping_graphic]: 05_RESOURCES/scraping_graphic.gif



[website_link]: https://www.mohfw.gov.in/



[linkedin]: https://www.linkedin.com/in/uditkumarchatterjee/

[twitter]: https://twitter.com/quantumudit



[linkedinbadge]: https://img.shields.io/badge/-uditkumarchatterjee-0e76a8?style=flat&labelColor=0e76a8&logo=linkedin&logoColor=white

[twitterbadge]: https://img.shields.io/badge/-@quantumudit-1ca0f1?style=flat&labelColor=1ca0f1&logo=twitter&logoColor=white&link=https://twitter.com/quantumudit