{"id":16596958,"url":"https://github.com/trannhatnguyen2/bi_cloud_kientap","last_synced_at":"2026-06-23T13:32:46.021Z","repository":{"id":179429207,"uuid":"663479572","full_name":"trannhatnguyen2/BI_Cloud_KienTap","owner":"trannhatnguyen2","description":"Building a Business Intelligence Solution on the Microsoft Azure Cloud Platform with Dynamic ELT Integration","archived":false,"fork":false,"pushed_at":"2023-07-07T15:22:40.000Z","size":39386,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-29T18:49:28.233Z","etag":null,"topics":["azure","datalake","datawarehouse","powerbi"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trannhatnguyen2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-07T11:41:59.000Z","updated_at":"2024-07-03T17:52:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"07be2f72-6b15-4f87-96a8-95bc4e7d5a72","html_url":"https://github.com/trannhatnguyen2/BI_Cloud_KienTap","commit_stats":null,"previous_names":["trannhatnguyen2/bi_cloud_kientap"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/trannhatnguyen2/BI_Cloud_KienTap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2FBI_Cloud_KienTap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2FBI_Cloud_KienTap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2FBI_Cloud_KienTap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2FBI_Cloud_KienTap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trannhatnguyen2","download_url":"https://codeload.github.com/trannhatnguyen2/BI_Cloud_KienTap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trannhatnguyen2%2FBI_Cloud_KienTap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34691840,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","datalake","datawarehouse","powerbi"],"created_at":"2024-10-11T23:54:35.627Z","updated_at":"2026-06-23T13:32:46.001Z","avatar_url":"https://github.com/trannhatnguyen2.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [Building a Business Intelligence Solution on the Microsoft Azure Cloud Platform with Dynamic ELT Integration](https://github.com/trannhatnguyen2/BI_Cloud_KienTap)\n\n## Member of group\n\n### **`BoKho`**\n\n| student_id | class   | full_name        | role   |\n| ---------- | ------- | ---------------- | ------ |\n| K204061440 | K20406T | Tran Nhat Nguyen | Leader |\n| K204061446 | K20406C | Man Dac Sang     | Member |\n\n# 📕 Table of contents\n\n\u003c!--ts--\u003e\n\n- 🛠️ [Requirements](#️-requirements)\n- 🧙‍♂️ [Data Source](#-data-source)\n- 🚀 [Solution](#-solution)\n- 🌊 [Building Data Lake](#-building-data-lake)\n- 🧱 [Building Data Warehouse](#-building-data-warehouse)\n- 📊 [Result](#️-result)\n\u003c!--te--\u003e\n\n \u003cbr /\u003e\n\n# 🛠️ Requirements\n\nMany businesses are using multiple systems and data is distributed across multiple sources and formatted in different file types. This leads to difficulties in importing and storing data, and if there are discrepancies, it can lead to many negative consequences such as loss of consistency, unnecessary costs, and impact on the business decision-making process.\n\n# 🧙‍♂️ Data Source\n\nThe data was obtained from Kaggle for experimentation. It was divided into three different sources:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/DataSources.png\" width=60% height=60%\u003e\n\n\u003cp align=\"center\"\u003e\n    Data Sources\n\u003c/p\u003e\n\n1. `Databases`: recording sales activities on an e-commerce platform in Brazil regarding orders.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/ERD_model.png\" width=75% height=75%\u003e\n\n\u003cp align=\"center\"\u003e\n    ERD model\n\u003c/p\u003e\n\n2. `Accounting Systems`: recording and managing customer payment information for orders.\n3. `Web Services`: customer comments on products and services.\n\n# 🚀 Solution\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/BI_Process.png\" width=100% height=100%\u003e\n\n\u003cp align=\"center\"\u003e\n    BI Solution\n\u003c/p\u003e\n\n- Step 1: Identify data sources and file formats for each source.\n- Step 2: Extract data into the `rawdata` zone using a Python script; perform dynamic ELT processes into the `curated` zone to store and upload necessary data for analysis to Azure SQL Server.\n- Step 3: Perform ETL processes into the Data Warehouse using Data Factory.\n- Step 4: Visualize data using Power BI.\n\n# 🌊 Building Data Lake\n\n## Container\n\nThe tool used to create data storage zones on the Azure platform is Blob Storage.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/Container.png\" width=60% height=60%\u003e\n\n\u003cp align=\"center\"\u003e\n    Containers\n\u003c/p\u003e\n\n### **rawdata**\n\nAn exact copy of the data from sources, organized in an orderly folder structure.\n\n```bash\n./RAWDATA\n├── .accounting_systems/ \u003c- Accounting System Source\n│ ├── Payment_2018_01.csv\n│ ├── Payment_2018_02.csv\n│ └── Payment_2018_03.csv\n│\n├── .databases/ \u003c- Databases Source\n│ ├── Customer_2018_01.csv\n│ ├── Customer_2018_02.csv\n│ ├── Customer_2018_03.csv\n│ ├── Order_2018_01.csv\n│ ├── Order_2018_02.csv\n│ |── Order_2018_03.csv\n| ├── OrderItem_2018_01.csv\n│ |── OrderItem_2018_02.csv\n│ └── OrderItem_2018_03.csv\n│\n├── .web_services/ \u003c- Web Services Source\n│ ├── Review_2018_01.zip\n│ ├── Review_2018_01.zip\n│ └── Review_2018_01.zip\n```\n\n### **staging**\n\nUsed to extract all compressed files to prepare for importing data into the `curated` zone.\n\n### **curated**\n\n```bash\n./CURATED\n├── .EXTERNAL/\n│ ├── .Review/\n    ├── .2018/\n        ├── .01/\n            └── Review_2018_01.json\n        ├── .02/\n            └── Review_2018_02.json\n        ├── .03/\n            └── Review_2018_03.json\n├── .INTERNAL/\n│ ├── .Accounting/\n    ├── .Payment/\n        ├── .2018/\n            ├── .01/\n                └── Payment_2018_01.csv\n            ├── .02/\n                └── Payment_2018_02.csv\n            ├── .03/\n                └── Payment_2018_03.csv\n│ ├── .Sales/\n    ├── .Customer/\n        ├── .2018/\n            ├── .01/\n                └── Customer_2018_01.csv\n            ├── .02/\n                └── Customer_2018_02.csv\n            ├── .03/\n                └── Customer_2018_03.csv\n    ├── .Order/\n        ├── .2018/\n            ├── .01/\n                └── Order_2018_01.csv\n            ├── .02/\n                └── Order_2018_02.csv\n            ├── .03/\n                └── Order_2018_03.csv\n    ├── .OrderItem/\n        ├── .2018/\n            ├── .01/\n                └── OrderItemm_2018_01.csv\n            ├── .02/\n                └── OrderItemm_2018_02.csv\n            ├── .03/\n                └── OrderItemm_2018_03.csv\n```\n\n## Dynamic ELT\n\nDynamic ELT process is the process of extracting raw data and uploading it to data storage zones accurately according to the predefined structure through adjusting input parameters.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/DataLake_Process.png\" width=100% height=100%\u003e\n\n\u003cp align=\"center\"\u003e\n    Dynamic ELT Process\n\u003c/p\u003e\n\n# 🧱 Building Data Warehouse\n\n`Bus Matrix`, `Master Data`, `Transaction Data`, `ETL Mapping`, etc. are deployed to support the data warehouse construction process.\n\n## Data Warehouse model\n\nThe diagram below illustrates the fundamental conceptual diagram of the proposed data warehouse in Star format.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/DataWarehouse_StarSchema.png\" width=70% height=70%\u003e\n\n\u003cp align=\"center\"\u003e\n    Data Warehouse Star Schema\n\u003c/p\u003e\n\n## ETL process\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/ETL_Pipeline.png\" width=100% height=100%\u003e\n\n\u003cp align=\"center\"\u003e\n    ETL Pipeline\n\u003c/p\u003e\n\nBased on the pipeline shown above, it is divided into 2 phases:\n\n- Phase 1: Load data from Azure SQL Server --\u003e Dimension Tables\n- Phase 2: Load data from Azure SQL Server --\u003e Fact Table\n\n# 📊 Result\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./img/Dashboard.png\" width=80% height=80%\u003e\n\n\u003cp align=\"center\"\u003e\n    Sales Performance Dashboard\n\u003c/p\u003e\n\n---\n\n\u003cp\u003e\u0026copy; 2023 BoKho\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrannhatnguyen2%2Fbi_cloud_kientap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrannhatnguyen2%2Fbi_cloud_kientap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrannhatnguyen2%2Fbi_cloud_kientap/lists"}