https://github.com/olamide100/datawiz

Agricultural Monitoring Systems Overview This dataset simulates real-time agricultural sensor data, providing a rich source of information to design and construct a robust data engineering solution.
https://github.com/olamide100/datawiz

agriculture datafest dbt

Last synced: 3 months ago
JSON representation

Agricultural Monitoring Systems Overview This dataset simulates real-time agricultural sensor data, providing a rich source of information to design and construct a robust data engineering solution.

• Host: GitHub
• URL: https://github.com/olamide100/datawiz
• Owner: OLAMIDE100
• Created: 2023-09-15T20:55:27.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2023-09-17T20:17:38.000Z (over 2 years ago)
• Last Synced: 2025-03-14T08:43:02.949Z (about 1 year ago)
• Topics: agriculture, datafest, dbt
• Homepage:
• Size: 1.24 MB
• Stars: 1
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          







 


  
Team-DataWiz DatafestAfrica Datathon 2023 - Data Engineering Project


  


    A wrap-up project for the 3-day DatafestAfrica Datathon 2023!

    


    Explore the docs »

    


    


    ·

   

  



  Table of Contents

  


    


      About The Project

      


        
Built With

        
Architecture

      

    

    
Usage

    
Roadmap

    
Pipeline

   


      Deliverables

      


        
Jobs

        
Environments

        
Runs

      

    

    

    

  


## About The Project



  
Agricultural Monitoring Systems

  

  

    

  

 


Our business operates in the agricultural sector, where monitoring various environmental conditions and sensor data is crucial for optimizing crop yields and ensuring sustainable farming practices. This datathon is aligned with our commitment to leverage data engineering to streamline the processing and analysis of agricultural data.

(back to top)


### Architecture

![architecture diagram](https://github.com/OLAMIDE100/DATAWIZ/blob/main/image/ETL.png)

### Built With

Language, frameworks, libraries, Services and Tools used to bootstrap this project.

* DBT 

* Snowflake

* GitHub

* SQL

* Jinja Template

(back to top)


## Usage

This dataset simulates real-time agricultural sensor data, providing a rich source of information to design and construct a robust data engineering solution. The primary objective of this project is to extract, transform, and load (ETL) the raw sensor data into structured fact and dimension tables within our snowflake data warehouse.

(back to top)


## Roadmap

- [x] Create a Snowflakes Account

- [x] Setup DBT cloud or core using the snowflake as its warehouse

- [x] Data Modelling (logical and business)

- [x] Develop

- [x] Test and Documentation

- [x] Create Jobs

- [x] Deploy to Production

## Pipeline

![pipeline diagram](https://github.com/OLAMIDE100/DATAWIZ/blob/main/image/pipeline.png)

## Deliverables

### Jobs

![jobs diagram](https://github.com/OLAMIDE100/DATAWIZ/blob/main/image/Jobs.png)

### Environments

![environments diagram](https://github.com/OLAMIDE100/DATAWIZ/blob/main/image/Environments.png)

### Runs

![Hourly_transform_load_run_history](https://github.com/OLAMIDE100/DATAWIZ/blob/main/image/Hourly_transform_load_run_history.png)

![6_hours_test_run_history diagram](https://github.com/OLAMIDE100/DATAWIZ/blob/main/image/6_hours_test_run_history.png)