{"id":26154703,"url":"https://github.com/justus-coded/electric_vehicle_de","last_synced_at":"2025-12-24T02:18:40.911Z","repository":{"id":232821883,"uuid":"785272815","full_name":"Justus-coded/electric_vehicle_de","owner":"Justus-coded","description":"My Data Engineering Zoomcamp project","archived":false,"fork":false,"pushed_at":"2024-04-16T21:30:17.000Z","size":30586,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T08:41:10.586Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Justus-coded.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T14:52:34.000Z","updated_at":"2024-05-10T18:48:44.000Z","dependencies_parsed_at":"2024-04-16T22:44:17.700Z","dependency_job_id":null,"html_url":"https://github.com/Justus-coded/electric_vehicle_de","commit_stats":null,"previous_names":["justus-coded/electric_vehicle_de"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Justus-coded/electric_vehicle_de","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justus-coded%2Felectric_vehicle_de","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justus-coded%2Felectric_vehicle_de/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justus-coded%2Felectric_vehicle_de/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justus-coded%2Felectric_vehicle_de/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Justus-coded","download_url":"https://codeload.github.com/Justus-coded/electric_vehicle_de/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justus-coded%2Felectric_vehicle_de/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27992999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-11T08:34:09.230Z","updated_at":"2025-12-24T02:18:40.891Z","avatar_url":"https://github.com/Justus-coded.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Electric Vehicle Data Engineering Project\n\n## Overview\nThis project is my submission for the Data Engineering Zoomcamp 2024 Capstone. The aim of the project is to explore and gain insights into the use of electric vehicles (EVs) in the United States. The dataset utilized for this project was obtained from [Data.gov](https://catalog.data.gov/dataset/electric-vehicle-population-data), more information about the data can be found [here](https://www.opendatanetwork.com/dataset).\n\n## Project Architecture\n ![Project Architecture](project_architecture_.PNG)\n## Project Workflow\n\n### Cloud Infrastructure Setup\n1. **GCP Infrastructure Creation**\n   - Create a project in Google Cloud Platform (GCP).\n   - Create a service account with the following roles: Storage Admin, BigQuery Admin, and Compute Admin.\n   - Generate service account keys and download the key (.json file).\n\n2. **Terraform Setup**\n   - Install Terraform:\n     ```bash\n     wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg\n     echo \"deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/hashicorp.list\n     sudo apt update \u0026\u0026 sudo apt install terraform\n     ```\n   - Create a new folder (e.g., `terraform`) and navigate into it.\n   - Create `main.tf` and `variables.tf` files.\n   - Update `main.tf` with project ID, credentials, region, and location using the variables declared in `variables.tf`.\n   - Update `main.tf` with Google Cloud Storage bucket name and BigQuery dataset name.\n   - Set service account key as an environment variable:\n     ```bash\n     export GOOGLE_CREDENTIALS='/path/to/your/keys.json'\n     ```\n   - Initialize Terraform and apply changes:\n     ```bash\n     terraform init\n     terraform plan\n     terraform apply\n     ```\n\n### Data Orchestration\n1. **Mage and Docker Setup**\n   - Create a folder for Mage (e.g., `Mage`).\n   - Add a Dockerfile to the folder that contains the Mage image:\n     ```Dockerfile\n     FROM mageai/mageai:latest\n     \n     ARG USER_CODE_PATH=/home/src/${PROJECT_NAME}\n     \n     COPY requirements.txt ${USER_CODE_PATH}requirements.txt \n     \n     RUN pip3 install -r ${USER_CODE_PATH}requirements.txt\n     ```\n   - Add a `docker-compose.yml` file to include the project name, Dockerfile, GCP service keys, and other environment variables.\n\n2. **Running Mage**\n   - Build and spin up Mage on localhost:\n     ```bash\n     docker-compose up\n     ```\n   - This may take some time to initialize.\n\n3. **Pipeline Creation**\n   - In Mage, create a pipeline to load the data into Google Cloud Storage and BigQuery.\n\n### Data Transformation\n\n#### dbt Cloud Setup\n1. **Create a New Project in dbt**\n   - Sign in to dbt Cloud and create a new project.\n   - Add a project name and select BigQuery as the data source.\n   - Include the service account keys for authentication.\n   - Set up a repository to run your transformations and create the project.\n\n2. **Initialize dbt Project**\n   - Click 'initialize dbt project' and then 'commit and sync'.\n\n3. **Model Creation and Transformation**\n   - Create your SQL files in the models directory for transformation.\n   - Perform data transformation using dbt:\n     ```bash\n     dbt run\n     ```\n\n4. **Resources**\n   - [dbt Documentation for BigQuery](https://docs.getdbt.com/guides/bigquery?step=8)\n\n### Dashboard Creation\n\n1. **Access Looker Studio**\n   - Go to [Looker Studio](https://lookerstudio.google.com).\n\n2. **Create a New Report**\n   - Create a new report in Looker Studio.\n   - Connect to BigQuery as the data source.\n   - Select datasets to use for building the dashboard.\n\n3. **Dashboard Design**\n   - Design your dashboard in Looker Studio.\n   - Include relevant visualizations and insights.\n\n4. **[Electric Vehicle Dashboard](https://lookerstudio.google.com/reporting/fb76d5d5-193d-46b9-8b06-b75b885f9b03)**\n\n   ![Dashboard](ev_dashboard.PNG)\n\n\n\n## Resources\n- [Google Cloud Platform](https://cloud.google.com/)\n- [Terraform Documentation](https://learn.hashicorp.com/terraform)\n- [Mage Documentation](https://docs.mage.ml/)\n- [Docker Documentation](https://docs.docker.com/)\n- [dbt Documentation](https://docs.getdbt.com/)\n- [Looker Documentation](https://docs.looker.com/)\n- [Data.gov](https://www.data.gov/) - Source of the Electric Vehicle Population Data\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustus-coded%2Felectric_vehicle_de","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustus-coded%2Felectric_vehicle_de","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustus-coded%2Felectric_vehicle_de/lists"}