{"id":13465565,"url":"https://github.com/iusztinpaul/energy-forecasting","last_synced_at":"2025-05-15T12:05:05.008Z","repository":{"id":65687268,"uuid":"597351385","full_name":"iusztinpaul/energy-forecasting","owner":"iusztinpaul","description":"🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 \u0026 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 \u0026 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴","archived":false,"fork":false,"pushed_at":"2024-04-03T13:42:05.000Z","size":4296,"stargazers_count":905,"open_issues_count":1,"forks_count":206,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-22T16:58:33.151Z","etag":null,"topics":["3-pipeline-design","airflow","batch-processing","cicd","data-versioning","docker","fastapi","feature-store","gcp","github-actions","great-expectations","hopsworks","ml-monitoring","mlops","model-registry","poetry","python","sktime","streamlit","weights-and-biases"],"latest_commit_sha":null,"homepage":"https://www.pauliusztin.me/courses/the-full-stack-7-steps-mlops-framework","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iusztinpaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-02-04T09:20:13.000Z","updated_at":"2025-04-17T15:05:36.000Z","dependencies_parsed_at":"2023-02-18T16:32:14.862Z","dependency_job_id":"8705ae3a-730d-4891-8aa7-d105112dbf0c","html_url":"https://github.com/iusztinpaul/energy-forecasting","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iusztinpaul%2Fenergy-forecasting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iusztinpaul%2Fenergy-forecasting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iusztinpaul%2Fenergy-forecasting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iusztinpaul%2Fenergy-forecasting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iusztinpaul","download_url":"https://codeload.github.com/iusztinpaul/energy-forecasting/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254337612,"owners_count":22054253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3-pipeline-design","airflow","batch-processing","cicd","data-versioning","docker","fastapi","feature-store","gcp","github-actions","great-expectations","hopsworks","ml-monitoring","mlops","model-registry","poetry","python","sktime","streamlit","weights-and-biases"],"created_at":"2024-07-31T15:00:31.974Z","updated_at":"2025-05-15T12:04:59.973Z","avatar_url":"https://github.com/iusztinpaul.png","language":"Python","funding_links":["https://www.buymeacoffee.com/pauliusztin)."],"categories":["Python"],"sub_categories":[],"readme":"# The Full Stack 7-Steps MLOps Framework\n\n`Learn MLE \u0026 MLOps for free by designing, building, deploying and monitoring an end-to-end ML batch system | source code + 2.5 hours of reading \u0026 video materials on Medium`\n\nThis repository contains a **7-lesson FREE course** to teach you how to **build a production-ready ML batch system**. Its primary focus is to engineer a scalable system using MLOps good practices. You will implement an ML system for forecasting hourly energy consumption levels across Denmark.\n\nYou will **learn how to build, train, serve, and monitor an ML system** using a batch architecture. We will show you how to integrate an experiment tracker, a model registry, a feature store, Docker, Airflow, GitHub Actions and more!\n\n**Level:** Intermediate to Advanced | This **course targets** MLEs who want to build end-to-end ML systems and SWEs who wish to transition to MLE.\n\n------\n\nFollowing the **documentation on GitHub** and the [lessons on Medium](#lessons), you have *2.5 hours of reading \u0026 video materials*, which will help you understand every piece of the code!\n\n**At the end of the course, you will know how to build everything from the diagram below 👇**\n\nDon't worry if something doesn't make sense to you. We will explain everything in detail in the [Medium lessons](#lessons).\n\nIf you are unsure if this course is for you, [here is an article presenting a high-level overview](https://pub.towardsai.net/the-full-stack-7-steps-mlops-framework-6599a0c6e295) of all the components you will build during the series.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/architecture.png\"\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://youtu.be/OKk9U310qYE\"\u003e\n    \u003cstrong\u003eCheck out this short video to see what you will build during the course 👇\u003c/strong\u003e\n    \u003cbr/\u003e\n    \u003cbr/\u003e\n    \u003cimg src=\"images/screenshot_introduction_video.png\" alt=\"Introduction Video\" style=\"width:75%;\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n*You can safely use this code as you like, as long as you respect the terms and agreement of the MIT License.*\n\n``\u003c\u003c\u003c Using all the tools suggested in the course will be free of charge, except the ones from Lesson 7, where you will be deploying your application to GCP which will cost you ~20$. \u003e\u003e\u003e``\n\n# Table of Contents\n1. [What You Will Learn](#learn)\n2. [Lessons \u0026 Tutorials](#lessons)\n3. [Costs](#costs)\n4. [Ask Questions](#ask-questions)\n5. [Data](#data)\n6. [Code Structure](#structure)\n7. [Set Up Additional Tools](#tools)\n8. [Usage](#usage)\n9. [Installation \u0026 Usage for Development](#installation)\n10. [Licensing \u0026 Contributing](#licensing)\n11. [Support](#support)\n\n--------\n\n# 🤔 1. What You Will Learn \u003ca name=learn\u003e\u003c/a\u003e\n**At the end of this 7 lessons course, you will know how to:**\n* design a batch-serving architecture\n* use Hopsworks as a feature store\n* design a feature engineering pipeline that reads data from an API\n* build a training pipeline with hyper-parameter tunning\n* use W\u0026B as an ML Platform to track your experiments, models, and metadata\n* implement a batch prediction pipeline\n* use Poetry to build your own Python packages\n* deploy your own private PyPi server\n* orchestrate everything with Airflow\n* use the predictions to code a web app using FastAPI and Streamlit\n* use Docker to containerize your code\n* use Great Expectations to ensure data validation and integrity\n* monitor the performance of the predictions over time\n* deploy everything to GCP\n* build a CI/CD pipeline using GitHub Actions\n\nIf that sounds like a lot, don't worry. After you cover this course, you will understand everything we said before. Most importantly, you will know WHY we used all these tools and how they work together as a system.\n\n# 🤌 2. Lessons \u0026 Tutorials \u003ca name=lessons\u003e\u003c/a\u003e\n\nThe course consists of 7 lessons hosted on Medium Towards Data Science publication. We also provide a bonus lesson where we openly discuss potential improvements that could be made to the current architecture and trade-offs we had to take during the course. The course adds up to *2.5 hours of reading and video materials*. \n\n`We recommend running the code along the articles to get the best out of this course, as we provide detailed instructions to set everything up.`\n\n**👇 Access the step-by-step lessons on Medium 👇**\n\nHere is an [article presenting a high-level overview](https://pub.towardsai.net/the-full-stack-7-steps-mlops-framework-6599a0c6e295) of all the components you will build during the course.\n\n1. [Batch Serving. Feature Stores. Feature Engineering Pipelines.](https://medium.com/towards-data-science/a-framework-for-building-a-production-ready-feature-engineering-pipeline-f0b29609b20f)\n2. [Training Pipelines. ML Platforms. Hyperparameter Tuning.](https://medium.com/towards-data-science/a-guide-to-building-effective-training-pipelines-for-maximum-results-6fdaef594cee)\n3. [Batch Prediction Pipeline. Package Python Modules with Poetry.](https://medium.com/towards-data-science/unlock-the-secret-to-efficient-batch-prediction-pipelines-using-python-a-feature-store-and-gcs-17a1462ca489)\n4. [Private PyPi Server. Orchestrate Everything with Airflow.](https://towardsdatascience.com/unlocking-mlops-using-airflow-a-comprehensive-guide-to-ml-system-orchestration-880aa9be8cff)\n5. [Data Validation for Quality and Integrity using GE. Model Performance Continuous Monitoring.](https://towardsdatascience.com/ensuring-trustworthy-ml-systems-with-data-validation-and-real-time-monitoring-89ab079f4360)\n6. [Consume and Visualize your Model's Predictions using FastAPI and Streamlit. Dockerize Everything.](https://towardsdatascience.com/fastapi-and-streamlit-the-python-duo-you-must-know-about-72825def1243)\n7. [Deploy All the ML Components to GCP. Build a CI/CD Pipeline Using Github Actions.](https://towardsdatascience.com/seamless-ci-cd-pipelines-with-github-actions-on-gcp-your-tools-for-effective-mlops-96f676f72012)\n8. [\\[Bonus\\] Behind the Scenes of an ‘Imperfect’ ML Project — Lessons and Insights.](https://towardsdatascience.com/imperfections-unveiled-the-intriguing-reality-behind-our-mlops-course-creation-6ff7d52ecb7e)\n\n# 💵 3. Costs \u003ca name=costs\u003e\u003c/a\u003e\n\nThe code from GitHub is released under the MIT license. Thus, as long as you redistribute our LICENSE and give credit to our work, you can use it as you wish.\n\nThe Medium lessons are released under Medium's paid wall. If you already have it, then they are free. Otherwise, you must pay a $5 monthly fee to read the articles.\n\nOn the tools side, I will dig deeper when covering each tool independently, but TL/DR, it will cost you around ~20$ to deploy everything to GCP. \n\nTo conclude, the course will cost you ~25$-30$ to read and run end-to-end. \n\n# ❔ 4. Ask Questions \u003ca name=ask-questions\u003e\u003c/a\u003e\n\nIf you have any questions or issues during the course, please create an [issue](https://github.com/iusztinpaul/energy-forecasting/issues).  I will do my best to respond.\n\nAlso, you can contact me directly on [LinkedIn](https://www.linkedin.com/in/pauliusztin/).\n\n-------\n\n# 📊 5. Data \u003ca name=data\u003e\u003c/a\u003e\nWe used an open API that provides hourly energy consumption values for all the energy consumer types within Denmark.\n\nThey provide an intuitive interface where you can easily query and visualize the data. You can access the data [here](https://www.energidataservice.dk/tso-electricity/ConsumptionDE35Hour).\n\nThe data has 4 main attributes:\n* **Hour UTC**: the UTC datetime when the data point was observed. \n* **Price Area**: Denmark is divided into two price areas: DK1 and DK2 - divided by the Great Belt. DK1 is west of the Great Belt, and DK2 is east of the Great Belt.\n* **Consumer Type**: The consumer type is the Industry Code DE35, owned and maintained by Danish Energy.\n* **Total Consumption**: Total electricity consumption in kWh\n\n**Note:** The observations have a lag of 15 days! But for our demo use case, that is not a problem, as we can simulate the same steps as it would be in real time.\n\n### IMPORTANT OBSERVATION\n\nThe API will become obsolete during 2023. Its latest data points are from June 2023, and the API will become unavailable during 2023. We created a copy of the data from 2020-07-01 and 2023-06-30 to bypass this issue. Thus, there are 3 years of data to play with. More than enough for the purpose of this course. The file is stored in Google Drive accessible [at this link](https://drive.google.com/file/d/1y48YeDymLurOTUO-GeFOUXVNc9MCApG5/view?usp=drive_link).\n\nThus, instead of querying the API, we will mock the same behavior by loading the data from the file. Therefore, you don't have to download your file yourself. The code will download it and load the data from the file instead of the API, simulating 100% the same behavior.\n\n**---\u003e All Rights Reserved to: www.energidataservice.dk**\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/forecasting_demo_screenshot.png\"\u003e\n\u003c/p\u003e\n\nThe data points have an hourly resolution. For example: \"2023–04–15 21:00Z\", \"2023–04–15 20:00Z\", \"2023–04–15 19:00Z\", etc.\n\nWe will model the data as multiple time series. Each unique price area and consumer type tuple represents its unique time series. \n\nThus, we will build a model that independently forecasts the energy consumption for the next 24 hours for every time series.\n\n[Check out this video to better understand what the data looks like.](https://youtu.be/OKk9U310qYE)\n\n----------\n\n# 🧬 6. Code Structure \u003ca name=structure\u003e\u003c/a\u003e\n\nThe code is split into two main components: the `pipeline` and the `web app`.\n\nThe **pipeline** consists of 3 modules:\n- `feature-pipeline`\n- `training-pipeline`\n- `batch-prediction-pipeline`\n\nThe **web app** consists of other 3 modules:\n- `app-api`\n- `app-frontend`\n- `app-monitoring`\n\n**Also,** we have the following folders:\n- `airflow` : Airflow files | Orchestration\n- `.github` : GitHub Actions files | CI/CD\n- `deploy` : Build \u0026 Deploy\n\u003cbr/\u003e\n\u003cbr/\u003e\n\nTo follow the structure in its natural flow, read the folders in the following order:\n1. `feature-pipeline`\n2. `training-pipeline`\n3. `batch-prediction-pipeline`\n4. `airflow`\n5. `app-api`\n6. `app-frontend` \u0026 `app-monitoring`\n7. `.github`\n\n**Read the Medium articles listed in the [Lessons \u0026 Tutorials](#lessons) section for the whole experience.**\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n-------\n\n# 🔧 7. Set Up Additional Tools \u003ca name=tools\u003e\u003c/a\u003e\n\n**The code is tested only on Ubuntu 20.04 and 22.04 using Python 3.9.**\n\nWe use a `.env` file to store all our credentials. Every module that needs a `.env` file has a `.env.default` in the module's main directory that acts as a template. Thus, you have to run:\n```shell\ncp .env.default .env\n```\n... and complete what is surrounded by `\u003c...\u003e`. For now, don't do anything. We will explain in detail in later steps what you have to do. \n\n## Poetry\n##### ``\u003c\u003c free usage \u003e\u003e``\n\n**Note:** During the course, we used `Poetry 1.4.2`. To avoid potential issues when installing the dependencies using Poetry, we recommend you use the same version (or if there are any errors \u0026 you have a different version, you can delete and regenerate the `poetry.lock` file).\n\nInstall Python system dependencies:\n```shell\nsudo apt-get install -y python3-distutils\n```\nDownload and install Poetry:\n```shell\ncurl -sSL https://install.python-poetry.org | python3 -\n```\nOpen the `.bashrc` file to add the Poetry PATH: \n```shell\nnano ~/.bashrc\n```\n\nAdd `export PATH=~/.local/bin:$PATH`\n\nto `~/.bashrc`\n\nCheck if Poetry is installed:\n```shell\nsource ~/.bashrc\npoetry --version\n```\n\n[If necessary, here are the official Poetry installation instructions.](https://python-poetry.org/docs/#installation)\n\n\n#### macOS M1/M2 Poetry Issues\n**!!!** If you have issues creating Poetry environments on macOS M1/M2 devices, [Hongnan Gao](https://github.com/gao-hongnan) implemented a script that will solve all the dependency issues. Just run the following before creating a Poetry environment:\n```shell\nbash scripts/install_poetry_macos_m1_chip.sh\n```\n\n\n## Docker\n##### ``\u003c\u003c free usage \u003e\u003e``\n\nDuring the course we used `Docker version 24.0.5`.\n\n* [Install Docker on Ubuntu.](https://docs.docker.com/engine/install/ubuntu/)\n* [Install Docker on Mac.](https://docs.docker.com/desktop/install/mac-install/)\n* [Install Docker on Windows.](https://docs.docker.com/desktop/install/windows-install/)\n\n## Configure Credentials for the Private PyPi Server\n##### ``\u003c\u003c free usage \u003e\u003e``\n\n**\u003cbr/\u003eWe will run the private PyPi server using Docker down the line. But it will already expect the credentials configured.\u003cbr/\u003e**\n\nCreate credentials using `passlib`:\n```shell\n# Install dependencies.\nsudo apt install -y apache2-utils\npip install passlib\n\n# Create the credentials under the energy-forecasting name.\nmkdir ~/.htpasswd\nhtpasswd -sc ~/.htpasswd/htpasswd.txt energy-forecasting\n```\n\nSet `poetry` to use the credentials:\n```shell\npoetry config repositories.my-pypi http://localhost\npoetry config http-basic.my-pypi energy-forecasting \u003cpassword\u003e\n```\n\nCheck that the credentials are set correctly in your poetry `auth.toml` file:\n```shell\ncat ~/.config/pypoetry/auth.toml\n```\n\n## Hopsworks \n##### ``\u003c\u003c free usage \u003e\u003e``\n\nYou will use [Hopsworks](https://www.hopsworks.ai/) as your serverless feature store. Thus, you have to create an account and a project on Hopsworks. We will show you how to configure the code to use your Hopsworks project later.\n\n[We explained on Medium in **Lesson 1** how to create a Hopsworks API Key.](https://medium.com/towards-data-science/a-framework-for-building-a-production-ready-feature-engineering-pipeline-f0b29609b20f) But long story short, you can go to your Hopsworks account settings and get the API Key from there. Afterward, you must create a new project (or use the default one) and add these credentials to the `.env` file under the `FS_` prefix.\n\n**!!!** Be careful to name your project differently than **energy_consumption,** as Hopsworks requires unique names across its serverless deployment.\n\n[Click here to start with Hopsworks](https://www.hopsworks.ai/).\n\n**Note:** Our course will use only the Hopsworks freemium plan, making it free of charge to replicate the code within the series. \n\n\n## Weights \u0026 Biases\n##### ``\u003c\u003c free usage \u003e\u003e``\n\nYou will use Weights \u0026 Biases as your serverless ML platform. Thus, you must create an account and a project on Weights \u0026 Biases. We will show you how to configure the code to use your W\u0026B project later.\n\n[On Medium, we explained in **Lesson 2** how to create an API Key on W\u0026B.](https://towardsdatascience.com/a-guide-to-building-effective-training-pipelines-for-maximum-results-6fdaef594cee) But long story short, you can go to your W\u0026B and create an entity \u0026 project. Afterward, you must navigate to user settings and create the API Key from there. In the end, you must add these credentials to the `.env` file under the `WANDB_` prefix.\n\n**!!!** Be careful to name your entity differently than **teaching-mlops,** as W\u0026B requires unique names across its serverless deployment.\n\n[Click here to start with Weights \u0026 Biases](https://wandb.ai/).\n\n**Note:** Our course will use only the W\u0026B freemium plan, making it free of charge to replicate the code within the series. \n\n## GCP\n\nFirst, you must install the `gcloud` GCP CLI on your machine.\n\n[Follow this tutorial to install it.](https://cloud.google.com/sdk/docs/install)\n\n**If you only want to run the code locally, go straight to the \"Storage\" section.**\u003cbr/\u003e\n\nAs before, you have to create an account and a project on GCP. Using solely the bucket as storage will be free of charge.\n\nWhen we were writing this documentation, GCS was free until 5GB.\n\n### Storage\n##### ``\u003c\u003c free usage \u003e\u003e``\n\nAt this step, you have to do 5 things:\n- create a project\n- create a non-public bucket\n- create a service account that has admin permissions to the newly created bucket\n- create a service account that has read-only permissions to the newly created bucket\n- download a JSON key for the newly created service accounts.\n\nYour `bucket admin service account` should have assigned the following role: `Storage Object Admin`\u003cbr/\u003e\nYour `bucket read-only service account` should have assigned the following role: `Storage Object Viewer`\u003cbr/\u003e\n\n![Bucket Creation](images/gcp_gcs_screenshot.png)\n\n* [Docs for creating a bucket on GCP.](https://cloud.google.com/storage/docs/creating-buckets)\u003cbr/\u003e\n* [Docs for creating a service account on GCP.](https://cloud.google.com/iam/docs/service-accounts-create)\u003cbr/\u003e\n* [Docs for creating a JSON key for a GCP service account.](https://cloud.google.com/iam/docs/keys-create-delete)\u003cbr/\u003e\n\n**NOTE:** When we were writing this documentation, GCS was free until 5GB.\n\n[Check out **Lesson 3** on Medium to better understand **how we set up the GCP bucket** and its role in the batch prediction pipeline.](https://towardsdatascience.com/unlock-the-secret-to-efficient-batch-prediction-pipelines-using-python-a-feature-store-and-gcs-17a1462ca489).\n\n**NOTE:** Don't forget to add the GCP credentials to the `.env` file under the `GOOGLE_CLOUD_` prefix:\n* *GOOGLE_CLOUD_PROJECT*: your project name (e.g., \"energy_consumption\")\n* *GOOGLE_CLOUD_BUCKET_NAME*: your bucket name (e.g., \"hourly-batch-predictions\")\n* *GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON_PATH*: absolute path to your JSON key file. (e.g., \"/absolute/path/to/your/service-account.json\")\n\n\n### Deployment\n##### ``\u003c\u003c ~20$ \u003e\u003e``\n\nThis step must only be finished if you want to deploy the code on GCP VMs and build the CI/CD with GitHub Actions.\n\nNote that this step might result in a few costs on GCP. It won't be much. While developing this course, we spent only ~20$.\n\nAlso, you can get some free credits if you create a new GCP account (we created a new account and received 300$ in GCP credits). Just be sure to delete the resources after you finish the course.\n\nSee [this document](/README_DEPLOY.md) for detailed instructions.\n\n-------\n\n# 🔎 8. Usage \u003ca name=usage\u003e\u003c/a\u003e\n\n**The code is fully tested on Ubuntu 20.04 \u0026 22.04 using Python 3.9 and Poetry 1.4.2.**\n\n**Note:** If you are working on macOS M1/M2, be sure to check the [macOS M1/M2 Poetry Issues](https://github.com/iusztinpaul/energy-forecasting/tree/main#macos-m1m2-poetry-issues) section.\n\n## The Pipeline\n\nCheck out [Lesson 4](https://towardsdatascience.com/unlocking-mlops-using-airflow-a-comprehensive-guide-to-ml-system-orchestration-880aa9be8cff) on Medium to better understand how everything is orchestrated using Airflow. \n\n#### Run \nYou will run the pipeline using Airflow (`free usage`). Don't be scared. Docker makes everything very simple to set up.\n\n**Note:** We also hooked the **private PyPi server** in the same docker-compose.yaml file with Airflow. Thus, everything will start with one command.\n\n**Important:** If you plan to run the pipeline outside Airflow, be sure to check the [🧑‍💻 7. Installation \u0026 Usage for Development](https://github.com/iusztinpaul/energy-forecasting/tree/main#-7-installation--usage-for-development-) section.\n\nRun:\n```shell\n# Move to the airflow directory.\ncd airflow\n\n# Make expected directories and environment variables\nmkdir -p ./logs ./plugins\nsudo chmod 777 ./logs ./plugins\n\n# It will be used by Airflow to identify your user.\necho -e \"AIRFLOW_UID=$(id -u)\" \u003e .env\n# This shows where our project root directory is located.\necho \"ML_PIPELINE_ROOT_DIR=/opt/airflow/dags\" \u003e\u003e .env\n```\n\nNow from the `airflow` directory move to the `dags` directory and run:\n```shell\ncd ./dags\n\n# Make a copy of the env default file.\ncp .env.default .env\n# Open the .env file and complete the FS_API_KEY, FS_PROJECT_NAME and WANDB_API_KEY credentials \n\n# Create the folder where the program expects its GCP credentials.\nmkdir -p credentials/gcp/energy_consumption\n# Copy the GCP service credetials that gives you admin access to GCS. \ncp -r /path/to/admin/gcs/credentials/admin-buckets.json credentials/gcp/energy_consumption\n# NOTE that if you want everything to work outside the box your JSON file should be called admin-buckets.json.\n# Otherwise, you have to manually configure the GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON_PATH variable from the .env file. \n```\n\nNow go back to the `airflow` directory and run the following:\n```shell\ncd ..\n\n# Initialize the Airflow database\ndocker compose up airflow-init\n\n# Start up all services\n# Note: You should set up the private PyPi server credentials before running this command.\ndocker compose --env-file .env up --build -d\n```\n\n[Read the official Airflow installation using Docker, but NOTE that we modified their official docker-compose.yaml file.](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html)\n\nWait a while for the containers to build and run. After access `127.0.0.1:8080` to login into Airflow.\u003cbr/\u003e\nUse the following default credentials to log in:\n* username: `airflow`\n* password: `airflow`\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/airflow_login_screenshot.png\"\u003e\n\u003c/p\u003e\n\nBefore starting the pipeline DAG, you must deploy the modules to the private PyPi server. Go back to the `root folder` of the `energy-forecasting` repository and run the following to build and deploy the pipeline modules to your private PyPi server:\n```shell\n# Set the experimental installer of Poetry to False. For us, it crashed when it was on True.\npoetry config experimental.new-installer false\n# Build \u0026 deploy the pipelines modules.\nsh deploy/ml-pipeline.sh\n```\nAirflow will know how to install the packages from the private PyPi server. \u003cbr/\u003e\n\nOne final step is to configure the parameters used to run the pipeline. Go to the `Admin` tab, then hit `Variables.` There you can click on the `blue` `+` button to add a new variable.\nThese are the three parameters you can configure with our suggested values:\n* `ml_pipeline_days_export = 30`\n* `ml_pipeline_feature_group_version = 5`\n* `ml_pipeline_should_run_hyperparameter_tuning = False`\n\u003cbr/\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/airflow_variables_screenshot.png\"\u003e\n\u003c/p\u003e\n\nNow, go to the `DAGS/All` section and search for the `ml_pipeline` DAG. Toggle the activation button. It should automatically start in a few seconds. Also, you can manually run it by hitting the play button from the top-right side of the `ml_pipeline` window.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/airflow_ml_pipeline_dag_overview_screenshot.png\"\u003e\n\u003c/p\u003e\n\nThat is it. You can run the entire pipeline with a single button if all the credentials are set up correctly. How cool is that?\n\nHere is what the DAG should look like 👇\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/airflow_ml_pipeline_dag_screenshot.png\"\u003e\n\u003c/p\u003e\n\n\n#### Clean Up\n```shell\ndocker compose down --volumes --rmi all\n```\n\n#### Backfil Using Airflow\n\nFind your `airflow-webserver` docker container ID:\n```shell\ndocker ps\n```\nStart a shell inside the `airflow-webserver` container and run `airflow dags backfill` as follows (in this example, we did a backfill between `2023/04/11 00:00:00` and `2023/04/13 23:59:59`):\n```shell\ndocker exec -it \u003ccontainer-id-of-airflow-webserver\u003e sh\nairflow dags backfill --start-date \"2023/04/11 00:00:00\" --end-date \"2023/04/13 23:59:59\" ml_pipeline\n```\nIf you want to clear the tasks and run them again, run these commands:\n```shell\ndocker exec -it \u003ccontainer-id-of-airflow-webserver\u003e sh\nairflow tasks clear --start-date \"2023/04/11 00:00:00\" --end-date \"2023/04/13 23:59:59\" ml_pipeline\n```\n\n\n### Run Private PyPi Server Separately\n\nThe private PyPi server is already hooked to the airflow docker compose file. But if you want to run it separately for whatever reason, you can run this command instead:\n```shell\ndocker run -p 80:8080 -v ~/.htpasswd:/data/.htpasswd pypiserver/pypiserver:v1.5.2 run -P .htpasswd/htpasswd.txt --overwrite\n```\n\n------\n\n## The Web App\n\nCheck out [Lesson 6](https://medium.com/towards-data-science/fastapi-and-streamlit-the-python-duo-you-must-know-about-72825def1243) on Medium to better understand how the web app components work together.\n\nFortunately, everything is a lot simpler when setting up the web app. This time, we need to configure only a few credentials. \u003cbr/\u003e\n\n**Important:** If you plan to run the web app components without docker-compose, check the [🧑‍💻 7. Installation \u0026 Usage for Development](https://github.com/iusztinpaul/energy-forecasting/tree/main#-7-installation--usage-for-development-) section.\n\nCopy the bucket read-only GCP credentials to the root directory of your `energy-forecasting` project:\n```shell\n# Create the folder where the program expects its GCP credentials.\nmkdir -p credentials/gcp/energy_consumption\n# Copy the GCP service credetials that gives you read-only access to GCS. \ncp -r /path/to/admin/gcs/credentials/read-buckets.json credentials/gcp/energy_consumption\n# NOTE that if you want everything to work outside the box your JSON file should be called read-buckets.json.\n# Otherwise, you have to manually configure the APP_API_GCP_SERVICE_ACCOUNT_JSON_PATH variable from the .env file of the API.\n```\n\nGo to the API folder and make a copy of the `.env.default` file:\n```shell\ncd ./app-api\ncp .env.default .env\n```\n\n**NOTE:** Remember to complete the `.env` file with your own variables. \n\nThat is it!\n\nGo back to the root directory of your `energy-forecasting` project and run the following docker command, which will build and run all the docker containers of the web app:\n```shell\ndocker compose -f deploy/app-docker-compose.yml --project-directory . up --build\n```\n\nIf you want to run it in development mode, run the following command:\n```shell\ndocker compose -f deploy/app-docker-compose.yml -f deploy/app-docker-compose.local.yml --project-directory . up --build\n```\n\n**Now you can see the apps running at:**\n* [API](http://127.0.0.1:8001/api/v1/docs)\n* [Frontend](http://127.0.0.1:8501/)\n* [Monitoring](http://127.0.0.1:8502/)\n\n-----\n\n## Deploy the Code to GCP\n\n[Check out this section.](./README_DEPLOY.md)\n\n## Set UP CI/CD with GitHub Actions\n\n[Check out this section.](./README_CICD.md)\n\n\n------\n\n\n# 🧑‍💻 9. Installation \u0026 Usage for Development \u003ca name=installation\u003e\u003c/a\u003e\n\nAll the modules support Poetry. Thus the installation is straightforward.\n\n**Note 1:** Just ensure you have installed Python 3.9, not Python 3.8 or Python 3.10.\n\n**Note 2:** During the course, we used `Poetry 1.4.2`. To avoid potential issues when installing the dependencies using Poetry, we recommend you use the same version (or if there are any errors \u0026 you have a different version, you can delete and regenerate the `poetry.lock` file).\n\n**Note 3:** If you are working on macOS M1/M2, be sure to check the [macOS M1/M2 Poetry Issues](https://github.com/iusztinpaul/energy-forecasting/tree/main#macos-m1m2-poetry-issues) section.\n\n## The Pipeline\n\n**We support Docker to run the whole pipeline. Check out the [Usage](#usage) section if you only want to run it as a whole.**\u003cbr/\u003e\n\nIf Poetry is not using Python 3.9, you can follow the next steps:\n1. Install Python 3.9 on your machine.\n2. `cd /path/to/project`, for example, `cd ./feature-pipeline`\n3. run `which python3.9` to find where Python3.9 is located\n4. run `poetry env use /path/to/python3.9`\n\n\n**Every pipeline component must load its credential from the `.env` file. Thus, you have two options:**\n1. **Recommended option:** run `cp .env.default .env` into the folder where the `ML_PIPELINE_ROOT_DIR` env var is pointing to \u0026 fill in the credentials of the `.env` file. Check the [section below](https://github.com/iusztinpaul/energy-forecasting/tree/main#set-up-the-ml_pipeline_root_dir-variable) to see how to set it up. \n2. Create a copy by running `cp .env.default .env` in every pipeline directory individually. But note that by taking this approach, you won't be able to run the system as a whole.\n\n\n**See here how to install every project individually:**\n- [Feature Pipeline](/feature-pipeline/README.md)\n- [Training Pipeline](/training-pipeline/README.md)\n- [Batch Prediction Pipeline](/batch-prediction-pipeline/README.md)\n\u003cbr/\u003e\n\n### Set Up the ML_PIPELINE_ROOT_DIR Variable\n\n**Important:** Before installing and running every module individually, **one key step** is to set the `ML_PIPELINE_ROOT_DIR` variable to your root directory of the `energy-forecasting` project (or any other directory - just make sure to set it):\nExport it to your `~/.bashrc` file:\n```shell\ngedit ~/.bashrc\nexport ML_PIPELINE_ROOT_DIR=/path/to/root/directory/repository/energy-forecasting/\n```\nOr run every Python script proceeded by the `ML_PIPELINE_ROOT_DIR` variables. For example:\n```shell\nML_PIPELINE_ROOT_DIR=/path/to/root/directory/repository/energy-forecasting/ python -m feature_pipeline.pipeline\n```\n\nBy doing so, all the 3 pipeline projects (feature, training, batch) will load and save the following files from the same location:\n* `.env` configuration;\n* JSON metadata files;\n* logs \u0026 plots.\n\n**NOTE:** This step is **critical** as every pipeline component needs to access the JSON metadata from other pipeline processes. By setting up the **ML_PIPELINE_ROOT_DIR** variable, all the metadata JSON files will be saved and accessed from the same location between different processes. For example, the batch prediction pipeline will read the model version it needs to use to make predictions from a JSON file generated by the training pipeline. Without settings the **ML_PIPELINE_ROOT_DIR**, the training and batch processes won't share the same output directory. Thus, they won't know how to talk to each other. When running the project inside `Airflow`, it is defaulted to `/opt/airflow/dags`; thus, you must set this variable only when running it outside Airflow.  \n\u003cbr/\u003e \n\n## The Web App\n**We support Docker to run the web app. Check out the [Usage](#usage) section if you only want to run it as a whole.**\u003cbr/\u003e\n\n**See here how to install every project individually:**\n- [API](/app-api/README.md)\n- [Frontend](/app-frontend/README.md)\n- [Monitoring](/app-monitoring/README.md)\n\nYou can also run the whole web app in development mode using Docker:\n```shell\ndocker compose -f deploy/app-docker-compose.yml -f deploy/app-docker-compose.local.yml --project-directory . up --build\n```\n\n------\n\n# 🏆 10. Licensing \u0026 Contributing \u003ca name=licensing\u003e\u003c/a\u003e\n\nThe code is under the MIT License. Thus, as long as you keep distributing the License, feel free to share, clone, or change the code as you like.\n\nAlso, if you find any bugs or missing pieces in the documentation, I encourage you to add an issue on GitHub or a PR. Based on your support, I will adapt the code and docs for future readers.\n\nFurthermore, you can contact me directly on [LinkedIn](https://www.linkedin.com/in/pauliusztin/) if you have any questions.\n\nI also want to thank [Kurtis Pykes](https://github.com/kurtispykes) for being an awesome copilot and helping me make this course happen.\n\n-----\n\n### Let's connect if you want to level up in designing and productionizing ML systems:\n\nI post almost daily AI content on 👇🏼\n\n[\u003cimg alt=\"linkedin\" width=\"40px\" src=\"images/linkedin.png\" align=\"left\" style=\"padding-right:20px;\"/\u003e](https://www.linkedin.com/in/pauliusztin)\n[\u003cimg alt=\"medium\" width=\"40px\" src=\"images/medium.png\" align=\"left\" style=\"padding-right:20px;\"/\u003e](https://pauliusztin.medium.com/)\n[\u003cimg alt=\"substack\" width=\"35px\" src=\"images/substack.png\" align=\"left\" style=\"padding-right:20px;\"/\u003e](https://pauliusztin.substack.com/)\n[\u003cimg alt=\"gmail\" width=\"40px\" src=\"images/gmail.png\" align=\"left\" style=\"padding-right:20px;\"/\u003e](mailto:p.b.iusztin@gmail.com?subject=[From%20GitHub]%20ML%20Collaborations)\n[\u003cimg alt=\"twitter\" width=\"40px\" src=\"images/twitter.png\" align=\"left\" style=\"padding-right:20px;\"/\u003e](https://twitter.com/iusztinpaul)\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\u003cbr/\u003e\n\nSubscribe to my [ML engineering weekly newsletter](https://pauliusztin.substack.com/).\n\n-----\n\n# 🖤 11. Support \u003ca name=support\u003e\u003c/a\u003e\n\n🎨 **Creating content takes me a lot of time. If you enjoyed my work, you could support me by [buying me a coffee](https://www.buymeacoffee.com/pauliusztin).**\n\n\u003cbr/\u003eThank you ✌🏼 !\u003cbr/\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiusztinpaul%2Fenergy-forecasting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiusztinpaul%2Fenergy-forecasting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiusztinpaul%2Fenergy-forecasting/lists"}