{"id":13720012,"url":"https://github.com/tezansahu/dvc-pycaret-fastapi-demo","last_synced_at":"2025-12-26T15:33:40.845Z","repository":{"id":121148423,"uuid":"449784700","full_name":"tezansahu/dvc-pycaret-fastapi-demo","owner":"tezansahu","description":"Repository for the Demo of using DVC with PyCaret \u0026 MLOps (DVC Office Hours - 20th Jan, 2022)","archived":false,"fork":false,"pushed_at":"2022-01-20T17:06:29.000Z","size":918,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T17:51:51.861Z","etag":null,"topics":["data-science","demo","deployment","dvc","fastapi","machine-learning","mlops-workflow","pycaret"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tezansahu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-01-19T17:11:26.000Z","updated_at":"2024-11-19T20:47:15.000Z","dependencies_parsed_at":"2024-01-27T11:09:10.381Z","dependency_job_id":"a2d9733b-597b-416f-aa3d-370559991f1d","html_url":"https://github.com/tezansahu/dvc-pycaret-fastapi-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tezansahu%2Fdvc-pycaret-fastapi-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tezansahu%2Fdvc-pycaret-fastapi-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tezansahu%2Fdvc-pycaret-fastapi-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tezansahu%2Fdvc-pycaret-fastapi-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tezansahu","download_url":"https://codeload.github.com/tezansahu/dvc-pycaret-fastapi-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252876287,"owners_count":21818157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","demo","deployment","dvc","fastapi","machine-learning","mlops-workflow","pycaret"],"created_at":"2024-08-03T01:00:58.918Z","updated_at":"2025-12-26T15:33:40.809Z","avatar_url":"https://github.com/tezansahu.png","language":"Jupyter Notebook","funding_links":[],"categories":["Tutorials"],"sub_categories":[],"readme":"# Using DVC with PyCaret \u0026 FastAPI (Demo)\n\nThis repo contains all the resources for my demo explaining how to use [DVC](https://dvc.org/) along with other interesting tools \u0026 frameworks like [PyCaret](https://pycaret.org/) \u0026 [FastAPI](https://fastapi.tiangolo.com/) for data \u0026 model versioning, experimentation with ML models \u0026 finally deploying these models quickly for inferencing.\n\n_This demo was presented at the DVC Office Hours on 20th Jan 2022._\n\n\u003e _**Note:** We will use Azure Blob Storage as our remote storage for this demo. To follow along, it is advised to either create an Azure account or use a different remote for storage._\n\n***\n\n## Steps Followed for the Demo\n\n### 0. Preliminaries\n\nCreate a virtual environment named `dvc-demo` \u0026 install required packages\n\n```bash\npython3 -m venv dvc-demo\nsource dvc-demo/bin/activate\n\npip install dvc[azure] pycaret fastapi uvicorn python-multipart\n```\n\nInitialize the repo with DVC tracking \u0026 create a `data/` folder\n\n```bash\nmkdir dvc-pycaret-fastapi-demo\ncd dvc-pycaret-fastapi-demo\ngit init\ndvc init\n\ngit remote add origin https://github.com/tezansahu/dvc-pycaret-fastapi-demo.git\n\nmkdir data\n```\n\n### 1. Tracking Data with DVC\n\nWe use the [Heart Failure Prediction Dataset](https://www.kaggle.com/fedesoriano/heart-failure-prediction) for this demo.\n\nFirst, we download the `heart.csv` file \u0026 retain ~800 rows from this file in the `data/` folder. _(We will use the file with all the rows later - this is to simulate the change/increase in data that an ML workflow sees during its lifetime)_\n\nTrack this `data/heart.csv` using DVC\n\n```bash\ndvc add data/heart.csv\ngit add data/heart.csv.dvc\ngit commit -m \"add data - phase 1\"\n```\n\n### 2. Setup the Remote for Storing Tracked Data \u0026 Models\n\n- Go to the Azure Portal \u0026 create a **Storage Account** (here, we name it `dvcdemo`)\n  ![Creating a Storage Account on Azure](./images/azure_create_storage_account.png)\n\n- Within the storage account, create a **Container** (here, we name it `demo20jan2022`)\n\n- Obtain the **Connection String** from the storage account as follows:\n  ![Obtaining the Connection String for a Storage Account on Azure](./images/azure_connection_string.png) \n\n- Install the Azure CLI from [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli) \u0026 log into Azure from within the terminal using `az login`\n\nNow, we store the tracked data in Azure:\n\n```bash\ndvc remote add -d storage azure://demo20jan2022/dvcstore\ndvc remote modify --local storage connection_string \u003cconnection-string\u003e\n\ndvc push\ngit push origin main\n```\n\n### 3. ML Experimentation with PyCaret\n\nCreate the `notebooks/` folders using `mkdir notebook` \u0026 download the `notebooks/experimentation_with_pycaret.ipynb` notebook from this repo into this `notebooks/` folder.\n\nTrack this notebook with Git:\n\n```bash\ngit add notebooks/\ngit commit -m \"add ml training notebook\"\n```\n\nRun all the cells mentioned under **Phase 1** in the notebook. This involves basics of PyCaret:\n- Setting up a vanilla experiment with `setup()`\n- Comparing various classification models with `compare_models()`\n- Evaluating the preformance a model with `evaluate_model()`\n- Making predictions on the held-out eval data using `predict_model()`\n- Finalizing the model by training on the full training + eval data using `finalize_model()`\n- Saving the model pipeline using `save_model()`\n\nThis will create a `model.pkl` file in the `models/` folder\n\n### 4. Tracking Models with DVC\n\nNow, we track the ML model using DVC \u0026 store it in our remote storage\n\n```bash\ndvc add models/model.pkl\ngit add models/model.pkl.dvc\ngit commit -m \"add model - phase 1\"\n\ndvc push\ngit push origin main\n```\n\n### 5. Deploy the Model with FastAPI\n\n_First, delete the `.dvc/cache/` \u0026 `models/model.pkl` (simulate production env). Then, pull the changes from the DVC remote storage._\n\n```bash\ndvc pull\n```\n\nCheck that the `model.pkl` file is now present in `models/` folder.\n\nNow, create a `server/` folder \u0026 place the `main.py` file in it after downloaidng the `server/main.py` file from this repo. This RESTful API server has 2 POST endpoints:\n- Inferencing on an individual record\n- Batch inferencing on a CSV file\n\nWe commit this to our repo:\n\n```bash\ngit add server/\ngit commit -m \"create basic fastapi server\"\n```\n\nNow, we can run our local server on port 8000\n\n```bash\ncd server\nuvicorn main:app --port=8000\n```\n\nGo to `http://localhost:8000/docs` \u0026 play with the endpoints present in the interactive documentation. \n\n![Swagger Interactive API Documentation for our Server](./images/api_server_docs.png)\n\nFor the individual inference, you could use teh following data:\n\n```json\n{\n  \"Age\": 61,\n  \"Sex\": \"M\",\n  \"ChestPainType\": \"ASY\",\n  \"RestingBP\": 148,\n  \"Cholesterol\": 203,\n  \"FastingBS\": 0,\n  \"RestingECG\": \"Normal\",\n  \"MaxHR\": 161,\n  \"ExerciseAngina\": \"N\",\n  \"Oldpeak\": 0,\n  \"ST_Slope\": \"Up\"\n}\n```\n\n### 6. Simulating the arrival of New Data\n\nNow, we use the **full** `heart.csv` file to simulate the arrival of new data with time. We place it within `data/` folder \u0026 upload it to DVC remote.\n\n```bash\ndvc add data/heart.csv\ngit add data/heart.csv.dvc\ngit commit -m \"add data - phase 2\"\n\ndvc push\ngit push origin main\n```\n\n### 7. More Experimentation with PyCaret\n\nNow, we run the experiment in **Phase 2** of the `notebooks/experimentation_with_pycaret.ipynb` notebook. This involves:\n- Feature engineering while setting up teh experient\n- Fine-tuning of models with `tune_model()`\n- Creating an ensemble of models with `blend_models()`\n\nThe blended model is saved as `models/modl.pkl`\n\nWe upload it to our DVC remote.\n\n```bash\ndvc add models/model.pkl\ngit add models/model.pkl.dvc\ngit commit -m \"add model - phase 2\"\n\ndvc push\ngit push origin main\n```\n### 8. Redeploying the New Model using FastAPI\n\nNow, we again start the server (no code changes required, because the model file has same name) \u0026 perform inference.\n\n```bash\ncd server\nuvicorn main:app --port=8000\n```\n\nWith this, we demonstrate how DVC can be used in conjunction with PyCaret \u0026 FastAPI for iterating \u0026 experimenting efficiently with ML models \u0026 deploying them with minimal effort.\n\n***\n\n## Additional Resources\n\n- Fundamentals of MLOps: A 4-blog series\n    - [A Gentle Introduction to MLOps](https://medium.com/analytics-vidhya/fundamentals-of-mlops-part-1-a-gentle-introduction-to-mlops-1b184d2c32a8)\n    - [Data \u0026 Model Management with DVC](https://medium.com/analytics-vidhya/fundamentals-of-mlops-part-2-data-model-management-with-dvc-6be2ad284ec4)\n    - [ML Experimentation using PyCaret](https://medium.com/analytics-vidhya/fundamentals-of-mlops-part-3-ml-experimentation-using-pycaret-747f14e4c28d)\n    - [Tracking with MLFlow \u0026 Deployment with FastAPI](https://medium.com/analytics-vidhya/fundamentals-of-mlops-part-4-tracking-with-mlflow-deployment-with-fastapi-61614115436)\n- [DVC Documentation](https://dvc.org/doc)\n- [PyCaret Documentation](https://pycaret.gitbook.io/docs/)\n- [FastAPI Documentation](https://fastapi.tiangolo.com/)\n\n***\n\n\u003cp align=\"center\"\u003eCreated with ❤️ by \u003ca href=\"https://www.linkedin.com/in/tezan-sahu/\"\u003eTezan Sahu\u003c/a\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftezansahu%2Fdvc-pycaret-fastapi-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftezansahu%2Fdvc-pycaret-fastapi-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftezansahu%2Fdvc-pycaret-fastapi-demo/lists"}