https://github.com/noahgift/mlflow-project-best-practices
An example MLFlow project
https://github.com/noahgift/mlflow-project-best-practices
Last synced: 6 months ago
JSON representation
An example MLFlow project
- Host: GitHub
- URL: https://github.com/noahgift/mlflow-project-best-practices
- Owner: noahgift
- License: cc0-1.0
- Created: 2022-03-12T23:57:11.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-01-10T22:57:38.000Z (9 months ago)
- Last Synced: 2025-04-09T15:06:11.595Z (6 months ago)
- Language: Jupyter Notebook
- Size: 2.47 MB
- Stars: 48
- Watchers: 2
- Forks: 41
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## 🎓 Pragmatic AI Labs | Join 1M+ ML Engineers
### 🔥 Hot Course Offers:
* 🤖 [Master GenAI Engineering](https://ds500.paiml.com/learn/course/0bbb5/) - Build Production AI Systems
* 🦀 [Learn Professional Rust](https://ds500.paiml.com/learn/course/g6u1k/) - Industry-Grade Development
* 📊 [AWS AI & Analytics](https://ds500.paiml.com/learn/course/31si1/) - Scale Your ML in Cloud
* ⚡ [Production GenAI on AWS](https://ds500.paiml.com/learn/course/ehks1/) - Deploy at Enterprise Scale
* 🛠️ [Rust DevOps Mastery](https://ds500.paiml.com/learn/course/ex8eu/) - Automate Everything
### 🚀 Level Up Your Career:
* 💼 [Production ML Program](https://paiml.com) - Complete MLOps & Cloud Mastery
* 🎯 [Start Learning Now](https://ds500.paiml.com) - Fast-Track Your ML Career
* 🏢 Trusted by Fortune 500 Teams
Learn end-to-end ML engineering from industry veterans at [PAIML.COM](https://paiml.com)
# mlflow-project-best-practices
An example MLFlow project
[](https://github.com/noahgift/mlflow-project-best-practices/actions/workflows/main.yml)
## High Level Architecture
You can Notebook see the Databricks code here: https://github.com/noahgift/mlflow-project-best-practices/blob/main/XGBoost-fake-news-automl.ipynb
## Example in Azure CloudShell
## Example in Github Codespaces
### Curl Example
```
curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json; format=pandas-records" \
-d@data.json \
https://adb-2951765055089996.16.azuredatabricks.net/model/Fake-News/1/invocations
```
## Deploy Architecture
## Query registered model
* [Azure Databricks Model Serving Architecture](https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/model-serving)
## Query with Databricks CLI
* [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
1. Create a databricks config:
`touch ~/.databrickscfg`
2. Put in host and token
3. Query jobs
`databricks jobs list --output JSON | jq`
4. List clusters
`databricks clusters list --output JSON | jq `
5. List contents of DBFS
`databricks fs ls dbfs:/`
## Query for Models with API
* [Refer to Quikstart Python](https://docs.databricks.com/applications/mlflow/quick-start-python.html)
* [Refer to Create or Register Model](https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/manage-model-lifecycle/#create-or-register-a-model)
* [Access tracking server external](https://docs.databricks.com/applications/mlflow/access-hosted-tracking-server.html)
You need to set the tracking URI.
```bash
export MLFLOW_TRACKING_URI=databricks
```
```python
from pprint import pprint
from mlflow.tracking import MlflowClient
client = MlflowClient()
for rm in client.list_registered_models():
pprint(dict(rm), indent=4)
```
## Download Model
* [Download model artifacts](https://docs.databricks.com/applications/mlflow/models.html#download-model-artifacts)
CLI version
```
mlflow artifacts download --artifact-uri models://
````
To use Python do the following:
```python
from mlflow.store.artifact.models_artifact_repo import ModelsArtifactRepository
model_uri = MlflowClient.get_model_version_download_uri(model_name, model_version)
ModelsArtifactRepository(model_uri).download_artifacts(artifact_path="")
```
## Register model for AWS Sagemaker
* [Register model for AWS Sagemaker](https://docs.databricks.com/applications/mlflow/scikit-learn-model-deployment-on-sagemaker.html)
## Goal for today
* Databricks Tensorflow run
* Feature Store
* Model Serving
### Model Serving
https://docs.databricks.com/applications/mlflow/model-serving.html
Run it with `mlserve`
```
mlflow models serve --model-uri /workspaces/mlflow-project-best-practices/tf-model
```
## References
* [Watch Youtube Walkthrough](https://studio.youtube.com/video/PUXhWZQW8BI/edit?c=UCNDfiL0D1LUeKWAkRE1xO5Q)
* [Watch Walkthrough on O'Reilly](https://learning.oreilly.com/videos/mlops-platforms-from/032232022VIDEOPAIML/)