{"id":18400648,"url":"https://github.com/databricks/mlops-stacks","last_synced_at":"2025-05-15T07:03:49.346Z","repository":{"id":59577510,"uuid":"515278486","full_name":"databricks/mlops-stacks","owner":"databricks","description":"This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.","archived":false,"fork":false,"pushed_at":"2025-04-30T07:00:25.000Z","size":40389,"stargazers_count":540,"open_issues_count":6,"forks_count":194,"subscribers_count":26,"default_branch":"main","last_synced_at":"2025-05-15T07:03:12.322Z","etag":null,"topics":["databricks","machine-learning","mlops"],"latest_commit_sha":null,"homepage":"https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-18T17:24:44.000Z","updated_at":"2025-05-14T06:55:15.000Z","dependencies_parsed_at":"2023-09-25T23:13:18.373Z","dependency_job_id":"77c84add-8134-49b0-9ea8-bdd30ef84f1e","html_url":"https://github.com/databricks/mlops-stacks","commit_stats":{"total_commits":134,"total_committers":28,"mean_commits":4.785714285714286,"dds":0.7238805970149254,"last_synced_commit":"702a228f75487acd0bcba9e8ceca3d7346fcf56b"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fmlops-stacks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fmlops-stacks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fmlops-stacks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fmlops-stacks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/mlops-stacks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254291961,"owners_count":22046424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["databricks","machine-learning","mlops"],"created_at":"2024-11-06T02:35:39.440Z","updated_at":"2025-05-15T07:03:49.312Z","avatar_url":"https://github.com/databricks.png","language":"Python","readme":"# Databricks MLOps Stacks\n\n\u003e **_NOTE:_**  This feature is in [public preview](https://docs.databricks.com/release-notes/release-types.html).\n\nThis repo provides a customizable stack for starting new ML projects\non Databricks that follow production best-practices out of the box.\n\nUsing Databricks MLOps Stacks, data scientists can quickly get started iterating on ML code for new projects while ops engineers set up CI/CD and ML resources\nmanagement, with an easy transition to production. You can also use MLOps Stacks as a building block in automation for creating new data science projects with production-grade CI/CD pre-configured. More information can be found at https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html.\n\nThe default stack in this repo includes three modular components: \n\n| Component                   | Description                                                                                                                                                           | Why it's useful                                                                                                                                                                         |\n|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [ML Code](template/{{.input_root_dir}}/{{template%20`project_name_alphanumeric_underscore`%20.}}/)                     | Example ML project structure ([training](template/{{.input_root_dir}}/{{template%20`project_name_alphanumeric_underscore`%20.}}/training) and [batch inference](template/{{.input_root_dir}}/{{template%20`project_name_alphanumeric_underscore`%20.}}/deployment/batch_inference), etc), with unit tested Python modules and notebooks                                                                                           | Quickly iterate on ML problems, without worrying about refactoring your code into tested modules for productionization later on.                                                        |\n| [ML Resources as Code](template/{{.input_root_dir}}/{{template%20`project_name_alphanumeric_underscore`%20.}}/resources) | ML pipeline resources ([training](template/{{.input_root_dir}}/{{template%20`project_name_alphanumeric_underscore`%20.}}/resources/model-workflow-resource.yml.tmpl) and [batch inference](template/{{.input_root_dir}}/{{template%20`project_name_alphanumeric_underscore`%20.}}/resources/batch-inference-workflow-resource.yml.tmpl) jobs, etc) defined through [databricks CLI bundles](https://docs.databricks.com/dev-tools/cli/bundle-cli.html)    | Govern, audit, and deploy changes to your ML resources (e.g. \"use a larger instance type for automated model retraining\") through pull requests, rather than adhoc changes made via UI. |\n| CI/CD([GitHub Actions](template/{{.input_root_dir}}/.github/) or [Azure DevOps](template/{{.input_root_dir}}/.azure/))                       | [GitHub Actions](https://docs.github.com/en/actions) or [Azure DevOps](https://azure.microsoft.com/en-us/products/devops) workflows to test and deploy ML code and resources | Ship ML code faster and with confidence: ensure all production changes are performed through automation and that only tested code is deployed to prod                                   |\n\nSee the [FAQ](#FAQ) for questions on common use cases.\n\n## ML pipeline structure and development loops\n\nAn ML solution comprises data, code, and models. These resources need to be developed, validated (staging), and deployed (production). In this repository, we use the notion of dev, staging, and prod to represent the execution\nenvironments of each stage. \n\nAn instantiated project from MLOps Stacks contains an ML pipeline with CI/CD workflows to test and deploy automated model training and batch inference jobs across your dev, staging, and prod Databricks workspaces. \n\n\u003cimg src=\"doc-images/mlops-stack-summary.png\"\u003e\n\nData scientists can iterate on ML code and file pull requests (PRs). This will trigger unit tests and integration tests in an isolated staging Databricks workspace. Model training and batch inference jobs in staging will immediately update to run the latest code when a PR is merged into main. After merging a PR into main, you can cut a new release branch as part of your regularly scheduled release process to promote ML code changes to production.\n\n### Develop ML pipelines\nhttps://github.com/databricks/mlops-stacks/assets/87999496/00eed790-70f4-4428-9f18-71771051f92a\n\n\n### Create a PR and CI\nhttps://github.com/databricks/mlops-stacks/assets/87999496/f5b3c82d-77a5-4ee5-85f5-8f00b026ae05\n\n\n### Merge the PR and deploy to Staging\nhttps://github.com/databricks/mlops-stacks/assets/87999496/7239e4d0-2327-4d30-91cc-5e7f8328ef73\n\nhttps://github.com/databricks/mlops-stacks/assets/87999496/013c0d32-c283-494b-8c3f-2a9a60366207\n\n\n### Deploy to Prod\nhttps://github.com/databricks/mlops-stacks/assets/87999496/0d220d55-465e-4a69-bd83-1e66ad2e8464\n\n\n[See this page](Pipeline.md) for detailed description and diagrams of the ML pipeline structure defined in the default stack. \n\n## Using MLOps Stacks\n\n### Prerequisites\n - Python 3.8+\n - [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/databricks-cli.html) \u003e= v0.236.0\n\n[Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/databricks-cli.html) contains [Databricks asset bundle templates](https://docs.databricks.com/en/dev-tools/bundles/templates.html) for the purpose of project creation.\n\nPlease follow [the instruction](https://docs.databricks.com/en/dev-tools/cli/databricks-cli-ref.html#install-the-cli) to install and set up databricks CLI. Releases of databricks CLI can be found in the [releases section](https://github.com/databricks/cli/releases) of databricks/cli repository.\n\n[Databricks asset bundles](https://docs.databricks.com/en/dev-tools/bundles/index.html) and [Databricks asset bundle templates](https://docs.databricks.com/en/dev-tools/bundles/templates.html) are in public preview.\n\n\n### Start a new project\n\nTo create a new project, run:\n\n    databricks bundle init mlops-stacks\n\nThis will prompt for parameters for initialization. Some of these parameters are required to get started:\n * ``input_setup_cicd_and_project`` : If both CI/CD and the project should be set up, or only one of them. \n   * ``CICD_and_Project`` - Setup both CI/CD and project, the default option.\n   * ``Project_Only`` - Setup project only, easiest for Data Scientists to get started with.\n   * ``CICD_Only`` - Setup CI/CD only, likely for monorepo setups or setting up CI/CD on an already initialized project.\n   We expect Data Scientists to specify ``Project_Only`` to get \n   started in a development capacity, and when ready to move the project to Staging/Production, CI/CD can be set up. We expect that step to be done by Machine Learning Engineers (MLEs) who can specify ``CICD_Only`` during initialization and use the provided workflow to setup CI/CD for one or more projects.\n * ``input_root_dir``: name of the root directory. When initializing with ``CICD_and_Project``, this field will automatically be set to ``input_project_name``.\n * ``input_cloud``: Cloud provider you use with Databricks (AWS, Azure, or GCP).\n\nOthers must be correctly specified for CI/CD to work:\n * ``input_cicd_platform`` : CI/CD platform of choice. Currently we support GitHub Actions, GitHub Actions for GitHub Enterprise Servers, Azure DevOps and GitLab.\n * ``input_databricks_staging_workspace_host``: URL of staging Databricks workspace, used to preview config changes before they're deployed to production.  \n We encourage granting data scientists working on the current ML project non-admin (read) access to this workspace,\n   to enable them to view and debug CI test results\n * ``input_databricks_prod_workspace_host``: URL of production Databricks workspace. We encourage granting data scientists working on the current ML project non-admin (read) access to this workspace,\n   to enable them to view production job status and see job logs to debug failures.\n * ``input_default_branch``: Name of the default branch, where the prod and staging ML resources are deployed from and the latest ML code is staged.\n * ``input_release_branch``: Name of the release branch. The production jobs (model training, batch inference) defined in this\n    repo pull ML code from this branch.\n\nOr used for project initialization:\n * ``input_project_name``: name of the current project\n * ``input_read_user_group``: User group name to give READ permissions to for project resources (ML jobs, integration test job runs, and machine learning resources). A group with this name must exist in both the staging and prod workspaces. Defaults to \"users\", which grants read permission to all users in the staging/prod workspaces. You can specify a custom group name e.g. to restrict read permissions to members of the team working on the current ML project.\n  * ``input_include_models_in_unity_catalog``: If selected, models will be registered to [Unity Catalog](https://docs.databricks.com/en/mlflow/models-in-uc.html#models-in-unity-catalog). Models will be registered under a three-level namespace of `\u003ccatalog\u003e.\u003cschema_name\u003e.\u003cmodel_name\u003e`, according the the target environment in which the model registration code is executed. Thus, if model registration code runs in the `prod` environment, the model will be registered to the `prod` catalog under the namespace `\u003cprod\u003e.\u003cschema\u003e.\u003cmodel_name\u003e`. This assumes that the respective catalogs exist in Unity Catalog (e.g. `dev`, `staging` and `prod` catalogs). Target environment names, and catalogs to be used are defined in the Databricks bundles files, and can be updated as needed.\n * ``input_schema_name``: If using [Models in Unity Catalog](https://docs.databricks.com/en/mlflow/models-in-uc.html#models-in-unity-catalog), specify the name of the schema under which the models should be registered, but we recommend keeping the name the same as the project name. We default to using the same `schema_name` across catalogs, thus this schema must exist in each catalog used. For example, the training pipeline when executed in the staging environment will register the model to `staging.\u003cschema_name\u003e.\u003cmodel_name\u003e`, whereas the same pipeline executed in the prod environment will register the mode to `prod.\u003cschema_name\u003e.\u003cmodel_name\u003e`. Also, be sure that the service principals in each respective environment have the right permissions to access this schema, which would be `USE_CATALOG`, `USE_SCHEMA`, `MODIFY`, `CREATE_MODEL`, and `CREATE_TABLE`.\n * ``input_unity_catalog_read_user_group``: If using [Models in Unity Catalog](https://docs.databricks.com/en/mlflow/models-in-uc.html#models-in-unity-catalog), define the name of the user group to grant `EXECUTE` (read \u0026 use model) privileges for the registered model. Defaults to \"account users\".\n * ``input_include_feature_store``: If selected, will provide [Databricks Feature Store](https://docs.databricks.com/machine-learning/feature-store/index.html) stack components including: project structure and sample feature Python modules, feature engineering notebooks, ML resource configs to provision and manage Feature Store jobs, and automated integration tests covering feature engineering and training.\n * ``input_include_mlflow_recipes``: If selected, will provide [MLflow Recipes](https://mlflow.org/docs/latest/recipes.html) stack components, dividing the training pipeline into configurable steps and profiles.\n\nSee the generated ``README.md`` for next steps!\n\n## Customize MLOps Stacks\nYour organization can use the default stack as is or customize it as needed, e.g. to add/remove components or\nadapt individual components to fit your organization's best practices. See the\n[stack customization guide](stack-customization.md) for more details.\n\n## FAQ\n\n### Do I need separate dev/staging/prod workspaces to use MLOps Stacks?\nWe recommend using separate dev/staging/prod Databricks workspaces for stronger\nisolation between environments. For example, Databricks REST API rate limits\nare applied per-workspace, so if using [Databricks Model Serving](https://docs.databricks.com/applications/mlflow/model-serving.html),\nusing separate workspaces can help prevent high load in staging from DOSing your\nproduction model serving endpoints.\n\nHowever, you can create a single workspace stack, by supplying the same workspace URL for\n`input_databricks_staging_workspace_host` and `input_databricks_prod_workspace_host`.\nIf you go this route, we\nrecommend using different service principals to manage staging vs prod resources,\nto ensure that CI workloads run in staging cannot interfere with production resources.\n\n### I have an existing ML project. Can I productionize it using MLOps Stacks?\nYes. Currently, you can instantiate a new project and copy relevant components\ninto your existing project to productionize it. MLOps Stacks is modularized, so\nyou can e.g. copy just the GitHub Actions workflows under `.github` or ML resource configs\n under ``{{.input_root_dir}}/{{template `project_name_alphanumeric_underscore` .}}/resources`` \nand ``{{.input_root_dir}}/{{template `project_name_alphanumeric_underscore` .}}/databricks.yml`` into your existing project.\n\n### Can I adopt individual components of MLOps Stacks?\nFor this use case, we recommend instantiating via [Databricks asset bundle templates](https://docs.databricks.com/en/dev-tools/bundles/templates.html) \nand copying the relevant subdirectories. For example, all ML resource configs\nare defined under ``{{.input_root_dir}}/{{template `project_name_alphanumeric_underscore` .}}/resources``\nand ``{{.input_root_dir}}/{{template `project_name_alphanumeric_underscore` .}}/databricks.yml``, while CI/CD is defined e.g. under `.github`\nif using GitHub Actions, or under `.azure` if using Azure DevOps.\n\n### Can I customize my MLOps Stack?\nYes. We provide the default stack in this repo as a production-friendly starting point for MLOps.\nHowever, in many cases you may need to customize the stack to match your organization's\nbest practices. See [the stack customization guide](stack-customization.md)\nfor details on how to do this.\n\n### Does the MLOps Stacks cover data (ETL) pipelines?\n\nSince MLOps Stacks is based on [databricks CLI bundles](https://docs.databricks.com/dev-tools/cli/bundle-commands.html),\nit's not limited only to ML workflows and resources - it works for resources across the Databricks Lakehouse. For instance, while the existing ML\ncode samples contain feature engineering, training, model validation, deployment and batch inference workflows,\nyou can use it for Delta Live Tables pipelines as well.\n\n### How can I provide feedback?\n\nPlease provide feedback (bug reports, feature requests, etc) via GitHub issues.\n\n## Contributing\n\nWe welcome community contributions. For substantial changes, we ask that you first file a GitHub issue to facilitate\ndiscussion, before opening a pull request.\n\nMLOps Stacks is implemented as a [Databricks asset bundle template](https://docs.databricks.com/en/dev-tools/bundles/templates.html)\nthat generates new projects given user-supplied parameters. Parametrized project code can be found under\nthe `{{.input_root_dir}}` directory.\n\n### Installing development requirements\n\nTo run tests, install [actionlint](https://github.com/rhysd/actionlint),\n[databricks CLI](https://docs.databricks.com/dev-tools/cli/databricks-cli.html), [npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm), and\n[act](https://github.com/nektos/act), then install the Python\ndependencies listed in `dev-requirements.txt`:\n\n    pip install -r dev-requirements.txt\n\n### Running the tests\n**NOTE**: This section is for open-source developers contributing to the default stack\nin this repo.  If you are working on an ML project using the stack (e.g. if you ran `databricks bundle init`\nto start a new project), see the `README.md` within your generated\nproject directory for detailed instructions on how to make and test changes.\n\nRun unit tests:\n\n```\npytest tests\n```\n\nRun all tests (unit and slower integration tests):\n\n```\npytest tests --large\n```\n\nRun integration tests only:\n\n```\npytest tests --large-only\n```\n\n### Previewing changes\nWhen making changes to MLOps Stacks, it can be convenient to see how those changes affect\na generated new ML project. To do this, you can create an example\nproject from your local checkout of the repo, and inspect its contents/run tests within\nthe project.\n\nWe provide example project configs for Azure (using both GitHub and Azure DevOps), AWS (using GitHub), and GCP (using GitHub) under `tests/example-project-configs`.\nTo create an example Azure project, using Azure DevOps as the CI/CD platform, run the following from the desired parent directory\nof the example project:\n\n```\n# Note: update MLOPS_STACKS_PATH to the path to your local checkout of the MLOps Stacks repo\nMLOPS_STACKS_PATH=~/mlops-stacks\ndatabricks bundle init \"$MLOPS_STACKS_PATH\" --config-file \"$MLOPS_STACKS_PATH/tests/example-project-configs/azure/azure-devops.json\"\n```\n\nTo create an example AWS project, using GitHub Actions for CI/CD, run:\n```\n# Note: update MLOPS_STACKS_PATH to the path to your local checkout of the MLOps Stacks repo\nMLOPS_STACKS_PATH=~/mlops-stacks\ndatabricks bundle init \"$MLOPS_STACKS_PATH\" --config-file \"$MLOPS_STACKS_PATH/tests/example-project-configs/aws/aws-github.json\"\n```\n\nTo create an example GCP project, using GitHub Actions for CI/CD, run:\n```\n# Note: update MLOPS_STACKS_PATH to the path to your local checkout of the MLOps Stacks repo\nMLOPS_STACKS_PATH=~/mlops-stacks\ndatabricks bundle init \"$MLOPS_STACKS_PATH\" --config-file \"$MLOPS_STACKS_PATH/tests/example-project-configs/gcp/gcp-github.json\"\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fmlops-stacks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fmlops-stacks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fmlops-stacks/lists"}