https://github.com/lazauk/azureml-blockcompute-serverlessspark

Quick guide on blocking Serverless Spark in Azure Machine Learning with Azure Policy
https://github.com/lazauk/azureml-blockcompute-serverlessspark

azure machine-learning policy security serverless-spark

Last synced: 3 months ago
JSON representation

Quick guide on blocking Serverless Spark in Azure Machine Learning with Azure Policy

• Host: GitHub
• URL: https://github.com/lazauk/azureml-blockcompute-serverlessspark
• Owner: LazaUK
• License: mit
• Created: 2024-07-18T10:06:55.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-08-15T22:13:08.000Z (9 months ago)
• Last Synced: 2025-01-12T05:11:07.375Z (4 months ago)
• Topics: azure, machine-learning, policy, security, serverless-spark
• Homepage:
• Size: 393 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Block Serverless Spark in Azure Machine Learning with Azure Policy

This brief guide will help you create an Azure Policy definition to block users from utilising Serverless Spark compute targets within Azure Machine Learning.

### Prerequisites:

- An Azure subscription with appropriate permissions to create policies;

- Familiarity with setup of Azure Policies programmatically or in Azure portal UI.

### Configuration Steps:

1. Create policy definition file by copy-pasting the following JSON code or download provided [blockServerlessSpark.json](blockServerlessSpark.json) file:

``` JSON

{

    "if": {

        "allOf": [

            {

                "field": "Microsoft.MachineLearningServices/workspaces/jobs/jobType",

                "in": [

                    "Spark"

                ]

            }

        ]

    },

    "then": {

        "effect": "deny"

    }

}

```

> This policy checks for resources of type "Microsoft.MachineLearningServices/workspaces/jobs" with the property "jobType" set to "Spark". If it finds any (typically at the time of Serverless Spark job submission), it will trigger _**Deny**_ effect.

2. Create the Azure Policy definition.

 - You can do it programmatically as described in [Azure Policy documentation](https://learn.microsoft.com/en-us/azure/governance/policy/how-to/programmatically-create). Here's an example of using Azure CLI:

``` Bash

az policy definition create --name "block-serverless-spark-azureml" --display-name "Block the use of Serverless Spark in Azure ML" --description "This policy blocks the use of Serverless Spark in Azure ML" --rules  --mode All

```

- Alternatively, you can create new policy definition directly in Azure portal:

![Az_Policy_Definition_UI](images/azpolicy_definition.png)

3. Assign the policy and define its scope, i.e. what to include and evaluate for compliance. You can refer to Microsoft documentation for details on [assigning policies](https://learn.microsoft.com/en-us/azure/governance/policy/concepts/assignment-structure).

4. To test the effect of newly created policy, try to create a new Serverless Spark job in Azure Machine Learning workspace. The policy should block its creation as shown on example below:

![Az_Policy_Effect](images/azpolicy_effect.png)

5. In selected geographies, such Azure policy would also block attempts to launch Serverless Spark sessions from within Azure Machine Learning's Jupyter notebooks:

![Az_Policy_Effect_Notebook](images/azpolicy_notebook.png)

> Note: This is a basic policy example. You can customise it further based on your needs.