{"id":19944242,"url":"https://github.com/lazauk/azureml-blockcompute-serverlessspark","last_synced_at":"2025-10-08T08:57:19.779Z","repository":{"id":249191761,"uuid":"830458417","full_name":"LazaUK/AzureML-BlockCompute-ServerlessSpark","owner":"LazaUK","description":"Quick guide on blocking Serverless Spark in Azure Machine Learning with Azure Policy","archived":false,"fork":false,"pushed_at":"2024-08-15T22:13:08.000Z","size":402,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-26T07:42:59.750Z","etag":null,"topics":["azure","machine-learning","policy","security","serverless-spark"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LazaUK.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-18T10:06:55.000Z","updated_at":"2024-08-15T22:13:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"afea5f45-2ca6-46e6-80fd-96cae00712bf","html_url":"https://github.com/LazaUK/AzureML-BlockCompute-ServerlessSpark","commit_stats":null,"previous_names":["lazauk/azureml-blockcompute-serverlessspark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LazaUK/AzureML-BlockCompute-ServerlessSpark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FAzureML-BlockCompute-ServerlessSpark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FAzureML-BlockCompute-ServerlessSpark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FAzureML-BlockCompute-ServerlessSpark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FAzureML-BlockCompute-ServerlessSpark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LazaUK","download_url":"https://codeload.github.com/LazaUK/AzureML-BlockCompute-ServerlessSpark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FAzureML-BlockCompute-ServerlessSpark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278916432,"owners_count":26068090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","machine-learning","policy","security","serverless-spark"],"created_at":"2024-11-13T00:19:38.769Z","updated_at":"2025-10-08T08:57:19.762Z","avatar_url":"https://github.com/LazaUK.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Block Serverless Spark in Azure Machine Learning with Azure Policy\nThis brief guide will help you create an Azure Policy definition to block users from utilising Serverless Spark compute targets within Azure Machine Learning.\n\n### Prerequisites:\n- An Azure subscription with appropriate permissions to create policies;\n- Familiarity with setup of Azure Policies programmatically or in Azure portal UI.\n\n### Configuration Steps:\n\n1. Create policy definition file by copy-pasting the following JSON code or download provided [blockServerlessSpark.json](blockServerlessSpark.json) file:\n``` JSON\n{\n    \"if\": {\n        \"allOf\": [\n            {\n                \"field\": \"Microsoft.MachineLearningServices/workspaces/jobs/jobType\",\n                \"in\": [\n                    \"Spark\"\n                ]\n            }\n        ]\n    },\n    \"then\": {\n        \"effect\": \"deny\"\n    }\n}\n```\n\n\u003e This policy checks for resources of type \"Microsoft.MachineLearningServices/workspaces/jobs\" with the property \"jobType\" set to \"Spark\". If it finds any (typically at the time of Serverless Spark job submission), it will trigger _**Deny**_ effect.\n\n2. Create the Azure Policy definition.\n - You can do it programmatically as described in [Azure Policy documentation](https://learn.microsoft.com/en-us/azure/governance/policy/how-to/programmatically-create). Here's an example of using Azure CLI:\n``` Bash\naz policy definition create --name \"block-serverless-spark-azureml\" --display-name \"Block the use of Serverless Spark in Azure ML\" --description \"This policy blocks the use of Serverless Spark in Azure ML\" --rules \u003cPATH TO JSON DEFINITION FILE, e.g. blockServerlessSpark.json\u003e --mode All\n```\n- Alternatively, you can create new policy definition directly in Azure portal:\n![Az_Policy_Definition_UI](images/azpolicy_definition.png)\n\n3. Assign the policy and define its scope, i.e. what to include and evaluate for compliance. You can refer to Microsoft documentation for details on [assigning policies](https://learn.microsoft.com/en-us/azure/governance/policy/concepts/assignment-structure).\n\n4. To test the effect of newly created policy, try to create a new Serverless Spark job in Azure Machine Learning workspace. The policy should block its creation as shown on example below:\n![Az_Policy_Effect](images/azpolicy_effect.png)\n\n5. In selected geographies, such Azure policy would also block attempts to launch Serverless Spark sessions from within Azure Machine Learning's Jupyter notebooks:\n![Az_Policy_Effect_Notebook](images/azpolicy_notebook.png)\n\n\u003e Note: This is a basic policy example. You can customise it further based on your needs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazauk%2Fazureml-blockcompute-serverlessspark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flazauk%2Fazureml-blockcompute-serverlessspark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazauk%2Fazureml-blockcompute-serverlessspark/lists"}