{"id":20515298,"url":"https://github.com/perfectthymetech/terraform-azurerm-data-management-zone","last_synced_at":"2025-08-18T05:44:41.591Z","repository":{"id":169063307,"uuid":"629679995","full_name":"PerfectThymeTech/terraform-azurerm-data-management-zone","owner":"PerfectThymeTech","description":"Cloud Scale Analytics - Data Management Zone Terraform Module","archived":false,"fork":false,"pushed_at":"2025-03-16T10:13:05.000Z","size":6909,"stargazers_count":3,"open_issues_count":9,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-27T14:21:18.547Z","etag":null,"topics":["architecture","azure","cloud-scale-analytics","cloudscaleanalytics","data-management","data-platform","datamesh","enterprise-architecture","enterprise-scale","enterprise-scale-analytics","terraform","terraform-module"],"latest_commit_sha":null,"homepage":"https://registry.terraform.io/modules/PerfectThymeTech/data-management-zone/azurerm/latest","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PerfectThymeTech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["PerfectThymeTech","marvinbuss"]}},"created_at":"2023-04-18T20:06:32.000Z","updated_at":"2024-12-23T21:11:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"f2fcc01d-7a47-4160-ac69-03522da22e10","html_url":"https://github.com/PerfectThymeTech/terraform-azurerm-data-management-zone","commit_stats":{"total_commits":76,"total_committers":3,"mean_commits":"25.333333333333332","dds":"0.35526315789473684","last_synced_commit":"3835c2d511612531da2489db1ff337a0c0190250"},"previous_names":["perfectthymetech/terraform-azurerm-data-management-zone"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectThymeTech%2Fterraform-azurerm-data-management-zone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectThymeTech%2Fterraform-azurerm-data-management-zone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectThymeTech%2Fterraform-azurerm-data-management-zone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerfectThymeTech%2Fterraform-azurerm-data-management-zone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PerfectThymeTech","download_url":"https://codeload.github.com/PerfectThymeTech/terraform-azurerm-data-management-zone/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248799968,"owners_count":21163404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["architecture","azure","cloud-scale-analytics","cloudscaleanalytics","data-management","data-platform","datamesh","enterprise-architecture","enterprise-scale","enterprise-scale-analytics","terraform","terraform-module"],"created_at":"2024-11-15T21:20:43.386Z","updated_at":"2025-04-14T00:18:25.313Z","avatar_url":"https://github.com/PerfectThymeTech.png","language":"HCL","funding_links":["https://github.com/sponsors/PerfectThymeTech","https://github.com/sponsors/marvinbuss"],"categories":[],"sub_categories":[],"readme":"\u003c!-- BEGIN_TF_DOCS --\u003e\n# CloudScaleAnalytics v2 - Data Management Zone\n\nThis project revisits the Cloud Scale Analytics data platform reference architecture for Microsoft Azure. While the core principles of the architecture design have not changed, the next generation of the design will and enhance and introduce many new capabilities that will simplify the overall management, onboarding and significantly reduce the time to market.\n\nOver the last couple of years, numerous data platforms have been built on the basis of Cloud Scale Analytics which resulted in a ton of learnings and insights. In addition to that, new services and features have been introduced, reached a GA status and common requirements have drifted. All these data points have been used to build this next iteration of the reference architecture for scalable data platforms on Azure.\n\nThe Cloud Scale Analytics reference architecture consists of the following core building blocks:\n\n1. The *Data Management Zone* is the core data governance entity of on organization. In this Azure subscription, an organization places all data management solution including their data catalog, the data lineage solution, the master data management tool and other data governance capabilities. Placing these tools inside a single subscription ensures a resusable data management framework that can be applied to all *Data Landing Zones* and other data sources across an organization.\n\n2. The *Data Landing Zone* is used for data retention and processing. A *Data Landing Zone* maps to a single Azure Subscription, but organizations are encouraged to have multiple of these for scaling purposes. Within a *Data Landing Zone* an orgnaization may implement one or multiple data applications.\n\n3. A *Data Application* environment is a bounded context within a *Data Landing Zone*. A *Data Application* is concerned with consuming, processing and producing data as an output. These outputs should no longer be treated as byproducts but rather be managed as a full product that has a defined service-level-agreement.\n\n![Cloud-scale Analytics v2](https://raw.githubusercontent.com/PerfectThymeTech/terraform-azurerm-data-management-zone/main/docs/media/CloudScaleAnalyticsv2.gif)\n\n## Architecture\n\nThe following architecture will be deployed by this module, whereby the module expects that the Vnet, Route Table and NSG already exists within the Azure Landing Zone and respective resource IDs are provided as input:\n\n![Data Management Zone Architecture](https://raw.githubusercontent.com/PerfectThymeTech/terraform-azurerm-data-management-zone/main/docs/media/DataManagementZoneArchitecture.png)\n\n## Prerequisites\n\n- An Azure subscription. If you don't have an Azure subscription, [create your Azure free account today](https://azure.microsoft.com/free/).\n- (1) [Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor) and [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) or (2) [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner) access to the subscription to be able to create resources and role assignments.\n- [Databricks Account Administrator](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/#--what-are-account-admins) role in the Databricks Account.\n- A [GitHub self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) or an [Azure DevOps self-hosted agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops) to be able to access the data-plane of services.\n\n## Usage\n\nWe recommend starting with the following configuration in your root module to learn what resources are created by the module and how it works.\n\n```hcl\n# Configure Terraform to set the required AzureRM provider version and features{} block.\n\nterraform {\n  required_providers {\n    azurerm = {\n      source  = \"hashicorp/azurerm\"\n      version = \"3.57.0\"\n    }\n    azapi = {\n      source  = \"azure/azapi\"\n      version = \"1.6.0\"\n    }\n    databricks = {\n      source  = \"databricks/databricks\"\n      version = \"1.17.0\"\n    }\n  }\n}\n\nprovider \"azurerm\" {\n  features {}\n}\n\nprovider \"azapi\" {}\n\n# Declare locals for the module\nlocals {\n  company_name     = \"\u003cmy-company-name\u003e\"\n  location         = \"northeurope\"\n  location_purview = \"northeurope\"\n  prefix           = \"\u003cmy-prefix\u003e\"\n  vnet_id          = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/virtualNetworks/\u003cmy-vnet-name\u003e\"\n  nsg_id           = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/networkSecurityGroups/\u003cmy-nsg-name\u003e\"\n  route_table_id   = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/routeTables/\u003cmy-rt-name\u003e\"\n\n  # If DNS A-records are deployed via Policy then you can also set these to an empty string (e.g. \"\") or remove them entirely\n  private_dns_zone_id_namespace          = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.servicebus.windows.net\"\n  private_dns_zone_id_purview_account    = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.purview.azure.com\"\n  private_dns_zone_id_purview_portal     = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.purviewstudio.azure.com\"\n  private_dns_zone_id_blob               = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net\"\n  private_dns_zone_id_dfs                = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.dfs.core.windows.net\"\n  private_dns_zone_id_queue              = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.queue.core.windows.net\"\n  private_dns_zone_id_container_registry = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.azurecr.io\"\n  private_dns_zone_id_synapse_portal     = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.azuresynapse.net\"\n  private_dns_zone_id_key_vault          = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net\"\n  private_dns_zone_id_databricks         = \"/subscriptions/\u003cmy-subscription-id\u003e/resourceGroups/\u003cmy-rg-name\u003e/providers/Microsoft.Network/privateDnsZones/privatelink.azuredatabricks.net\"\n}\n\n# Declare the Data Management Zone Terraform module and provide a base configuration.\nmodule \"data_management_zone\" {\n  source  = \"PerfectThymeTech/data-management-zone/azurerm\"\n  version = \"0.1.1\"\n  providers = {\n    azurerm = azurerm\n    azapi   = azapi\n  }\n\n  company_name                           = local.company_name\n  location                               = local.location\n  location_purview                       = local.location_purview\n  prefix                                 = local.prefix\n  vnet_id                                = local.vnet_id\n  nsg_id                                 = local.nsg_id\n  route_table_id                         = local.route_table_id\n  private_dns_zone_id_namespace          = local.private_dns_zone_id_namespace\n  private_dns_zone_id_purview_account    = local.private_dns_zone_id_purview_account\n  private_dns_zone_id_purview_portal     = local.private_dns_zone_id_purview_portal\n  private_dns_zone_id_blob               = local.private_dns_zone_id_blob\n  private_dns_zone_id_dfs                = local.private_dns_zone_id_dfs\n  private_dns_zone_id_queue              = local.private_dns_zone_id_queue\n  private_dns_zone_id_container_registry = local.private_dns_zone_id_container_registry\n  private_dns_zone_id_synapse_portal     = local.private_dns_zone_id_synapse_portal\n  private_dns_zone_id_key_vault          = local.private_dns_zone_id_key_vault\n  private_dns_zone_id_databricks         = local.private_dns_zone_id_databricks\n}\n```\n\n## Documentation\n\u003c!-- markdownlint-disable MD033 --\u003e\n\n## Requirements\n\nThe following requirements are needed by this module:\n\n- \u003ca name=\"requirement_terraform\"\u003e\u003c/a\u003e [terraform](#requirement\\_terraform) (\u003e=0.13)\n\n- \u003ca name=\"requirement_azapi\"\u003e\u003c/a\u003e [azapi](#requirement\\_azapi) (~\u003e 2.0)\n\n- \u003ca name=\"requirement_azurerm\"\u003e\u003c/a\u003e [azurerm](#requirement\\_azurerm) (~\u003e 4.0)\n\n- \u003ca name=\"requirement_databricks\"\u003e\u003c/a\u003e [databricks](#requirement\\_databricks) (~\u003e 1.58)\n\n- \u003ca name=\"requirement_time\"\u003e\u003c/a\u003e [time](#requirement\\_time) (~\u003e 0.9)\n\n## Modules\n\nThe following Modules are called:\n\n### \u003ca name=\"module_datamanagement\"\u003e\u003c/a\u003e [datamanagement](#module\\_datamanagement)\n\nSource: ./modules/datamanagement\n\nVersion:\n\n### \u003ca name=\"module_platform\"\u003e\u003c/a\u003e [platform](#module\\_platform)\n\nSource: ./modules/platform\n\nVersion:\n\n\u003c!-- markdownlint-disable MD013 --\u003e\n\u003c!-- markdownlint-disable MD034 --\u003e\n## Required Inputs\n\nThe following input variables are required:\n\n### \u003ca name=\"input_company_name\"\u003e\u003c/a\u003e [company\\_name](#input\\_company\\_name)\n\nDescription: Specifies the name of the company.\n\nType: `string`\n\n### \u003ca name=\"input_location\"\u003e\u003c/a\u003e [location](#input\\_location)\n\nDescription: Specifies the location for all Azure resources.\n\nType: `string`\n\n### \u003ca name=\"input_location_purview\"\u003e\u003c/a\u003e [location\\_purview](#input\\_location\\_purview)\n\nDescription: Specifies the location for Microsoft Purview. The location of Purview is bound to the Microsoft Entra ID location.\n\nType: `string`\n\n### \u003ca name=\"input_nsg_id\"\u003e\u003c/a\u003e [nsg\\_id](#input\\_nsg\\_id)\n\nDescription: Specifies the resource ID of the default network security group for the Data Management Zone\n\nType: `string`\n\n### \u003ca name=\"input_prefix\"\u003e\u003c/a\u003e [prefix](#input\\_prefix)\n\nDescription: Specifies the prefix for all resources created in this deployment.\n\nType: `string`\n\n### \u003ca name=\"input_route_table_id\"\u003e\u003c/a\u003e [route\\_table\\_id](#input\\_route\\_table\\_id)\n\nDescription: Specifies the resource ID of the default route table for the Data Management Zone\n\nType: `string`\n\n### \u003ca name=\"input_vnet_id\"\u003e\u003c/a\u003e [vnet\\_id](#input\\_vnet\\_id)\n\nDescription: Specifies the resource ID of the Vnet used for the Data Management Zone\n\nType: `string`\n\n## Optional Inputs\n\nThe following input variables are optional (have default values):\n\n### \u003ca name=\"input_customer_managed_key\"\u003e\u003c/a\u003e [customer\\_managed\\_key](#input\\_customer\\_managed\\_key)\n\nDescription: Specifies the customer managed key configurations.\n\nType:\n\n```hcl\nobject({\n    key_vault_id                     = string,\n    key_vault_key_versionless_id     = string,\n    user_assigned_identity_id        = string,\n    user_assigned_identity_client_id = string,\n  })\n```\n\nDefault: `null`\n\n### \u003ca name=\"input_databricks_account_id\"\u003e\u003c/a\u003e [databricks\\_account\\_id](#input\\_databricks\\_account\\_id)\n\nDescription: Specifies the id of the databricks account.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_environment\"\u003e\u003c/a\u003e [environment](#input\\_environment)\n\nDescription: Specifies the environment of the deployment.\n\nType: `string`\n\nDefault: `\"dev\"`\n\n### \u003ca name=\"input_locations_databricks\"\u003e\u003c/a\u003e [locations\\_databricks](#input\\_locations\\_databricks)\n\nDescription: Specifies the list of locations where Databricks workspaces will be deployed.\n\nType: `list(string)`\n\nDefault: `[]`\n\n### \u003ca name=\"input_log_analytics_workspace_id\"\u003e\u003c/a\u003e [log\\_analytics\\_workspace\\_id](#input\\_log\\_analytics\\_workspace\\_id)\n\nDescription: Specifies the log analytics workspace used to configure diagnostics.\n\nType: `string`\n\nDefault: `null`\n\n### \u003ca name=\"input_private_dns_zone_id_blob\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_blob](#input\\_private\\_dns\\_zone\\_id\\_blob)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Storage blob endpoints. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_container_registry\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_container\\_registry](#input\\_private\\_dns\\_zone\\_id\\_container\\_registry)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Container Registry. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_databricks\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_databricks](#input\\_private\\_dns\\_zone\\_id\\_databricks)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Databricks UI endpoints. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_dfs\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_dfs](#input\\_private\\_dns\\_zone\\_id\\_dfs)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Storage dfs endpoints. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_purview_platform\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_purview\\_platform](#input\\_private\\_dns\\_zone\\_id\\_purview\\_platform)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Key Vault. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_queue\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_queue](#input\\_private\\_dns\\_zone\\_id\\_queue)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Storage queue endpoints. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_synapse_portal\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_synapse\\_portal](#input\\_private\\_dns\\_zone\\_id\\_synapse\\_portal)\n\nDescription: Specifies the resource ID of the private DNS zone for Synapse PL Hub. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_private_dns_zone_id_vault\"\u003e\u003c/a\u003e [private\\_dns\\_zone\\_id\\_vault](#input\\_private\\_dns\\_zone\\_id\\_vault)\n\nDescription: Specifies the resource ID of the private DNS zone for Azure Key Vault. Not required if DNS A-records get created via Azure Policy.\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_purview_account_root_collection_admins\"\u003e\u003c/a\u003e [purview\\_account\\_root\\_collection\\_admins](#input\\_purview\\_account\\_root\\_collection\\_admins)\n\nDescription: Specifies the root collection admins of the Purview account.\n\nType:\n\n```hcl\nmap(object({\n    object_id = string\n  }))\n```\n\nDefault: `{}`\n\n### \u003ca name=\"input_purview_enabled\"\u003e\u003c/a\u003e [purview\\_enabled](#input\\_purview\\_enabled)\n\nDescription: Specifies whether Purview should be enabled.\n\nType: `bool`\n\nDefault: `false`\n\n### \u003ca name=\"input_subnet_cidr_ranges\"\u003e\u003c/a\u003e [subnet\\_cidr\\_ranges](#input\\_subnet\\_cidr\\_ranges)\n\nDescription: Specifies the cidr ranges of the subnets used for the Data Management Zone. If not specified, the module will automatically define the right subnet cidr ranges. For this to work, the provided vnet must have no subnets.\n\nType:\n\n```hcl\nobject(\n    {\n      private_endpoint_subnet = optional(string, \"\")\n    }\n  )\n```\n\nDefault: `{}`\n\n### \u003ca name=\"input_tags\"\u003e\u003c/a\u003e [tags](#input\\_tags)\n\nDescription: Specifies the tags that you want to apply to all resources.\n\nType: `map(string)`\n\nDefault: `{}`\n\n### \u003ca name=\"input_zone_redundancy_enabled\"\u003e\u003c/a\u003e [zone\\_redundancy\\_enabled](#input\\_zone\\_redundancy\\_enabled)\n\nDescription: Specifies whether zone-redundancy should be enabled for all resources.\n\nType: `bool`\n\nDefault: `true`\n\n## Outputs\n\nNo outputs.\n\n\u003c!-- markdownlint-enable --\u003e\n## License\n\n[MIT License](/LICENSE)\n\n## Contributing\n\nThis project accepts public contributions. Please use issues, pull requests and the discussins feature in case you have any questions or want to enhance this module.\n\u003c!-- END_TF_DOCS --\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperfectthymetech%2Fterraform-azurerm-data-management-zone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperfectthymetech%2Fterraform-azurerm-data-management-zone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperfectthymetech%2Fterraform-azurerm-data-management-zone/lists"}