{"id":13515058,"url":"https://github.com/microsoft/azure-synapse-spark-metrics","last_synced_at":"2025-03-31T04:36:20.980Z","repository":{"id":46125591,"uuid":"290124193","full_name":"microsoft/azure-synapse-spark-metrics","owner":"microsoft","description":"Azure Synapse Spark Metrics provides easy metrics monitoring functions for Synapse services, especially, Apache Spark pool instances, by leveraging Prometheus, Grafana and  Azure APIs.","archived":false,"fork":false,"pushed_at":"2023-05-23T00:17:45.000Z","size":1081,"stargazers_count":22,"open_issues_count":7,"forks_count":13,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T21:37:42.914Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-25T05:42:13.000Z","updated_at":"2024-12-30T19:02:41.000Z","dependencies_parsed_at":"2024-11-01T19:30:35.687Z","dependency_job_id":"6086eecf-09c5-410e-9e6d-39b93cb679ac","html_url":"https://github.com/microsoft/azure-synapse-spark-metrics","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fazure-synapse-spark-metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fazure-synapse-spark-metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fazure-synapse-spark-metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fazure-synapse-spark-metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/azure-synapse-spark-metrics/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246418658,"owners_count":20773934,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T05:01:05.786Z","updated_at":"2025-03-31T04:36:20.191Z","avatar_url":"https://github.com/microsoft.png","language":"Python","funding_links":[],"categories":["Repositories"],"sub_categories":["Tools \u0026 Samples"],"readme":"# Azure Synapse Spark Metrics\n\n## Introduction\n\nThis project mainly aims to provide:\n - **Azure Synapse Apache Spark metrics** monitoring for Azure Synapse Spark applications by leveraging Prometheus, Grafana and Azure APIs.\n - **Azure Synapse Prometheus connector** for connecting the on-premises Prometheus server to Azure Synapse Analytics workspace metrics API. \n - **Grafana dashboards** for synapse spark metrics visualization.\n - **Helm chart** for Prometheus and Grafana deployment on AKS, including the connector, Prometheus servers and Grafana dashboards for metrics users.\n\nThe dataflow:\n\n![Dataflow Chart](docs/image/dataflow.png)\n\nGrafana dashboard screenshot:\n\n![Grafana dashboard](docs/image/screenshot-dashboard-application.png)\n\n## Prerequisites\n\n1. [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)\n2. [Helm 3.30+](https://github.com/helm/helm/releases)\n3. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n\nOr just use the out-of-box [Azure Cloud Shell](https://shell.azure.com/), which includes all above tools.\n\n## Getting Started\n\n1. Create a Azure Kubernetes (1.16+, or use Minikube instead)\n\n    ```bash\n    az login\n    az account set --subscription \"\u003csubscription_id\u003e\"\n    az aks create --name \u003ckubernetes_cluster_name\u003e --resource-group \u003ckubernetes_cluster_rg\u003e --location eastus --node-vm-size Standard_D2s_v3\n    az aks get-credentials --name \u003ckubernetes_cluster_name\u003e --resource-group \u003ckubernetes_cluster_rg\u003e\n    ```\n\n2. Create a service principal and grant permission to synapse workspace\n\n    ```bash\n    az ad sp create-for-rbac --name \u003cservice_principal_name\u003e\n    ```\n\n    The result should look like:\n\n    ```json\n    {\n        \"appId\": \"abcdef...\",\n        \"displayName\": \"\u003cservice_principal_name\u003e\",\n        \"name\": \"http://\u003cservice_principal_name\u003e\",\n        \"password\": \"abc....\",\n        \"tenant\": \"\u003ctenant_id\u003e\"\n    }\n    ```\n\n    Note down the appId, password, and tenant id.\n\n    1. Login to your [Azure Synapse Analytics workspace](https://web.azuresynapse.net/) as Synapse Administrator\n    2. In Synapse Studio, on the left-side pane, select **Manage** \u003e **Access control**\n    3. Click the **Add** button on the upper left to add a role assignment\n    4. For **Scope** choose **Workspace**\n    5. For **Role** choose **Synapse Compute Operator**\n    6. For **Select user** input your \u003cservice_principal_name\u003e and click your service principal\n    7. Click **Apply**\n\n    Wait 3 minutes for permission to take effect.\n\n    ![screenshot-grant-permission-srbac](docs/image/screenshot-grant-permission-srbac.png)\n\n    \u003e Note: Make sure your service principal has at least a \"Reader\" role in your Synapse workspace. Go to **Access Control (IAM)** tab of the Azure portal and check the permission settings.\n\n3. Install Synapse Prometheus Operator\n\n    Add synapse-prometheus-operator repo to Helm client\n\n    ```bash\n    helm repo add synapse-charts https://github.com/microsoft/azure-synapse-spark-metrics/releases/download/helm-chart\n    ```\n\n    Install by Helm client:\n\n    ```bash\n    helm install spo synapse-charts/synapse-prometheus-operator --create-namespace --namespace spo \\\n        --set synapse.workspaces[0].workspace_name=\"\u003cworkspace_name\u003e\" \\\n        --set synapse.workspaces[0].tenant_id=\"\u003ctenant_id\u003e\" \\\n        --set synapse.workspaces[0].service_principal_name=\"\u003cservice_principal_app_id\u003e\" \\\n        --set synapse.workspaces[0].service_principal_password=\"\u003cservice_principal_password\u003e\" \\\n        --set synapse.workspaces[0].subscription_id=\"\u003csubscription_id\u003e\" \\\n        --set synapse.workspaces[0].resource_group=\"\u003cworkspace_resource_group_name\u003e\"\n    ```\n\n     - workspace_name: Synapse workspace name.\n     - subscription_id: Synapse workspace subscription id.\n     - workspace_resource_group_name:  Synapse workspace resource group name.\n     - tenant_id: Synapse workspace tenant id.\n     - service_principal_name: The service principal name (or known as \"appId\")\n     - service_principal_password: The service principal password you just created.\n\n    For more details, please refer to [config.example.yaml](https://github.com/microsoft/azure-synapse-spark-metrics/blob/main/synapse-prometheus-connector/src/config/config.example.yaml)\n\n4. Open Grafana and enjoy!\n\n    ```bash\n    # Get password\n    kubectl get secret --namespace spo spo-grafana -o jsonpath=\"{.data.admin-password}\" | base64 --decode ; echo\n    # Get service ip, copy \u0026 paste the external ip to browser, and login with username 'admin' and the password.\n    kubectl -n spo get svc spo-grafana\n    ```\n\n    Find Synapse Dashboard on the upper left corner of the Grafana page (Home -\u003e Synapse Workspace / Synapse Application),\n    try to run a example code in Synapse Studio notebook and wait a few seconds for the metrics pulling.\n\n## Uninstall\n\nRemove the operators.\n\n```bash\n# helm delete \u003crelease\u003e -n \u003cnamespace\u003e\nhelm delete spo -n spo\n```\n\nRemove the Kubernetes cluster.\n\n```bash\naz aks delete --name \u003ckubernetes_cluster_name\u003e --resource-group \u003ckubernetes_cluster_rg\u003e\n```\n\n## Install Helm Chart Locally\n\n```\nhelm install spo ./synapse-prometheus-operator --create-namespace --namespace spo \\\n    --set synapse.workspaces[0].workspace_name=\"\u003cworkspace_name\u003e\" \\\n    --set synapse.workspaces[0].tenant_id=\"\u003ctenant_id\u003e\" \\\n    --set synapse.workspaces[0].service_principal_name=\"\u003cservice_principal_app_id\u003e\" \\\n    --set synapse.workspaces[0].service_principal_password=\"\u003cservice_principal_password\u003e\" \\\n    --set synapse.workspaces[0].subscription_id=\"\u003csubscription_id\u003e\" \\\n    --set synapse.workspaces[0].resource_group=\"\u003cworkspace_resource_group_name\u003e\"\n```\n\n## Build Docker Image\n\n```bash\ncd synapse-prometheus-connector\ndocker build -t \"synapse-prometheus-connector:${Version}\" -f Dockerfile .\n```\n\n## Contributing\n\nThis project welcomes contributions and suggestions.  Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fazure-synapse-spark-metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fazure-synapse-spark-metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fazure-synapse-spark-metrics/lists"}