{"id":29202576,"url":"https://github.com/ai-hypercomputer/cloud-tpu-monitoring-debugging","last_synced_at":"2025-07-02T13:32:44.522Z","repository":{"id":156802387,"uuid":"629630409","full_name":"AI-Hypercomputer/cloud-tpu-monitoring-debugging","owner":"AI-Hypercomputer","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-18T21:24:14.000Z","size":77,"stargazers_count":12,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-09T05:51:25.050Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AI-Hypercomputer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-18T17:47:46.000Z","updated_at":"2025-05-13T16:24:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"3f7f7753-fe33-4558-9057-9c6c44f64ce0","html_url":"https://github.com/AI-Hypercomputer/cloud-tpu-monitoring-debugging","commit_stats":null,"previous_names":["google/cloud-tpu-monitoring-debugging"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/AI-Hypercomputer/cloud-tpu-monitoring-debugging","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fcloud-tpu-monitoring-debugging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fcloud-tpu-monitoring-debugging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fcloud-tpu-monitoring-debugging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fcloud-tpu-monitoring-debugging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AI-Hypercomputer","download_url":"https://codeload.github.com/AI-Hypercomputer/cloud-tpu-monitoring-debugging/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AI-Hypercomputer%2Fcloud-tpu-monitoring-debugging/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263148129,"owners_count":23421117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-02T13:30:32.341Z","updated_at":"2025-07-02T13:32:44.510Z","avatar_url":"https://github.com/AI-Hypercomputer.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\n Copyright 2023 Google LLC\n \n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n \n      https://www.apache.org/licenses/LICENSE-2.0\n \n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n --\u003e\n# Cloud TPU Monitoring Debugging\n\n## Overview\n\nCloud TPU Monitoring Debugging repository contains all the infrastructure and logic required to monitor and debug jobs running on Cloud TPU.\n\nTerraform is used to deploy resources in google cloud project.\nTerraform is an open-source tool to set up and manage google cloud\ninfrastructure based on configuration files. This repository will help the\ncustomers to deploy various google cloud resources via script, without any\nmanual effort.\n\n[cloud-tpu-diagnostics PyPI package](https://pypi.org/project/cloud-tpu-diagnostics) contains all the logic to monitor, debug and profile the jobs running on Cloud TPU.\n\n## Getting Started with Terraform\n\n-   Follow [this link](https://developer.hashicorp.com/terraform/tutorials/gcp-get-started/install-cli) to install Terraform on desktop.\n-   Run `terraform init` to\n    initialize google cloud Terraform provider version. This command will add\n    the necessary plugins and build the `.terraform` directory.\n-   If there is an update to terraform google cloud provider version, run\n    `terraform init --upgrade` for the update to take place.\n-   You can also run `terraform plan` to validate resource declarations,\n    identify any syntax errors, version mismatch before deploying the resources.\n\n### Configure Terraform to store state in Cloud Storage\n\nBy default, Terraform stores [state](https://www.terraform.io/docs/state/) locally in a file named `terraform.tfstate`. This default configuration can make Terraform usage difficult for teams, especially when many users run Terraform at the same time and each machine has its own understanding of the current infrastructure. To help avoid such issues, this section configures a remote state that points to Google Cloud Storage (GCS) bucket.\n\n1. In Cloud Shell, create the GCS bucket:\n\n        gsutil mb gs://${GCS_BUCKET_NAME}\n\n2. Enable [Object Versioning](https://cloud.google.com/storage/docs/object-versioning) to keep the history of your deployments. Enabling Object Versioning increases [storage costs](https://cloud.google.com/storage/pricing), which you can mitigate by configuring\n[Object Lifecycle Management](https://cloud.google.com/storage/docs/lifecycle) to delete old state versions.\n\n        gsutil versioning set on gs://${GCS_BUCKET_NAME}\n\n3. Enter the name of GCS bucket created above when you run `terraform init` to initialize Terraform.\n\n        Initializing the backend...\n        bucket\n          The name of the Google Cloud Storage bucket\n\n          Enter a value: \u003cGCS_BUCKET_NAME\u003e\n\n## Deploy GCP Resources\nThere are following resources managed in this directory:\n\n1. **Monitoring Dashboard**: This is an outlier dashboard that displays statistics and outlier mode for TPU metrics.\n2. **Debugging Dashboard**: This dashboard displays the stack traces collected in Cloud Logging for the process running on TPU VMs.\n3. **Logging Storage**: This is an user-defined log bucket to store stack traces. Creating a new log storage is completely optional. If you choose not to create a separate log bucket, the stack traces will be collected in [_Default log bucket](https://cloud.google.com/logging/docs/routing/overview#default-bucket).\n\n### Deploy Resources for Workloads on GCE\n\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gce` directory to deploy all the resources mentioned above for TPU workloads running on GCE. You will be prompted to provide values for some input variables. After confirming the action, all the resources will get automatically deployed in your gcp project.\n\n### Deploy Resources for Workloads on GKE\n\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gke` directory to deploy all the resources mentioned above for TPU workloads running on GKE. You will be prompted to provide values for some input variables. After confirming the action, all the resources will get automatically deployed in your gcp project.\n\n\u003e **_NOTE:_** Please check the below guide for more details about GCE/GKE specific resources and prerequisites.\n\nFollow the below guide to deploy the resources individually:\n### Monitoring Dashboard\n#### GCE\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gce/resources/dashboard/monitoring_dashboard/` to deploy only monitoring dashboard for GCE in your gcp project.\n\nIf the `node_prefix` parameter is not specified in the input variable `var.monitoring_dashboard_config` or is set to an empty string, the metrics on the dashboard will plot the data points for all TPU VMs in your GCP project.\n\nFor instance, if you provide `{\"node_prefix\": \"test\"}` as the input value for the input variable `var.monitoring_dashboard_config`, then the metrics on the monitoring dashboard will only show the data points for the TPU VMs with node names that start with `test`. Refer to this [doc](https://cloud.google.com/sdk/gcloud/reference/alpha/compute/tpus/queued-resources/create#--node-prefix) for more information on node prefix for TPUs in multislice.\n\n#### GKE\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gke/resources/dashboard/monitoring_dashboard/` to deploy only monitoring dashboard for GKE in your gcp project.\n\n### Debugging Dashboard\n#### GCE\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gce/resources/dashboard/logging_dashboard/` to deploy only debugging dashboard for GCE in your gcp project.\n\n#### GKE\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gke/resources/dashboard/logging_dashboard/` to deploy only debugging dashboard for GKE in your gcp project.\n\nUsers need to add a sidecar container to their TPU workload running on GKE to view traces in the debugging dashboard. The sidecar container must be named in a specific way, matching the regex `[a-z-0-9]*stacktrace[a-z-0-9]*`. Here is an example of the sidecar container that should be added:\n\n```\ncontainers:\n- name: stacktrace-log-collector\n  image: busybox:1.28\n  resources:\n    limits:\n      cpu: 100m\n      memory: 200Mi\n  args: [/bin/sh, -c, \"while [ ! -d /tmp/debugging ]; do sleep 60; done; while [ ! -e /tmp/debugging/* ]; do sleep 60; done; tail -n+1 -f /tmp/debugging/*\"]\n  volumeMounts:\n  - name: tpu-debug-logs\n    readOnly: true\n    mountPath: /tmp/debugging\n- name: \u003cmain_container\u003e\n.....\n.....\nvolumes:\n- name: tpu-debug-logs\n```\n\n### Log Storage\n#### GCE\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gce/resources/log_storage/` to deploy a separate log bucket to store stack traces for GCE. You will be prompted to provide name of your gcp project and also the bucket configuration. You can also set the retention period for the bucket.\n\n#### GKE\nRun `terraform init \u0026\u0026 terraform apply` inside `gcp_resources/gke/resources/log_storage/` to deploy a separate log bucket to store stack traces for GKE. You will be prompted to provide name of your gcp project and also the bucket configuration. You can also set the retention period for the bucket. Make sure that you have the sidecar container running in your GKE cluster as mentioned in [Debugging Dashboard section for GKE](#debugging-dashboard).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-hypercomputer%2Fcloud-tpu-monitoring-debugging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai-hypercomputer%2Fcloud-tpu-monitoring-debugging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-hypercomputer%2Fcloud-tpu-monitoring-debugging/lists"}