{"id":51507921,"url":"https://github.com/gke-labs/tpu-operator","last_synced_at":"2026-07-08T02:30:36.834Z","repository":{"id":362153015,"uuid":"1244920093","full_name":"gke-labs/tpu-operator","owner":"gke-labs","description":" A Kubernetes operator designed to automate the lifecycle of TPU slices within self-managed clusters on Google Compute Engine (GCE).","archived":false,"fork":false,"pushed_at":"2026-07-07T18:59:25.000Z","size":12583,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-07T20:21:45.481Z","etag":null,"topics":["gce","kubernetes","tpu","tpu-slice"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gke-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-20T18:22:01.000Z","updated_at":"2026-07-07T18:59:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/gke-labs/tpu-operator","commit_stats":null,"previous_names":["gke-labs/tpu-operator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gke-labs/tpu-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gke-labs%2Ftpu-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gke-labs%2Ftpu-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gke-labs%2Ftpu-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gke-labs%2Ftpu-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gke-labs","download_url":"https://codeload.github.com/gke-labs/tpu-operator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gke-labs%2Ftpu-operator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35249883,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gce","kubernetes","tpu","tpu-slice"],"created_at":"2026-07-08T02:30:35.994Z","updated_at":"2026-07-08T02:30:36.820Z","avatar_url":"https://github.com/gke-labs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tpu-operator\n\nA Kubernetes operator designed to automate the lifecycle of TPU slices within self-managed clusters on Google Compute Engine (GCE).\n\n## Overview: What the Controller Does for You\n\nWhen running self-managed Kubernetes on Google Compute Engine (GCE), provisioning TPU slices manually requires orchestrating multiple GCE APIs, managing complex metadata for hardware discovery, and bootstrapping nodes.\n\nThe TPUNodeGroup controller acts as a Kubernetes-native infrastructure operator that automates the complexities of provisioning TPU slices. Unlike standard GCE VMs, TPU slices involve multiple VMs interconnected with a specialized high-speed network, forming a single, atomic unit for large scale ML workloads.\n\nFrom your perspective, you simply provide a TPUNodeGroup CR declaring your desired TPU shape. Behind the scenes, the controller will:\n\n1. Abstract GCE Complexity: To properly provision TPUs, the controller automatically creates three critical GCE resources on your behalf:\n   * GCE Workload Policy: Responsible for specifying the physical properties of the underlying infrastructure. For multi-host slices, this policy specifies the acceleratorTopology (e.g., 4x4x4), which tells Google's data centers exactly how to physically wire the TPU chips together using high-speed Inter-Chip Interconnects (ICI).\n   * Instance Template: Defines the VM configuration (machine type, image, disks, metadata, etc.) used by the Managed Instance Group.\n   * Managed Instance Group (MIG): Responsible for actually provisioning and managing the lifecycle of the Virtual Machines. For multi-host slices, users must explicitly configure `targetSizePolicyMode: \"BULK\"` in the TPUNodeGroup CR. The controller then configures the MIG to use \"bulk mode\", which guarantees atomic provisioning. This ensures GCE will only create the VMs if it can successfully allocate the entire slice at once, preventing partial, unusable setups.\n2. Inject TPU Metadata: Automatically inject TPU configuration metadata needed by the TPU Device Plugin (like kube-labels and accelerator\\_topology\\_id) into the VMs via the MIG.\n3. Secure \"Pull-Model\" Bootstrapping: To handle long GCE provisioning times (where standard short-lived kubeadm tokens might expire), the controller implements a **Pull Model**. Instead of injecting tokens at VM creation, the controller waits for the VM state to be `RUNNING`, then injects a dynamic, short-lived join token into Instance Metadata. A startup script on the VM polls for this token, ensuring a secure and reliable cluster join even after extended provisioning delays. (Note: This feature is primarily provided to facilitate rapid prototyping and testing. For production environments, it is expected that users will leverage their existing mechanisms for node bootstrapping, such as custom OS images or configuration management tools).\n4. Deploy the TPU Device Plugin: The controller automatically deploys a streamlined, open-source TPU device plugin to your cluster as a DaemonSet. This plugin acts as the bridge between the raw TPU hardware and the Kubernetes scheduler. It discovers the attached TPU chips, registers them as schedulable resources (e.g., google.com/tpu), and handles local hardware health monitoring.\n5. Handle Graceful Teardown: When the TPUNodeGroup CR is deleted, the controller orchestrates a graceful teardown using a Kubernetes Finalizer. It cordons the nodes, deletes the GCE resources (MIG, Instance Template, and Workload Policy) in reverse order, and removes stale Node objects from the cluster.\n\n*Note: The current release supports v6e and v7x TPU generations.*\n\n---\n\n## Design Philosophy: Reference Bootstrapping\n\nThis project is intended as a **Reference Implementation** for experimental and testing purposes. \n\n**Core Assumption:** We assume that production customers already have mature solutions for their node infrastructure and lifecycle. For those who need a rapid, \"turn-key\" way to experiment with TPUs on self-managed clusters, we provide a **Reference Bootstrapping** flow using standard Kubernetes mechanics.\n\n## Architecture: The Composite Pattern\n\nThis controller is designed using the **Composite Pattern** to maximize modularity and reusability. Instead of a single monolithic state machine, the project is structured as a parent controller that orchestrates three dedicated child Custom Resources:\n\n*   **ResourcePolicy Controller:** Manages GCE Resource Policies (Workload Policies) for ICI networking.\n*   **InstanceTemplate Controller:** Manages GCE Instance Templates.\n*   **ManagedInstanceGroup Controller:** Manages GCE MIGs, with specific support for bulk allocation.\n\nAll infrastructure resources are managed asynchronously via official **Google Cloud Go Client Libraries**. This architecture allows the `TPUNodeGroup` controller to focus solely on high-level workload orchestration and node lifecycle, leaving the low-level GCE \"3C\" (Compute) resource management to specialized, decoupled sub-controllers.\n\n![TPUNodeGroup Architecture](docs/images/tpu_node_group_architecture.svg)\n\n### The Join-to-Workload Lifecycle (Pull-Model Bootstrapping)\n\nWhen `bootstrapKubernetes` is enabled, the controller orchestrates a secure, asynchronous **Pull-Model Bootstrapping** flow. This design safely coordinates dynamic token generation and join operations over potentially extended GCE VM booting delays.\n\nThe configuration sequence proceeds as follows:\n\n1. **GCE VM Initialization (VM Starts Booting):** The Managed Instance Group (MIG) provisions the VM.\n2. **GCE VM (Polling Begins):** The VM's startup script immediately begins polling its own GCE Instance Metadata for the `kubeadm-join-token`.\n3. **Controller (RUNNING Detection):** In parallel, the controller detects that the GCE VM state has transitioned to `RUNNING`.\n4. **Controller (Token Generation):** The controller generates a unique, short-lived `kubeadm` join token (valid for 1 hour) and creates a corresponding **Bootstrap Secret** in the `kube-system` namespace.\n5. **Controller (Metadata Injection):** The controller fetches the cluster's **CA Cert Hash** and injects it along with the **Join Token** and **Control Plane IP** directly into the GCE Instance Metadata for that specific VM.\n6. **GCE VM (Poll Succeeds):** The VM's polling loop succeeds and retrieves the injected configuration values.\n7. **GCE VM (Pull-Model Join):** The VM executes `kubeadm join` using the retrieved credentials to securely attach itself to the cluster.\n8. **Kubernetes (Node Registered):** The K8s API Server registers the node and marks it as `Ready`.\n9. **Controller (Node Matching \u0026 Labeling):** Once the new worker node appears in the cluster, the controller matches it to the GCE instance using the node's **`ProviderID`** and automatically **patches the Node with TPU labels** (topology, accelerator type, chip count).\n\nThe following sequence diagram visualizes this chronological pull-model orchestration:\n\n![Pull-Model Bootstrapping Flow](docs/images/bootstrapping_flow.svg)\n\n---\n\n## Prerequisites\n\n### Configure GCP IAM Permissions\n\nThe TPUNodeGroup controller requires permissions to create and manage GCE resources (Instance Templates, Workload Policies, and MIGs) on your behalf.\n\nYou can assign the roles/compute.admin role, or for least-privilege environments, ensure the service account has the following specific permissions:\n\n* compute.instanceTemplates.\\*\n* compute.resourcePolicies.\\*\n* compute.instanceGroupManagers.\\*\n* compute.instanceGroups.\\*\n* compute.instances.\\*\n* compute.subnetworks.get\n\n### Set Up Kubernetes Control Plane (GCE VM)\n\nIf you are setting up a new self-managed cluster, you can initialize a control plane on a GCE VM using `kubeadm`.\n\n1. **Create a GCE VM** (if you don't have one):\n\n```shell\nexport ZONE=us-central1-c\nexport SERVICE_ACCOUNT_EMAIL=your-service-account@project.iam.gserviceaccount.com\n\ngcloud compute instances create k8s-control-plane \\\n    --zone=$ZONE \\\n    --machine-type=e2-standard-4 \\\n    --image-family=ubuntu-2404-lts-amd64 \\\n    --image-project=ubuntu-os-cloud \\\n    --tags=k8s-control-plane \\\n    --service-account=$SERVICE_ACCOUNT_EMAIL \\\n    --scopes=https://www.googleapis.com/auth/cloud-platform\n```\n\n2. **Initialize the cluster**: SSH into the VM and run the following commands.\n   Ensure `conntrack`, `containerd`, `kubeadm` and a CNI plugin are installed.\n\n```shell\n# Initialize the control plane with a specific pod CIDR (Flannel requires 10.244.0.0/16)\nsudo kubeadm init --pod-network-cidr=10.244.0.0/16\n```\n\nNote: kubeadm is required if enabling `bootstrapKubernetes` in the TPUNodeGroup CR.\n\n### Configure Your Subnet\n\nEnsure your subnet has internet access (e.g., via Cloud NAT) so nodes can download packages during bootstrapping.\n\n### Configure VPC Firewall Rules\n\nAllow API Server access (TCP 6443\\) from the TPU nodes and allow internal traffic between the Control Plane and the TPU VMs. Adjust the source-ranges to match your VPC CIDR:\n\n```shell\nexport VPC_NETWORK=default\nexport CONTROL_PLANE_CIDR=0.0.0.0/0 # Restrict this in production\nexport CLUSTER_CIDR=10.128.0.0/9 # Adjust to your VPC CIDR\n\n# Allow API Server access (6443)\ngcloud compute firewall-rules create allow-k8s-apiserver \\\n    --allow tcp:6443 \\\n    --network=$VPC_NETWORK \\\n    --source-ranges=$CONTROL_PLANE_CIDR \\\n    --target-tags=k8s-control-plane\n\n# Allow all internal traffic between nodes (Control Plane \u003c-\u003e TPU VM)\ngcloud compute firewall-rules create allow-k8s-internal \\\n    --allow tcp,udp,icmp,ipip \\\n    --network=$VPC_NETWORK \\\n    --source-ranges=$CLUSTER_CIDR \\\n    --target-tags=k8s-control-plane\n```\n\n### Create Your TPU Reservation (Optional)\n\nIf you are using a reservation-bound provisioning model, you must reserve the TPU capacity in your target zone.\n\n```shell\nexport RESERVATION_NAME=my-tpu-reservation\nexport VM_FAMILY=tpu7x # or tpu6e\nexport ACCELERATOR_COUNT=4\nexport HOST_COUNT=2\nexport ZONE=us-central1-c\n\ngcloud compute reservations create $RESERVATION_NAME \\\n    --vm-family=$VM_FAMILY \\\n    --accelerator-count=$ACCELERATOR_COUNT \\\n    --host-count=$HOST_COUNT \\\n    --zone=$ZONE\n```\n\n##\n\n## Step 1: Build and Deploy the Controller\n\nBefore creating the TPUNodeGroup resource, you need to build the controller image, push it to your container registry, and deploy it using the provided manifests.\n\n1. Build and Push the Image:\n\n```shell\nexport PROJECT_ID=your-project-id\n\ndocker build -t gcr.io/$PROJECT_ID/tpunodegroup-controller:latest .\ndocker push gcr.io/$PROJECT_ID/tpunodegroup-controller:latest\n```\n\n2. Update the Deployment Manifest: Open `deploy/kustomization.yaml` and update the `newName` and `newTag` under the `images` section to match the image you just pushed.\n3. Install Components: Apply the kustomization to deploy all components (CRDs, Controller, Device Plugin, etc.).\n\n```shell\nkubectl apply -k deploy/\n```\n\n4. Verify the Controller is Running:\n\n```shell\nkubectl get pods -n tpu-node-group\n```\n\n*Note: If you are running a single-node cluster (control plane only), you may need to untaint the node to allow the controller to run: `kubectl taint nodes --all node-role.kubernetes.io/control-plane-`*\n\n## Step 2: Create the TPUNodeGroup CR\n\nOnce the controller is running, create a TPUNodeGroup resource in your cluster. This tells the controller what capacity and configuration to instantiate.\n\nThe controller manages the creation of the Instance Template dynamically based on the `instanceConfig` fields. The `bootstrapKubernetes` block instructs the controller to automatically bootstrap the Kubernetes nodes and join them to the control plane.\n\nSet the required environment variables:\n\n```shell\nexport PROJECT=my-gcp-project\nexport ZONE=us-central1-c\nexport RESERVATION_NAME=my-tpu-reservation\nexport IMAGE=projects/ubuntu-os-accelerator-images/global/images/ubuntu-accel-2404-amd64-tpu-tpu7x-v20260320\nexport TOPOLOGY=2x2x2\nexport NODE_COUNT=2\nexport CONTROL_PLANE_IP=10.128.0.2\nexport TPU_NODE_GROUP_NAME=my-tpu-node-group\nexport NAMESPACE=default\nexport ACCELERATOR=tpu7x\nexport MACHINE_TYPE=tpu7x-standard-4t\n```\n\nExample CR creation using these variables:\n\n```shell\ncat \u003c\u003cEOF | kubectl apply -f -\napiVersion: tpu.google.com/v1alpha1\nkind: TPUNodeGroup\nmetadata:\n  name: $TPU_NODE_GROUP_NAME\n  namespace: $NAMESPACE\nspec:\n  project: $PROJECT\n  nodeLocation: $ZONE\n  nodeCount: $NODE_COUNT\n  acceleratorConnectionMode: \"STATIC\"\n  topology: $TOPOLOGY\n  targetSizePolicyMode: \"BULK\" # Required in latest versions\n  instanceConfig:\n    machineType: $MACHINE_TYPE\n    provisioningModel: \"RESERVATION_BOUND\"\n    reservation: $RESERVATION_NAME\n    image: $IMAGE\n    subnetwork: projects/$PROJECT/regions/us-central1/subnetworks/default # Required if network is in custom mode\n    bootDiskSizeGB: 100\n  bootstrapKubernetes:\n    version: \"1.31\"\n    controlPlaneIP: $CONTROL_PLANE_IP\nEOF\n```\n\n### Bring Your Own Instance Template (Optional)\n\nIf you want to further customize the VM, you can bring your own pre-provisioned GCE Instance Template and bypass the controller's template creation process. You can provide its URI using the `instanceTemplateURI` field. When this field is set, the controller will skip generating an Instance Template and use yours directly to create the Managed Instance Group. The controller will perform basic validation.\n\n*Note: Because the controller skips the bootstrapping phase when using a custom Instance Template, you must configure your own bootstrapping solution. When doing so, you must ensure that your K8s Node objects are bootstrapped with a `spec.providerID` that matches the standard GCE instance format (`gce://\u003cproject\u003e/\u003czone\u003e/\u003cinstance-name\u003e`). The controller relies on this `providerID` to correctly identify, match, and label the nodes as they join the cluster. Once matched, the controller will automatically perform node labeling and deploy the TPU device plugin DaemonSet.*\n\n## Step 3: Monitor Provisioning Status\n\nThe controller provides real-time readiness feedback in the CRD status. You don't need to check the GCP console; simply inspect the resource in Kubernetes to see its exact lifecycle state.\n\n```shell\nkubectl get tpunodegroup $TPU_NODE_GROUP_NAME -o yaml\n```\n\nYou can view the detailed conditions array to see the health and provisioning phase of your TPU capacity. The conditions dynamically adapt based on your configuration:\n\n* Ready: Set to True only when 100% of the nodes in the slice are healthy, registered, and their ICI links are established.\n\n## Step 4: Schedule Your Workloads\n\nIn your workload manifest, you can add Kubernetes node selectors to ensure that your TPU workload is scheduled on the correct TPU machine type and topology. The controller automatically injects these labels into the nodes.\n\n*Note: For JAX workloads, please ensure you use JAX 0.10.0 or newer to avoid the need for manual worker network endpoint metadata configuration.*\n\nExample deployment using node selectors:\n\n```shell\ncat \u003c\u003cEOF | kubectl apply -f -\napiVersion: v1\nkind: Service\nmetadata:\n  name: tpu-svc-v7x\n  namespace: $NAMESPACE\nspec:\n  clusterIP: None\n  publishNotReadyAddresses: true\n  selector:\n    app: tpu-worker-v7x\n  ports:\n  - name: coordinator\n    port: 1234\n  - name: runtime-1\n    port: 8470\n  - name: runtime-2\n    port: 8471\n---\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: tpu-worker-v7x\n  namespace: $NAMESPACE\nspec:\n  serviceName: \"tpu-svc-v7x\"\n  replicas: 2\n  podManagementPolicy: Parallel\n  selector:\n    matchLabels:\n      app: tpu-worker-v7x\n  template:\n    metadata:\n      labels:\n        app: tpu-worker-v7x\n    spec:\n      hostNetwork: true\n      dnsPolicy: ClusterFirstWithHostNet\n      nodeSelector:\n        cloud.google.com/tpu-node-group: $NAMESPACE-$TPU_NODE_GROUP_NAME\n        cloud.google.com/gke-tpu-topology: $TOPOLOGY\n        cloud.google.com/gke-tpu-accelerator: $ACCELERATOR\n      tolerations:\n      - key: \"google.com/tpu\"\n        operator: \"Exists\"\n        effect: \"NoSchedule\"\n      containers:\n      - name: tpu-job\n        image: us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:latest\n        securityContext:\n          privileged: true\n        command:\n        - python3\n        - -c\n        - |\n          import jax, os, time, socket\n\n          def wait_for_dns(hostname):\n              for _ in range(30):\n                  try:\n                      socket.gethostbyname(hostname)\n                      return True\n                  except socket.gaierror:\n                      time.sleep(5)\n              return False\n\n          svc_name = \"tpu-svc-v7x\"\n          pod_base_name = \"tpu-worker-v7x\"\n          namespace = \"default\"\n\n          dns0 = f\"{pod_base_name}-0.{svc_name}.{namespace}.svc.cluster.local\"\n          dns1 = f\"{pod_base_name}-1.{svc_name}.{namespace}.svc.cluster.local\"\n\n          if not wait_for_dns(dns0) or not wait_for_dns(dns1):\n              print(\"Failed to resolve DNS for TPU workers\")\n              exit(1)\n\n          os.environ.update({\n              \"JAX_COORDINATOR_ADDRESS\": f\"{dns0}:1234\",\n              \"MEGASCALE_COORDINATOR_ADDRESS\": f\"{dns0}:8471\",\n              \"TPU_PROCESS_ADDRESSES\": f\"{dns0}:8470,{dns1}:8470\",\n              \"TPU_WORKER_HOSTNAMES\": f\"{dns0}:8471,{dns1}:8471\"\n          })\n\n          p_id = int(os.environ[\"K8S_POD_NAME\"].split(\"-\")[-1])\n\n          print(f\"Initializing JAX distributed: coordinator={dns0}:1234, total_processes=2, process_id={p_id}\")\n          jax.distributed.initialize(coordinator_address=f\"{dns0}:1234\", num_processes=2, process_id=p_id)\n\n          print(\"Devices:\", jax.devices())\n          assert len(jax.devices()) == 8, f\"Expected 8 devices, got {len(jax.devices())}\"\n          print(\"TPU OK\")\n\n        env:\n        - name: JAX_PLATFORMS\n          value: \"tpu\"\n        - name: JAX_PROCESS_COUNT\n          value: \"2\"\n        - name: TPU_TOPOLOGY\n          value: \"$TOPOLOGY\"\n        - name: TPU_ACCELERATOR_TYPE\n          value: \"tpu7x-4\"\n        - name: K8S_POD_NAME\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.name\n        - name: NODE_IP\n          valueFrom:\n            fieldRef:\n              fieldPath: status.hostIP\n        - name: TPU_CHIPS_PER_HOST_BOUNDS\n          value: \"2,2,1\"\n        - name: TPU_HOST_BOUNDS\n          value: \"1,1,2\"\n        - name: VBAR_CONTROL_SERVICE_URL\n          value: \"127.0.0.1:8353\"\n        - name: TPU_PROCESS_PORT\n          value: \"8470\"\n        - name: TPU_RUNTIME_METRICS_PORTS\n          value: \"8431,8432,8433,8434\"\n        resources:\n          limits:\n            google.com/tpu: \"4\"\n          requests:\n            google.com/tpu: \"4\"\nEOF\n```\n\n## Step 5: Teardown\n\nAfter the job is done, you can clean up the resources using\n\n```shell\nkubectl delete tpunodegroup $TPU_NODE_GROUP_NAME\n```\n\nThe controller orchestrates a graceful teardown using a Kubernetes Finalizer. It executes the teardown in three strict phases:\n\n1. Cordon: Marks all nodes in the slice as Unschedulable to prevent the scheduler from placing new Pods on them.\n2. Infrastructure Removal: Deletes GCE resources (MIGs, Instance Templates and Workload Policies) in reverse creation order.\n3. Kubernetes Cleanup: Deletes any remaining stale Node objects from the cluster and finally removes the finalizer, allowing the CR to be fully removed from Kubernetes.\n\n---\n\n## Using a GCP Service Account for the Controller\n\nThe TPUNodeGroup controller needs GCP permissions to create and manage GCE resources. You can provide these credentials in two ways:\n\n### Option 1: VM Attached Service Account\n\nIf your Kubernetes control plane or operator is running on a GCE VM, the controller transparently uses the VM's attached service account via **Application Default Credentials (ADC)**. This is the recommended approach for development within GCE, leveraging the existing node identity and predefined IAM scopes for GCE worker nodes.\n\n1. Ensure the GCE VM was created with the required IAM permissions (or `roles/compute.admin`).\n2. Ensure the VM has the `https://www.googleapis.com/auth/cloud-platform` scope enabled.\n\n### Option 2: Service Account Key File\n\nIf you are running the controller outside of GCE or want to use a specific service account, you can provide a key file via a Kubernetes Secret.\n\n1. **Create a Service Account** in GCP and grant it the required permissions (see Prerequisites).\n2. **Generate a JSON key** for the service account.\n3. **Create a Kubernetes Secret** in the `tpu-node-group` namespace:\n\n```shell\nexport KEY_PATH=/path/to/your/service-account-key.json\n\nkubectl create secret generic tpu-node-group-credentials \\\n    --from-file=key.json=$KEY_PATH \\\n    --namespace tpu-node-group\n```\n\n4. **Update `deploy/controller/deployment.yaml`**: Uncomment the `env` section for `GOOGLE_APPLICATION_CREDENTIALS` in the controller deployment:\n\n```yaml\n        env:\n        - name: GOOGLE_APPLICATION_CREDENTIALS\n          value: /etc/gcp/key.json\n```\n\n\n\n## Contributing\n\nThis project is licensed under the [Apache 2.0 License](LICENSE).\n\nWe welcome contributions! Please see [docs/contributing.md](docs/contributing.md) for more information.\n\nWe follow [Google's Open Source Community Guidelines](https://opensource.google.com/conduct/).\n\n## Disclaimer\n\nThis is not an officially supported Google product.\n\nThis project is not eligible for the Google Open Source Software Vulnerability Rewards Program.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgke-labs%2Ftpu-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgke-labs%2Ftpu-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgke-labs%2Ftpu-operator/lists"}