{"id":15213134,"url":"https://github.com/grpc/psm-interop","last_synced_at":"2025-07-31T02:07:46.774Z","repository":{"id":211965100,"uuid":"715833859","full_name":"grpc/psm-interop","owner":"grpc","description":"Proxyless Security Mesh End-to-End Tests","archived":false,"fork":false,"pushed_at":"2025-01-28T17:04:05.000Z","size":943,"stargazers_count":5,"open_issues_count":8,"forks_count":25,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-01-28T18:22:26.687Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grpc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-07T23:30:53.000Z","updated_at":"2025-01-28T17:04:12.000Z","dependencies_parsed_at":"2023-12-11T21:23:59.959Z","dependency_job_id":"b46570d0-5b68-49d6-84c3-411f354d810e","html_url":"https://github.com/grpc/psm-interop","commit_stats":{"total_commits":376,"total_committers":34,"mean_commits":"11.058823529411764","dds":0.4601063829787234,"last_synced_commit":"a752d2b832a4055e0a4a297fff5a748f2b53edc8"},"previous_names":["grpc/psm-interop"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grpc%2Fpsm-interop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grpc%2Fpsm-interop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grpc%2Fpsm-interop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grpc%2Fpsm-interop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grpc","download_url":"https://codeload.github.com/grpc/psm-interop/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237116994,"owners_count":19258341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-28T09:22:14.515Z","updated_at":"2025-02-04T12:32:02.359Z","avatar_url":"https://github.com/grpc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# xDS Kubernetes Interop Tests\n\nProxyless Security Mesh Interop Tests executed on Kubernetes.\n\n### Experimental\nWork in progress. Internal APIs may and will change. Please refrain from making\nchanges to this codebase at the moment.\n\n### Stabilization roadmap \n- [x] Replace retrying with tenacity\n- [x] Generate namespace for each test to prevent resource name conflicts and\n      allow running tests in parallel\n- [x] Security: run server and client in separate namespaces\n- [ ] Make framework.infrastructure.gcp resources [first-class\n      citizen](https://en.wikipedia.org/wiki/First-class_citizen), support\n      simpler CRUD\n- [x] Security: manage `roles/iam.workloadIdentityUser` role grant lifecycle for\n      dynamically-named namespaces \n- [x] Restructure `framework.test_app` and `framework.xds_k8s*` into a module\n      containing xDS-interop-specific logic\n- [ ] Address inline TODOs in code\n- [x] Improve README.md documentation, explain helpers in bin/ folder\n\n## Installation\n\n#### Requirements\n1. Python v3.10+\n2. [Google Cloud SDK](https://cloud.google.com/sdk/docs/install)\n3. `kubectl`\n\n`kubectl` can be installed via `gcloud components install kubectl`, or system package manager: https://kubernetes.io/docs/tasks/tools/#kubectl\n\nPython3 venv tool may need to be installed from APT on some Ubuntu systems:\n```shell\nsudo apt-get install python3-venv\n```\n\n##### Getting Started\n\n1. If you haven't, [initialize](https://cloud.google.com/sdk/docs/install-sdk) gcloud SDK\n2. Activate gcloud [configuration](https://cloud.google.com/sdk/docs/configurations) with your project\n3. Enable gcloud services:\n   ```shell\n   gcloud services enable \\\n     artifactregistry.googleapis.com \\\n     compute.googleapis.com \\\n     container.googleapis.com \\\n     logging.googleapis.com \\\n     monitoring.googleapis.com \\\n     networksecurity.googleapis.com \\\n     networkservices.googleapis.com \\\n     secretmanager.googleapis.com \\\n     trafficdirector.googleapis.com\n   ```\n\n#### Configure GKE cluster\nThis is an example outlining minimal requirements to run the [baseline tests](#xds-baseline-tests).\nUpdate gloud sdk:\n```shell\ngcloud -q components update\n```\n\nPre-populate environment variables for convenience. To find project id, refer to\n[Identifying projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n```shell\nexport PROJECT_ID=\"your-project-id\"\nexport PROJECT_NUMBER=$(gcloud projects describe \"${PROJECT_ID}\" --format=\"value(projectNumber)\")\n# Compute Engine default service account\nexport GCE_SA=\"${PROJECT_NUMBER}-compute@developer.gserviceaccount.com\"\n# The prefix to name GCP resources used by the framework\nexport RESOURCE_PREFIX=\"xds-k8s-interop-tests\"\n\n# The zone name your cluster, f.e. xds-k8s-test-cluster\nexport CLUSTER_NAME=\"${RESOURCE_PREFIX}-cluster\"\n# The zone of your cluster, f.e. us-central1-a\nexport ZONE=\"us-central1-a\" \n# Dedicated GCP Service Account to use with workload identity.\nexport WORKLOAD_SA_NAME=\"${RESOURCE_PREFIX}\"\nexport WORKLOAD_SA_EMAIL=\"${WORKLOAD_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\"\n```\n\n##### Create the cluster \nMinimal requirements: [VPC-native](https://cloud.google.com/traffic-director/docs/security-proxyless-setup)\ncluster with [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) enabled. \n```shell\ngcloud container clusters create \"${CLUSTER_NAME}\" \\\n --scopes=cloud-platform \\\n --zone=\"${ZONE}\" \\\n --enable-ip-alias \\\n --workload-pool=\"${PROJECT_ID}.svc.id.goog\" \\\n --workload-metadata=GKE_METADATA \\\n --tags=allow-health-checks\n```\nFor security tests you also need to create CAs and configure the cluster to use those CAs\nas described\n[here](https://cloud.google.com/traffic-director/docs/security-proxyless-setup#configure-cas).\n\n##### Create the firewall rule\nAllow [health checking mechanisms](https://cloud.google.com/traffic-director/docs/set-up-proxyless-gke#creating_the_health_check_firewall_rule_and_backend_service)\nto query the workloads health.  \nThis step can be skipped, if the driver is executed with `--ensure_firewall`.\n```shell\ngcloud compute firewall-rules create \"${RESOURCE_PREFIX}-allow-health-checks\" \\\n  --network=default --action=allow --direction=INGRESS \\\n  --source-ranges=\"35.191.0.0/16,130.211.0.0/22\" \\\n  --target-tags=allow-health-checks \\\n  --rules=tcp:8080-8100\n```\n\n##### Setup GCP Service Account\n\nCreate dedicated GCP Service Account to use\nwith [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).\n\n```shell\ngcloud iam service-accounts create \"${WORKLOAD_SA_NAME}\" \\\n  --display-name=\"xDS K8S Interop Tests Workload Identity Service Account\"\n```\n\nEnable the service account to [access the Traffic Director API](https://cloud.google.com/traffic-director/docs/prepare-for-envoy-setup#enable-service-account).\n```shell\ngcloud projects add-iam-policy-binding \"${PROJECT_ID}\" \\\n   --member=\"serviceAccount:${WORKLOAD_SA_EMAIL}\" \\\n   --role=\"roles/trafficdirector.client\" \\\n   --condition=\"None\"\n```\n\n##### Allow access to images\nThe test framework needs read access to the client and server images and the bootstrap\ngenerator image. You may have these images in your project but if you want to use these\nfrom the grpc-testing project you will have to grant\nthe [necessary access](https://cloud.google.com/artifact-registry/docs/access-control#permissions)\nto these images. To grant access to images stored in `grpc-testing` project GCR,\nrun:\n\n```sh\ngcloud artifacts repositories add-iam-policy-binding \"projects/grpc-testing/locations/us/repositories/psm-interop\" \\\n  --member=\"serviceAccount:${GCE_SA}\" \\\n  --role=\"roles/artifactregistry.reader\" \\\n  --condition=None\n```\n\n```sh\ngcloud artifacts repositories add-iam-policy-binding \"projects/grpc-testing/locations/us/repositories/trafficdirector\" \\\n  --member=\"serviceAccount:${GCE_SA}\" \\\n  --role=\"roles/artifactregistry.reader\" \\\n  --condition=None\n```\n\nIf you get `PERMISSION_DENIED`, contact one of the repo\n[maintainers](https://github.com/grpc/psm-interop/blob/master/MAINTAINERS.md).\n\n##### Allow test driver to configure workload identity automatically\nTest driver will automatically grant `roles/iam.workloadIdentityUser` to \nallow the Kubernetes service account to impersonate the dedicated GCP workload\nservice account (corresponds to the step 5\nof [Authenticating to Google Cloud](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to)).\nThis action requires the test framework to have `iam.serviceAccounts.create`\npermission on the project.\n\nIf you're running test framework locally, and you have `roles/owner` to your\nproject, **you can skip this step**.  \nIf you're configuring the test framework to run on a CI: use `roles/owner`\naccount once to allow test framework to grant `roles/iam.workloadIdentityUser`.\n\n```shell\n# Assuming CI is using Compute Engine default service account.\ngcloud projects add-iam-policy-binding \"${PROJECT_ID}\" \\\n  --member=\"serviceAccount:${GCE_SA}\" \\\n  --role=\"roles/iam.serviceAccountAdmin\" \\\n  --condition-from-file=\u003c(cat \u003c\u003c-END\n---\ntitle: allow_workload_identity_only\ndescription: Restrict serviceAccountAdmin to granting role iam.workloadIdentityUser\nexpression: |-\n  api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', [])\n        .hasOnly(['roles/iam.workloadIdentityUser'])\nEND\n)\n```\n\n##### Configure GKE cluster access\n```shell\n# Unless you're using GCP VM with preconfigured Application Default Credentials, acquire them for your user\ngcloud auth application-default login\n\n# Install authentication plugin for kubectl.\n# Details: https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke\ngcloud components install gke-gcloud-auth-plugin\n\n# Configuring GKE cluster access for kubectl\ngcloud container clusters get-credentials \"${CLUSTER_NAME}\" --zone \"${ZONE}\"\n\n# Save generated kube context name\nexport KUBE_CONTEXT=\"$(kubectl config current-context)\"\n``` \n\n#### Install python dependencies\n\n```shell\n# Create python virtual environment\npython3 -m venv venv\n\n# Activate virtual environment\n. ./venv/bin/activate\n\n# Install requirements\npip install -r requirements.lock\n\n# Generate protos\npython -m grpc_tools.protoc --proto_path=. \\\n  --python_out=. --grpc_python_out=. --pyi_out=. \\\n  protos/grpc/testing/*.proto protos/grpc/testing/xdsconfig/*.proto \n```\n\n# Basic usage\n\n## Local development\nThis test driver allows running tests locally against remote GKE clusters, right\nfrom your dev environment. You need:\n\n1. Follow [installation](#installation) instructions\n2. Authenticated `gcloud`\n3. `kubectl` context (see [Configure GKE cluster access](#configure-gke-cluster-access))\n4. Run tests with `--debug_use_port_forwarding` argument. The test driver \n   will automatically start and stop port forwarding using\n   `kubectl` subprocesses. (experimental)\n\n### Making changes to the driver\n1. Install additional dev packages: `pip install -r requirements-dev.txt`\n2. Use `./bin/black.sh` and `./bin/isort.sh` helpers to auto-format code.\n\n### Updating Python Dependencies\n\nWe track our Python-level dependencies using three different files:\n\n- `requirements.txt`\n- `dev-requirements.txt`\n- `requirements.lock`\n\n`requirements.txt` lists modules without specific versions supplied, though\nversions ranges may be specified. `requirements.lock` is generated from\n`requirements.txt` and _does_ specify versions for every dependency in the\ntransitive dependency tree.\n\nWhen updating `requirements.txt`, you must also update `requirements.lock`. To\ndo this, navigate to this directory and run `./bin/freeze.sh`.\n\n### Setup test configuration\n\nThere are many arguments to be passed into the test run. You can save the\narguments to a config file (\"flagfile\") for your development environment.\nUse [`config/local-dev.cfg.example`](https://github.com/grpc/psm-interop/blob/main/config/local-dev.cfg.example)\nas a starting point:\n\n```shell\ncp config/local-dev.cfg.example config/local-dev.cfg\n```\n\nIf you exported environment variables in the above sections, you can\ntemplate them into the local config (note this recreates the config):\n\n```shell\nenvsubst \u003c config/local-dev.cfg.example \u003e config/local-dev.cfg\n```\n\nLearn more about flagfiles in [abseil documentation](https://abseil.io/docs/python/guides/flags#a-note-about---flagfile).\n\n## Test suites\n\nSee the full list of available test suites in the [`tests/`](https://github.com/grpc/psm-interop/blob/main/tests) folder. \n\n### xDS Baseline Tests\n\nTest suite meant to confirm that basic xDS features work as expected. Executing\nit before other test suites will help to identify whether test failure related\nto specific features under test, or caused by unrelated infrastructure\ndisturbances.\n\n```shell\n# Help\npython -m tests.baseline_test --help\npython -m tests.baseline_test --helpfull\n\n# Run the baseline test with local-dev.cfg settings\npython -m tests.baseline_test --flagfile=\"config/local-dev.cfg\"\n  \n# Same as above, but using the helper script\n./run.sh tests/baseline_test.py\n```\n\n### xDS Security Tests\nTest suite meant to verify mTLS/TLS features. Note that this requires\nadditional environment configuration. For more details, and for the \nsetup for the security tests, see\n[\"Setting up Traffic Director service security with proxyless gRPC\"](https://cloud.google.com/traffic-director/docs/security-proxyless-setup) user guide.\n\n```shell\n# Run the security test with local-dev.cfg settings\npython -m tests.security_test --flagfile=\"config/local-dev.cfg\"\n\n# Same as above, but using the helper script\n./run.sh tests/security_test.py\n```\n\n## Helper scripts\nYou can use interop xds-k8s [`bin/`](https://github.com/grpc/psm-interop/blob/main/bin)\nscripts to configure TD, start k8s instances step-by-step, and keep them alive\nfor as long as you need. \n\n* To run helper scripts using local config:\n  * `python -m bin.script_name --flagfile=config/local-dev.cfg`\n  * `./run.sh bin/script_name.py` automatically appends the flagfile\n* Use `--help` to see script-specific argument\n* Use `--helpfull` to see all available argument\n\n#### Overview\n```shell\n# Helper tool to configure Traffic Director with different security options\npython -m bin.run_td_setup --help\n\n# Helper tools to run the test server, client (with or without security)\npython -m bin.run_test_server --help\npython -m bin.run_test_client --help\n\n# Helper tool to verify different security configurations via channelz\npython -m bin.run_channelz --help\n```\n\n#### `./run.sh` helper\nUse `./run.sh` to execute helper scripts and tests with `config/local-dev.cfg`.\n\n```sh\nUSAGE: ./run.sh script_path [arguments]\n   script_path: path to python script to execute, relative to driver root folder\n   arguments ...: arguments passed to program in sys.argv\n\nENVIRONMENT:\n   XDS_K8S_CONFIG: file path to the config flagfile, relative to\n                   driver root folder. Default: config/local-dev.cfg\n                   Will be appended as --flagfile=\"config_absolute_path\" argument\n   XDS_K8S_DRIVER_VENV_DIR: the path to python virtual environment directory\n                            Default: $XDS_K8S_DRIVER_DIR/venv\nDESCRIPTION:\nThis tool performs the following:\n1) Ensures python virtual env installed and activated\n2) Exports test driver root in PYTHONPATH\n3) Automatically appends --flagfile=\"\\$XDS_K8S_CONFIG\" argument\n\nEXAMPLES:\n./run.sh bin/run_td_setup.py --help\n./run.sh bin/run_td_setup.py --helpfull\nXDS_K8S_CONFIG=./path-to-flagfile.cfg ./run.sh bin/run_td_setup.py --resource_suffix=override-suffix\n./run.sh tests/baseline_test.py\n./run.sh tests/security_test.py --verbosity=1 --logger_levels=__main__:DEBUG,framework:DEBUG\n./run.sh tests/security_test.py SecurityTest.test_mtls --nocheck_local_certs\n```\n\n## Partial setups\n### Regular workflow\n```shell\n# Setup Traffic Director\n./run.sh bin/run_td_setup.py\n\n# Start test server\n./run.sh bin/run_test_server.py\n\n# Add test server to the backend service\n./run.sh bin/run_td_setup.py --cmd=backends-add\n\n# Start test client\n./run.sh bin/run_test_client.py\n```\n\n### Secure workflow\n```shell\n# Setup Traffic Director in mtls. See --help for all options\n./run.sh bin/run_td_setup.py --security=mtls\n\n# Start test server in a secure mode\n./run.sh bin/run_test_server.py --mode=secure\n\n# Add test server to the backend service\n./run.sh bin/run_td_setup.py --cmd=backends-add\n\n# Start test client in a secure more --mode=secure\n./run.sh bin/run_test_client.py --mode=secure\n```\n\n### Sending RPCs\n#### Start port forwarding\n```shell\n# Client: all services always on port 8079\nkubectl port-forward deployment.apps/psm-grpc-client 8079\n\n# Server regular mode: all grpc services on port 8080\nkubectl port-forward deployment.apps/psm-grpc-server 8080\n# OR\n# Server secure mode: TestServiceImpl is on 8080, \nkubectl port-forward deployment.apps/psm-grpc-server 8080\n# everything else (channelz, healthcheck, CSDS) on 8081\nkubectl port-forward deployment.apps/psm-grpc-server 8081\n```\n\n#### Send RPCs with grpccurl\n```shell\n# 8081 if security enabled\nexport SERVER_ADMIN_PORT=8080\n\n# List server services using reflection\ngrpcurl --plaintext 127.0.0.1:$SERVER_ADMIN_PORT list\n# List client services using reflection\ngrpcurl --plaintext 127.0.0.1:8079 list\n\n# List channels via channelz\ngrpcurl --plaintext 127.0.0.1:$SERVER_ADMIN_PORT grpc.channelz.v1.Channelz.GetTopChannels\ngrpcurl --plaintext 127.0.0.1:8079 grpc.channelz.v1.Channelz.GetTopChannels\n\n# Send GetClientStats to the client\ngrpcurl --plaintext -d '{\"num_rpcs\": 10, \"timeout_sec\": 30}' 127.0.0.1:8079 \\\n  grpc.testing.LoadBalancerStatsService.GetClientStats\n```\n\n### Cleanup\n* First, make sure to stop port forwarding, if any\n* Run `./bin/cleanup.sh`\n\n##### Partial cleanup\nYou can run commands below to stop/start, create/delete resources however you want.  \nGenerally, it's better to remove resources in the opposite order of their creation.\n\nCleanup regular resources:\n```shell\n# Cleanup TD resources\n./run.sh bin/run_td_setup.py --cmd=cleanup\n# Stop test client\n./run.sh bin/run_test_client.py --cmd=cleanup\n# Stop test server, and remove the namespace\n./run.sh bin/run_test_server.py --cmd=cleanup --cleanup_namespace\n```\n\nCleanup regular and security-specific resources:\n```shell\n# Cleanup TD resources, with security\n./run.sh bin/run_td_setup.py --cmd=cleanup --security=mtls\n# Stop test client (secure)\n./run.sh bin/run_test_client.py --cmd=cleanup --mode=secure\n# Stop test server (secure), and remove the namespace\n./run.sh bin/run_test_server.py --cmd=cleanup --cleanup_namespace --mode=secure\n```\n\nIn addition, here's some other helpful partial cleanup commands:\n```shell\n# Remove all backends from the backend services\n./run.sh bin/run_td_setup.py --cmd=backends-cleanup\n\n# Stop the server, but keep the namespace\n./run.sh bin/run_test_server.py --cmd=cleanup --nocleanup_namespace\n```\n\n### Known errors\n#### Error forwarding port\nIf you stopped a test with `ctrl+c`, while using `--debug_use_port_forwarding`,\nyou might see an error like this:\n\n\u003e `framework.infrastructure.k8s.PortForwardingError: Error forwarding port, unexpected output Unable to listen on port 8081: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:8081: bind: address already in use]`\n\nUnless you're running `kubectl port-forward` manually, it's likely that `ctrl+c`\ninterrupted python before it could clean up subprocesses.\n\nYou can do `ps aux | grep port-forward` and then kill the processes by id,\nor with `killall kubectl`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrpc%2Fpsm-interop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrpc%2Fpsm-interop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrpc%2Fpsm-interop/lists"}