{"id":15221979,"url":"https://github.com/googlecloudplatform/gke-tracing-demo","last_synced_at":"2026-02-25T08:40:49.277Z","repository":{"id":43338999,"uuid":"142072163","full_name":"GoogleCloudPlatform/gke-tracing-demo","owner":"GoogleCloudPlatform","description":" This project introduces you to Stackdriver's tracing feature, and provides a distributed tracing example that can serve as a basis for your own applications. You will deploy a multi-tier application to a Kubernetes Engine cluster and trace calls between the components.","archived":false,"fork":false,"pushed_at":"2025-03-22T00:36:21.000Z","size":982,"stargazers_count":62,"open_issues_count":15,"forks_count":37,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-03-30T15:42:57.161Z","etag":null,"topics":["gke","gke-helmsman","google-cloud-platform","kubernetes","kubernetes-engine","stackdriver","tracing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README-QWIKLABS.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-23T21:38:02.000Z","updated_at":"2023-12-09T07:57:28.000Z","dependencies_parsed_at":"2024-04-17T01:27:48.859Z","dependency_job_id":"6d927088-9f6b-424a-a5e8-5d99dede9791","html_url":"https://github.com/GoogleCloudPlatform/gke-tracing-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-tracing-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-tracing-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-tracing-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-tracing-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/gke-tracing-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247657281,"owners_count":20974345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gke","gke-helmsman","google-cloud-platform","kubernetes","kubernetes-engine","stackdriver","tracing"],"created_at":"2024-09-28T15:09:30.886Z","updated_at":"2026-02-25T08:40:49.250Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tracing with Stackdriver on Kubernetes Engine\n\n\u003c!-- omit in toc --\u003e\n\n  * [Tracing with Stackdriver on Kubernetes Engine ](#tracing-with-stackdriver-on-kubernetes-engine-)\n  * [Introduction](#introduction)\n  * [Architecture](#architecture)\n  * [Initialize gcloud](#initialize-gcloud)\n     * [Enable GCP APIs](#enable-gcp-apis)\n  * [Deployment](#deployment)\n     * [Introduction to Terraform](#introduction-to-terraform)\n     * [Running Terraform](#running-terraform)\n        * [Initialization](#initialization)\n        * [Deployment](#deployment-1)\n  * [Validation](#validation)\n     * [Generating Telemetry Data](#generating-telemetry-data)\n     * [Examining Traces](#examining-traces)\n     * [Pulling Pub/Sub Messages](#pulling-pubsub-messages)\n     * [Monitoring and Logging](#monitoring-and-logging)\n  * [Teardown](#teardown)\n  * [Troubleshooting](#troubleshooting)\n  * [Relevant Materials](#relevant-materials)\n     * [Kubernetes](#kubernetes)\n     * [OpenCensus](#opencensus)\n     * [Spring Sleuth](#spring-sleuth)\n     * [Stackdriver](#stackdriver)\n     * [Terraform](#terraform)\n     * [Zipkin](#zipkin)\n\n\u003c!-- omit in toc --\u003e\n\n## Introduction\n\nWhen supporting a production system that services HTTP requests or provides an\nAPI, it is important to measure the latency of your endpoints to detect when a\nsystem's performance is not operating within specification. In monolithic\nsystems this single latency measure may be useful to detect and diagnose\ndeteriorating behavior. With modern microservice architectures, however, this\nbecomes much more difficult because a single request may result in numerous\nadditional requests to other systems before the request can be fully handled.\nDeteriorating performance in an underlying system may impact all other systems\nthat rely on it. While latency can be measured at each service endpoint it can\nbe difficult to correlate slow behavior in the public endpoint with a\nparticular sub-service that is misbehaving.\n\nEnter distributed tracing. Distributed tracing uses metadata passed along with\nrequests to correlate requests across service tiers. By collecting telemetry\ndata from all the services in a microservice architecture and propagating a\n_trace id_ from an initial request to all subsidiary requests, developers can\nmuch more easily identify which service is causing slowdowns affecting the rest\nof the system.\n\n[Google Cloud Platform (GCP)](https://cloud.google.com/) provides the\n[Stackdriver](https://cloud.google.com/stackdriver/) suite of products to\nhandle logging, monitoring, and distributed tracing. This document will discuss\nthe latter feature and provide a distributed tracing demo that can service as a\nbasis for your own applications. The demo will be deployed to\n[Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) and will\ndemonstrate a multi-tier architecture implementing distributed tracing. It will\nalso take advantage of [Terraform](#terraform) to build out necessary\ninfrastructure.\n\nThis demo requires an active GCP account. You can sign up for a\n[free account](https://cloud.google.com/) if you don't already have one. It\nwill come with free credits for beginning your GCP experience.\n\nThis demo is designed for MacOS and Linux systems but users can alternatively\nuse the [Google Cloud Shell](https://cloud.google.com/shell/docs/) entirely in\nthe cloud without requiring a local system.\n\n## Architecture\n\nThis demonstration application will begin by deploying a Kubernetes Engine\ncluster. To this cluster will be deployed a simple web application fronted by a\nload balancer. The web app will publish messages provided by the user to a\n[Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/overview) topic. The\napplication is instrumented such that HTTP requests to it will result in the\ncreation of a trace whose context will be propagated to the Cloud Pub/Sub\npublish API request. The correlated telemetry data from these requests will be\navailable in the Stackdriver Trace Console.\n\n![](images/architecture.png)\n\n## Initialize gcloud\n\nWhen using Cloud Shell execute the following command in order to setup gcloud cli. When executing this command please setup your region and zone.\n\n```console\ngcloud init\n```\n\n### Enable GCP APIs\n\nThe following APIs need to be enabled:\n* Kubernetes Engine API\n* Stackdriver Trace API\n\nThe following commands will enable these APIs:\n\n```console\ngcloud services enable container.googleapis.com\ngcloud services enable cloudtrace.googleapis.com\n```\n\n## Deployment\n\n### Introduction to Terraform\n\nFollowing the principles of\n[infrastructure as code](https://en.wikipedia.org/wiki/Infrastructure_as_Code)\nand\n[immutable infrastructure](https://www.oreilly.com/ideas/an-introduction-to-immutable-infrastructure),\nTerraform supports the writing of declarative descriptions of the desired state\nof infrastructure. When the descriptor is applied, Terraform uses GCP APIs to\nprovision and update resources to match. Terraform compares the desired state\nwith the current state so incremental changes can be made without deleting\neverything and starting over.  For instance, Terraform can build out GCP\nprojects and compute instances, etc., even set up a Kubernetes Engine cluster\nand deploy applications to it. When requirements change, the descriptor can be\nupdated and Terraform will adjust the cloud infrastructure accordingly.\n\nThis example will start up a Kubernetes Engine cluster using Terraform. Then\nyou will use Kubernetes commands to deploy a demo application to the cluster.\nBy default, Kubernetes Engine clusters in GCP are launched with a\npre-configured [Fluentd](https://www.fluentd.org/)-based collector that\nforwards logging events for the cluster to Stackdriver. Interacting with the\ndemo app will produce trace events that are visible in the\n[Stackdriver Trace UI](https://console.cloud.google.com/traces).\n\n### Running Terraform\n\nThere are three Terraform files provided with this example, located in the `/terraform` subdirectory of the project. The first one, `main.tf`, is the starting point for Terraform. It describes the features that will be used, the resources that will be manipulated, and the outputs that will result. The second file is `provider.tf`, which indicates which cloud provider and version will be the target of the Terraform commands--in this case GCP. The final file is `variables.tf`, which contains a list of variables that are used as inputs into Terraform. Any variables referenced in the `main.tf` that do not have defaults configured in `variables.tf` will result in prompts to the user at runtime.\n\n#### Initialization\nGiven that authentication was [configured](#configure-authentication) above, we\nare now ready to deploy the infrastructure. Run the following command from the root directory of the project:\n\n```console\ncd terraform\n```\n\nOnce there, Terraform needs to be initialized. This will download the dependencies that Terraform requires to function. Enter:\n```console\nterraform init\n```\n\nFor this demo, Terraform needs two pieces of information in order to run: the GCP _project_ and the GCP _zone_ to which the demo application should be deployed. Terraform will prompt for these values if it does not know them already. By default, it will look for a file called `terraform.tfvars` or files with a suffix of `.auto.tfvars` in the current directory to obtain those values. This demo provides a convenience script to prompt for project and zone and persist them in a `terraform.tfvars` file. Run:\n\n```console\n../scripts/generate-tfvars.sh\n```\n\nIf the file already exists you will receive an error.\n\nThe script uses previously-configured values from the `gcloud` command. If they have not been configured, the error message will indicate how they should be set. The existing values can be viewed with the following command:\n\n```console\ngcloud config list\n```\n\nIf the displayed values don't correspond to where you intend to run the demo application, change the values in `terraform.tfvars` to your preferred values.\n\n#### Deployment\n\nHaving initialized Terraform you can see the work that Terraform will perform with the following command:\n\n```console\nterraform plan\n```\n\nThis command can be used to visually verify that settings are correct and Terraform will inform you if it detects any errors. While not necessary, it is a good practice to run it every time prior to changing infrastructure using Terraform.\n\nAfter verification, tell Terraform to set up the necessary infrastructure:\n\n```console\nterraform apply\n```\n\nIt display the changes that will be made and asks you to confirm with `yes`.\n\nAfter a few minutes you should find your [Kubernetes Cluster](https://console.cloud.google.com/kubernetes) and [Pub/Sub Topic and Subscription](https://console.cloud.google.com/cloudpubsub) in the GCP console.\n\nNow, deploy the demo application using Kubernetes's `kubectl` command:\n\n```console\nkubectl apply -f tracing-demo-deployment.yaml\n```\n\nOnce the app has been deployed, it can be viewed in the [Workload](https://console.cloud.google.com/kubernetes/workload) tab of the Kubernetes Engine Console. You can also see the load balancer that was created for the application in the [Services](https://console.cloud.google.com/kubernetes/discovery) section of the console.\n\nIncidentally, the endpoint can be programmatically acquired using the following command:\n\n```console\necho \"http://$(kubectl get svc tracing-demo -n default -ojsonpath='{.status.loadBalancer.ingress[0].ip}')\"\n```\n\n## Validation\n\n### Generating Telemetry Data\n\nOnce the demo application is deployed, go to\nhttps://console.cloud.google.com/kubernetes/discovery and if it isn't already\nselected select the project used for this demo with the combobox in the\nupper-left corner. You should see a list of your exposed services:\n\n![](images/services-ui.png)\n\nClicking on the endpoint listed next to the `tracing-demo` load balancer will\nopen the demo app web page in a new tab of your browser. Note that your IP\naddress will likely be different from the example above. The page displayed is\nsimple:\n\n![](images/hello-world-browser.png)\n\nTo the url, add the string: `?string=CustomMessage` and see that the message is\ndisplayed:\n\n![](images/custom-message-browser.png)\n\nAs you can see, if a `string` parameter is not provided it uses a default value\nof `Hello World`. The app itself isn't very interesting. We are simply using it\nto generate trace telemetry data.\n\n### Examining Traces\n\nNavigate to https://console.cloud.google.com/traces/traces and select your\nproject from the combobox in the upper left-hand corner of your screen if it is\nnot already selected. You should see a chart displaying trace events plotted on\na timeline with latency as the vertical metric.\n\n![](images/trace-ui.png)\n\nClick on one of the dots on the chart. You will see a chart with two bars, the\ntop one longer than the bottom one:\n\n![](images/span-ui-full.png)\n\nThe top bar is known as the _root span_ and represents the duration of the HTTP\nrequest, from the moment the first byte arrives until the moment the last byte\nof the response is sent. The bottom bar represents the duration of the request\nmade when sending the message to the Pub/Sub topic.\n\nSince the handling of the HTTP request is blocked by the call to the Pub/Sub\nAPI it is clear that the vast majority of the time spent within the HTTP\nrequest is taken up by the Pub/Sub interaction. This demonstrates that by\ninstrumenting each tier of your application you can easily identify where the\nbottlenecks are.\n\n### Pulling Pub/Sub Messages\n\nAs described in the [Architecture](#architecture) section of this document,\nmessages from the demo app are published to a Pub/Sub topic. These messages can\nbe consumed from the topic using the `gcloud` CLI:\n\n```console\ngcloud pubsub subscriptions pull --auto-ack --limit 10 tracing-demo-cli\n┌─────────────┬─────────────────┬────────────┐\n│     DATA    │    MESSAGE_ID   │ ATTRIBUTES │\n├─────────────┼─────────────────┼────────────┤\n│ Hello World │ 121487601177521 │            │\n└─────────────┴─────────────────┴────────────┘\n```\n\nPulling the messages from the topic has no impact on tracing. This section\nsimply provides a consumer of the messages for verification purposes.\n\n### Monitoring and Logging\n\nStackdriver monitoring and logging are not the subject of this demo, but it is\nworth noting that the application you deployed will publish logs to\n[Stackdriver Logging](https://console.cloud.google.com/logs/viewer) and\nmetrics to [Stackdriver Monitoring](https://app.google.stackdriver.com/)\n\nFor instance, see https://app.google.stackdriver.com/metrics-explorer and\nselect the `GKE Container` Resource Type and the `CPU usage` metric. You should\nsee a chart of this metric plotted for different pods running in the cluster:\n\n![](images/metrics.png)\n\nAs well, logs can be seen at https://console.cloud.google.com/logs/viewer:\n\n![](images/logs.png)\n\nYou may need to click the `Load older logs` button to display published logs.\n\n## Teardown\n\nWhen you are finished with this example you will want to clean up the resources\nthat were created so that you avoid accruing charges:\n\n```console\nterraform destroy\n```\n\nAs with `apply`, Terraform will prompt for a `yes` to confirm your intent.\n\nSince Terraform tracks the resources it created it is able to tear down the\ncluster, the Pub/Sub topic, and the Pub/Sub subscription.\n\n## Troubleshooting\n\n1. You may run into an issue where an API has not been enabled for your\n   project. If you receive such an error the necessary API name may be listed. You\n   can enable it with:\n\n   ```console\n   gcloud services list \u003cservice-name\u003e\n   ```\n\n   where `\u003cservice-name\u003e` is the identified service name. If the disabled API is\n   not provided you can see a list of those available with the following command:\n\n   ```console\n   gcloud services list --available\n   ```\n\n   Identify the necessary service and enable it with the above command. Keep in\nmind the\n[principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)\nand limit access to those strictly necessary to run your applications.\n2. A number of possible errors can be diagnosed using the `kubectl` command.\n   For instance, a deployment can be shown:\n   ```console\n   kubectl get deployment tracing-demo\n   NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\n   tracing-demo   1         1         1            1           2h\n   ```\n\n   More details can be shown with `describe`:\n\n   ```console\n   kubectl describe deployment tracing-demo\n   ...\n   ```\n\n   This command will show a list of deployed pods:\n\n   ```console\n   kubectl get pod\n   NAME                            READY     STATUS    RESTARTS   AGE\n   tracing-demo-59cc7988fc-h5w27   1/1       Running   0          2h\n   ```\n   Again, details of the pod can be seen with `describe`:\n   ```console\n   kubectl describe pod tracing-demo\n   ...\n   ```\n\n   Once the pod name is known logs can be viewed locally:\n\n   ```console\n   kubectl logs tracing-demo-59cc7988fc-h5w27\n   10.60.0.1 - - [22/Jun/2018:19:42:23 +0000] \"HEAD / HTTP/1.0\" 200 - \"-\" \"-\"\n   Publishing string: Hello World\n   10.60.0.1 - - [22/Jun/2018:19:42:23 +0000] \"GET / HTTP/1.1\" 200 669 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36\"\n   ```\n3. The install script fails with a `Permission denied` when running Terraform.\n   The credentials that Terraform is using do not provide the necessary\n   permissions to create resources in the selected projects. Ensure that the\n   account listed in `gcloud config list` has necessary permissions to create\n   resources. If it does, regenerate the application default credentials using\n   `gcloud auth application-default login`.\n\n## Relevant Materials\n\nThe following are some relevant materials for further investigation:\n\n### Kubernetes\n[Kubernetes](https://kubernetes.io/) is a container orchestration platform\npopular in microservice architectures. GCP provides a managed version of\nKubernetes called [Kubernetes Engine](https://cloud.google.com/kubernetes-engine/).\n\n### OpenCensus\n[OpenCensus](https://opencensus.io/) provides standardized libraries for\ncapturing and publishing trace telemetry data. Libraries are provided for a\nnumber of popular languages and numerous trace platforms are supported,\nincluding [Stackdriver](#stackdriver) and [Zipkin](#zipkin). The demo this\ndocument describes uses OpenCensus to publish telemetry data to Stackdriver.\n\n### Spring Sleuth\n[Spring Sleuth](https://cloud.spring.io/spring-cloud-sleuth/) provides\ninstrumentation of Java applications that use the popular\n[Spring](https://spring.io/) framework. Spring Sleuth supports an abstraction\nover distributed trace telemetry collectors so developers can seamlessly switch\nbetween [Zipkin](#zipkin), [Stackdriver](#stackdriver), and other trace engines.\n\n### Stackdriver\n[Stackdriver](https://cloud.google.com/stackdriver/) is a suite of tools in GCP\nproviding logging, monitoring, tracing, and related features. This document and\ndemo are particularly concerned with the\n[Stackdriver Trace](https://cloud.google.com/trace/) feature it provides.\n\n### Terraform\n[Terraform](https://www.terraform.io/) is a declarative\n[infrastructure as code](https://en.wikipedia.org/wiki/Infrastructure_as_Code)\ntool that enables configuration files to be used to automate the deployment and\nevolution of infrastructure in the cloud.\n\n### Zipkin\n[Zipkin](https://zipkin.io/) is a distributed tracing tool that helped\npopularize the practice. Many frameworks provide Zipkin\n[integration](https://zipkin.io/pages/existing_instrumentations.html) and those\nthat don't can [implement their own](https://zipkin.io/pages/instrumenting).\n\nApplications that are already instrumented for Zipkin can have their Zipkin\ntelemetry data adapted to Stackdriver events through the use of a\n[Zipkin Collector](https://cloud.google.com/trace/docs/zipkin). It can be\ndeployed to Kubernetes Engine:\n\n```console\nkubectl run stackdriver-zipkin   --image=gcr.io/stackdriver-trace-docker/zipkin-collector   --expose --port=9411\n```\n\nThis will deploy the collector to the well known Zipkin port `9411` and\napplications looking for it on their local port will find it indistinguishable\nfrom a Zipkin server but telemetry data will appear in Stackdriver Trace.\n\n\n**This is not an officially supported Google product**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-tracing-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fgke-tracing-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-tracing-demo/lists"}