{"id":19900079,"url":"https://github.com/m-lab/gcp-service-discovery","last_synced_at":"2025-05-02T23:30:28.441Z","repository":{"id":48405355,"uuid":"90635968","full_name":"m-lab/gcp-service-discovery","owner":"m-lab","description":"Prometheus service discovery for targets running in App Engine Flex, Kubernetes, and generic HTTP(s) targets.","archived":false,"fork":false,"pushed_at":"2024-06-10T04:47:56.000Z","size":135,"stargazers_count":8,"open_issues_count":15,"forks_count":3,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-10-29T16:01:46.302Z","etag":null,"topics":["prometheus","service-discovery"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-08T14:13:18.000Z","updated_at":"2023-06-19T11:21:02.000Z","dependencies_parsed_at":"2023-12-31T04:01:13.613Z","dependency_job_id":"fc7ecaf2-263e-4a03-9221-6cca317b7d2d","html_url":"https://github.com/m-lab/gcp-service-discovery","commit_stats":{"total_commits":106,"total_committers":6,"mean_commits":"17.666666666666668","dds":0.5377358490566038,"last_synced_commit":"905d802d95d15401000da15f86efa3e3c19ffddc"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fgcp-service-discovery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fgcp-service-discovery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fgcp-service-discovery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fgcp-service-discovery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/gcp-service-discovery/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224341480,"owners_count":17295271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["prometheus","service-discovery"],"created_at":"2024-11-12T20:11:05.296Z","updated_at":"2024-11-12T20:11:05.746Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![GoDoc](https://godoc.org/github.com/m-lab/pusher?status.svg)](https://godoc.org/github.com/m-lab/pusher) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/gcp-service-discovery)](https://goreportcard.com/report/github.com/m-lab/gcp-service-discovery) [![Build Status](https://travis-ci.org/m-lab/gcp-service-discovery.svg?branch=master)](https://travis-ci.org/m-lab/gcp-service-discovery) [![Coverage Status](https://coveralls.io/repos/github/m-lab/gcp-service-discovery/badge.svg?branch=master)](https://coveralls.io/github/m-lab/gcp-service-discovery?branch=master)\n\n# gcp-service-discovery\n\ngcp-service-discovery extends Prometheus service discovery by using Google\nCloud Platform (GCP) APIs to discover targets to scrape.\n\nUsing metadata collected during discovery, gcp-service-discovery generates\nand writes [file-based service discovery][filesd] target configuration, which\nPrometheus can read directly.\n\nSupported discovery sources:\n\n* Scraping individual AppEngine Flex Instances - using AppEngine Admin API\n* Scraping Prometheus federation in GKE Clusters - using Kubernetes Engine API\n* Download generic, pre-generated HTTP(s) targets - using Go http.Client\n\nAdditional configuration is necessary for AppEngine Flex Instances and GKE\nServices.\n\n[filesd]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E\n\n## AppEngine Flex Instances\n\nAppEngine Flex runs GCE VMs (instances) behind a public-facing load balancer.\nSo by default, these GCE VMs are not individually addressable, which makes\nstandard Prometheus scraping impossible.\n\ngcp-service-discovery uses the [GCP AppEngine Admin API][aeflexapi] to query\nall known services, versions, and instances to discovery instances with\nforwarded ports that *can* be scraped directly by Prometheus.\n\nFor this discovery to work, the AppEngine Flex service configuration must\ninclude a network `forwarded_ports` section like the one below. The config\nforwards a container port to a port on the GCE VM's public IP. The service you\ndeploy to AppEngine should export metrics on the same port.\n\nNOTE: metrics could be publicly visible and will be unencrypted by default.\n\nExample AppEngine Flex network configuration:\n```\nnetwork:\n  instance_tag: \u003ctag name\u003e\n  name: default\n  # Note: default AppEngine container port 8080 cannot be forwarded.\n  forwarded_ports:\n    - 9990/tcp\n```\n\n[aeflexapi]: https://cloud.google.com/appengine/docs/admin-api/reference/rest/\n\n## GKE Services\n\nPrometheus service discovery works automatically within a single Kubernetes\ncluster. So, for some it is common practice to run one Prometheus server per\ncluster. In order to take advantage of metric aggregation across clusters\nusing [Prometheus federation][federation], we want to automatically discover\nthese other Prometheus servers.\n\ngcp-service-discovery use the [GCP Kubernetes Engine API][gkeapi] to discover\ncredentials for all known clusters and then directly queries the master API\nto find services with a specific annotation.\n\nFor this discovery to work, the service must include the annotation\n`gke-prometheus-federation/scrape=true`.\n\nNOTE: depending on the service type and VPC network configuration, the\nPrometheus instance may be publicly accessible.\n\n```\napiVersion: v1\nkind: Service\nmetadata:\n  ... [snip]\n  annotations:\n    gke-prometheus-federation/scrape: 'true'\nspec:\n  ports:\n  - port: 9090\n    protocol: TCP\n    targetPort: 9090\n  ... [snip]\n```\n\n[federation]: https://prometheus.io/docs/prometheus/latest/federation/\n[gkeapi]: https://cloud.google.com/kubernetes-engine/docs/reference/rest/\n\n# Running gcp-service-discovery\n\nTo run this locally using docker, try:\n```\ndocker run -d measurementlab/gcp-service-discovery:latest \\\n    --aef-target=/targets/aeflex.json \\\n    --gke-target=/targets/gke.json \\\n    --http-target=/targets/http.json \\\n    --http-source=https://some-random-url-or-service.com/targets.json\n```\n\nIn kubernetes, the gcp-service-discovery container should be deployed as a\nsidecar service with Prometheus.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fgcp-service-discovery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fgcp-service-discovery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fgcp-service-discovery/lists"}