{"id":19900063,"url":"https://github.com/m-lab/prometheus-support","last_synced_at":"2025-05-02T22:32:10.403Z","repository":{"id":15437217,"uuid":"78147283","full_name":"m-lab/prometheus-support","owner":"m-lab","description":"Prometheus configuration for M-Lab running on GKE","archived":false,"fork":false,"pushed_at":"2025-04-24T09:10:09.000Z","size":3217,"stargazers_count":20,"open_issues_count":65,"forks_count":11,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-25T12:22:50.280Z","etag":null,"topics":["prometheus"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-01-05T20:50:03.000Z","updated_at":"2025-04-08T21:40:36.000Z","dependencies_parsed_at":"2023-02-16T23:45:56.136Z","dependency_job_id":"b87483b7-9b2b-49ec-93d1-19b00bd5f513","html_url":"https://github.com/m-lab/prometheus-support","commit_stats":{"total_commits":1511,"total_committers":19,"mean_commits":79.52631578947368,"dds":0.5757776307081404,"last_synced_commit":"81b6e3d7912b064deda37fb5931412206b82dbc3"},"previous_names":[],"tags_count":215,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-support","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-support/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-support/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-support/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/prometheus-support/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252116491,"owners_count":21697393,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["prometheus"],"created_at":"2024-11-12T20:11:02.852Z","updated_at":"2025-05-02T22:32:10.387Z","avatar_url":"https://github.com/m-lab.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# prometheus-support\n\nPrometheus configuration for M-Lab.\n\n# Deploying Prometheus to Kubernetes\n\nThe following instructions presume there is already a kubernetes cluster\ncreated in a GCP project and the cluster is accessible with `kubectl`.\n\nSee the [container engine quickstart guide][quickstart] for a simple howto.\n\n[quickstart]: https://cloud.google.com/container-engine/docs/quickstart\n\n# Creating a cluster for Prometheus\n\n## Federation\n\nTo support service discovery across GCP, the cloud-platform scope is currently\nnecessary.\n\n```\ngcloud container \\\n  --project \"mlab-sandbox\" clusters create \"prometheus-federation\" \\\n  --zone \"us-central1-a\" \\\n  --machine-type=n1-standard-8 \\\n  --scopes \"https://www.googleapis.com/auth/cloud-platform\" \\\n  --num-nodes 3 \\\n  --enable-autorepair --enable-autoupgrade \\\n  --node-labels=prometheus-node=true\n```\n\n### Upgrade deployment resource limits\n\nTo increase the resources assigned to the prometheus server or the set of all\nservices, you may need a larger node pool.\n\n```\ngcloud --project=mlab-staging container node-pools create default-pool-N \\\n  --cluster=prometheus-federation \\\n  --num-nodes=3 \\\n  --zone=us-east1-c \\\n  --scopes \"https://www.googleapis.com/auth/cloud-platform\" \\\n  --node-labels=prometheus-node=true \\\n  --enable-autorepair --enable-autoupgrade \\\n  --machine-type=n1-standard-8\n```\n\nAdjust the project, cluster name, and zone as appropriate. If the cluster is\nusing static public IPs, use the same zone as the current cluster; static IPs\nare associated with a single zone. It is necessary to configure a static IP\nby going to:\n\n    GCP -\u003e VPC Networking -\u003e External IP Addresses\n\nand then configure Cloud DNS for your\n`{grafana, prometheus, alertmanager}.\u003ccluster\u003e.measurementlab.net` subdomains\nto point at that IP address:\n\n    GCP -\u003e Network Services -\u003e Cloud DNS\n\nin order for automatic TLS certificates to be generated.\n\nThen update the resource \"request\" and \"limit\" in the appropriate deployment\nconfiguration, as well as the IP address:\n\n    `k8s/\u003cproject\u003e/\u003ccluster\u003e/deployment/\u003cdeployment.yml\u003e`\n\nIt may also be necessary to update the:\n\n    GCP -\u003e Networking -\u003e Load Balancer -\u003e Backend\n\nto include the new node pool. Otherwise, network requests will appear to hang.\n\nAlso, k8s may fail to detach a peristent volume from one node pool and make it\navailable to the other. This will manifest as a deployment's pod\n'CreatingContainer' indefinitely. And, the `kubectl get events` will include\nrelated error messages.\n\n## Within an existing cluster\n\nNo special scopes are required for the prometheus cluster configuration.\nHowever, node labels can only be added to a new node group using the command\nline.\n\nFor very large number of time series (e.g. production prometheus-federation) a\nhighmem node pool is necessary.\n\n```\ngcloud --project=mlab-oti container node-pools create prometheus-pool \\\n  --cluster=prometheus-federation \\\n  --num-nodes=2 \\\n  --zone=us-central1-a \\\n  --node-labels=prometheus-node=true \\\n  --enable-autorepair --enable-autoupgrade \\\n  --machine-type=n1-highmem-16\n```\n\n# Roles\n\nSince k8s version 1.6, stricter access controls are the default.\n\nPlease understand the [RBAC permission model][rbac] and the objects they apply\nto.\n\nBefore you can run either of the `apply-cluster.sh` or\n`apply-global-prometheus.sh` scripts, your user account must have the\n`cluster-admin` role.\n\nTo assign this role:\n\n* You may assign yourself the 'cluster-admin' role directly for the cluster.\n\n```\nkubectl create clusterrolebinding additional-cluster-admins \\\n    --clusterrole=cluster-admin \\\n    --user=\u003cyour-email\u003e\n```\n\n```\nkubectl create clusterrolebinding additional-cluster-admins \\\n    --clusterrole=cluster-admin \\\n    --user=\u003cservice-account-address\u003e\n```\n\nThe above steps also apply to other clusters managed by this repo, e.g.\ndata-pipeline.\n\nThis repository uses Cloud Build to build and deploy changes to the cluster. The\ndefault Cloud Build service account cannot deploy the cluster without having\ncertain RBAC permissions assigned to it in the cluster in advance, before\n`apply-global-prometheus.sh` runs for the first time. In order to apply\nthese changes, run the following script (located in this directory):\n\n```\n./bootstrap-prometheus-federation.sh \u003ccluster project\u003e \u003ccluster zone or region\u003e\n```\n\n[rbac]: https://kubernetes.io/docs/admin/authorization/rbac/\n\n# Using Kubernetes config files\n\nKubernetes config files preserve a deployment configuration and provide a\nconvenient mechanism for review and automated deployments.\n\nSome steps cannot be automated. For example, while a LoadBalancer can\nautomatically assign a public IP address to a service, it will not ([yet][dns])\nupdate corresponding DNS records for that IP address. So, we must reserve a\nstatic IP through the Cloud Console interface first.\n\n*Also, note*: Only one GKE cluster at a time can use the static IPs allocated\nin the `k8s/.../services.yml` files. If you are using an additional GKE cluster\n(e.g.  in mlab-sandbox project), update the services yaml files to use the\nnew static IP allocation.\n\n[dns]: https://github.com/kubernetes-incubator/external-dns\n\n## ConfigMaps\n\nMany services like prometheus provide canonical docker images published to\n[Dockerhub][dockerhub] (or other registry). We can customize the deployment by\nchanging the configuration at run time using ConfigMaps. For detailed background\nsee the [official docs][configmaps].\n\nCreate the ConfigMap for prometheus (only create a ConfigMap for one of cluster\nor federation, not both):\n\nTODO(soltesz): move environment variables (like gcloud-project) to a separate\ndirectory.\n\n    # For federation.\n    kubectl create configmap prometheus-federation-config \\\n        --from-literal=gcloud-project=mlab-sandbox \\\n        --from-file=config/federation/prometheus\n\n    # For cluster.\n    kubectl create configmap prometheus-cluster-config \\\n        --from-file=config/cluster/prometheus\n\nAlthough the flag is named `--from-file`, it accepts a directory. With this\nflag, kubernetes creates a ConfigMap with keys equal to the filenames, and\nvalues equal to the content of the file.\n\n    kubectl describe configmap prometheus-cluster-config\n\n      Name:       prometheus-cluster-config\n      Namespace:  default\n      Labels:     \u003cnone\u003e\n      Annotations:    \u003cnone\u003e\n\n      Data\n      ====\n      prometheus.yml: 9754 bytes\n\nWe can now refer to this ConfigMap in the \"deployment\" configuration later. For\nexample, `k8s/mlab-sandbox/prometheus-federation/deployments/prometheus.yml`\nuses a configmap as a volume so that the config file\n`config/federation/prometheus.yml` appears under `/etc/prometheus`.\n\nFor example, this will look something like (with abbreviated configuration):\n\n    - containers:\n      ...\n        volumeMounts:\n          # /etc/prometheus/prometheus.yml should contain the M-Lab Prometheus\n          # config.\n          - mountPath: /etc/prometheus\n            # This name refers to the 'volumes.name'\n            name: prometheus-config\n      volumes:\n      - name: prometheus-config\n        configMap:\n          # This name refers to the configmap name.\n          name: prometheus-federation-config\n\nNote: Configmaps only support text data. Secrets may be an alternative for\nbinary data. \u003chttps://github.com/kubernetes/kubernetes/issues/32432\u003e\n\n[dockerhub]: https://hub.docker.com/r/prom/prometheus/\n[configmaps]: https://kubernetes.io/docs/user-guide/configmap/\n\n### Verify that a ConfigMap is Mounted\n\nWhen a pod has mounted a configmap (or other resource), it is visible in the\n\"Volume Mounts\" status reported by `kubectl describe`.\n\nFor example (with abbreviated output):\n\n    podname=$( kubectl get pods -o name --selector='run=prometheus-server' )\n    kubectl describe ${podname}\n      ...\n      Containers:\n        prometheus-server:\n          ...\n          Image:              prom/prometheus:v1.5.2\n          ...\n          Port:               9090/TCP\n          ...\n          Volume Mounts:\n            /etc/prometheus from prometheus-cluster-config (rw)\n            /legacy-targets from prometheus-storage (rw)\n            /prometheus from prometheus-storage (rw)\n          ...\n\n### Update a ConfigMap\n\nWhen the content of a configmap value needs to change, you can either delete and\ncreate the configmap object (not ideal), or replace the new configuration all at\nonce.\n\n    kubectl create configmap prometheus-federation-config --from-file=prometheus \\\n        --dry-run -o json | kubectl apply -f -\n\nAfter updating a configmap, any pods that use this configmap will need to be\nrestarted for the change to take effect.\n\n    podname=$( kubectl get pods -o name --selector='run=prometheus-server' )\n    kubectl delete ${podname}\n\nThe deployment replica set will automatically recreate the pod and the new\nprometheus server will use the updated configmap. This is a known issue:\n\u003chttps://github.com/kubernetes/kubernetes/issues/13488\u003e\n\nPreferably the process (like prometheus) will support a 'reload' operation.\nHowever, it can take several minutes for the configmap to be updated from the\nperspective of the container. So, do not run reload until the configmap is up\nto date. (TODO: how to confirm this from outside the container?)\n\n# Using Kubernetes Secrets\n\nSecrets are like ConfigMaps in that they can be a source for environment\nvariables and volume mounts. Unlike ConfigMaps, secrets contain confidential\nmaterial, like certificates, passwords, access tokens, or similar.\n\n## Grafana secrets\n\nThe Grafana configuration needs a pre-defined password for the admin user.\nCreate one using a command like this:\n\n```\nkubectl create secret generic grafana-secrets \\\n    --from-literal=admin-password=[redacted text]\n```\n\nTo recover the password:\n\n```\njsonpath='{.items[?(@.metadata.name==\"grafana-secrets\")].data.admin-password}'\nkubectl get secrets -o jsonpath=\"${jsonpath}\" | base64 --decode \u0026\u0026 echo ''\n```\n\n# Create deployment\n\nBefore beginning, verify that you are [operating on the correct kubernetes\ncluster][cluster].\n\nThen, update `k8s/mlab-sandbox/prometheus-federation/deployments/prometheus.yml`\nto reference the current stable prometheus container tag. Now, deploy the\nservice.\n\nCreate a storage class for GCE persistent disks:\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluster-name\u003e/persistentvolumes/storage-class.yml\n\nCreate a persistent volume claim that Prometheus will bind to:\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluster-name\u003e/persistentvolumes/persistent-volumes.yml\n\nNote: Persistent volume claims are intended to exist longer than pods. This\nallows persistent disks to be dynamically allocated and preserved across pod\ncreations and deletions.\n\nCreate a service using the public IP address that will send traffic to pods\nwith the label \"run=prometheus-server\":\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluser-name\u003e/services\n\n## Federation deployment\n\nThe cluster and federation deployments are mutually exclusive. Do not deploy\nboth to the same cluster.\n\nThe federation deployment is a super-set of the prometheus cluster deployment.\nIt is designed to monitor the local cluster as well as aggregate metrics from\nother prometheus clusters.\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/prometheus.yml\n\n## Check deployment\n\nAfter completing the above steps, you can view the status of all objects using\nsomething like:\n\n    kubectl get services,deployments,pods,configmaps,secrets,pvc,pv,events\n\n`kubectl get` is your friend. See also `kubectl describe` for even more details.\n\n[cluster]: https://cloud.google.com/container-engine/docs/clusters/operations\n[setup]: #grafana-setup\n[daemonset]: https://kubernetes.io/docs/admin/daemons/\n\n# Custom Targets\n\nWith the M-Lab configuration, we can add or remove two kinds of targets at\nruntime: legacy and federation targets.\n\nUsing the file service discovery configuration, we create a JSON or YAML input\nfile in the [correct form][file_sd_config], and copy the file into the pod\nfilesystem.\n\nFor example:\n\n```\n[\n    {\n        \"labels\": {\n            \"service\": \"sidestream\"\n        },\n        \"targets\": [\n            \"npad.iupui.mlab4.mia03.measurement-lab.org:9090\"\n        ]\n    }\n]\n```\n\nCopy file(s) to the correct directory in the prometheus pod.\n\n    DIRECTORY=/legacy-targets\n    podname=$( kubectl get pods -o name --selector='run=prometheus-server' )\n    kubectl cp \u003cfilename.json\u003e ${podname##*/}:${DIRECTORY}\n\nTo look at the available files the target directory:\n\n    kubectl exec -c prometheus-server -t ${podname##*/} -- /bin/ls -l ${DIRECTORY}\n\nWithin five minutes, any file ending with `*.json` or `*.yaml` will be scanned\nand the new targets should be reported by the prometheus server.\n\n[file_sd_config]: https://prometheus.io/docs/operating/configuration/#file_sd_config\n\n## Legacy Targets\n\nFor legacy targets (e.g. sidestream), copy the JSON file into the prometheus\ncontainer under the `/legacy-targets` directory.\n\nIn the Prometheus server, targets are listed under:\n\n* Status -\u003e Targets -\u003e \"legacy-targets\"\n\n## Federation Targets\n\nFor federation targets (i.e. other prometheus services), copy the file to the\nprometheus container under the `/federation-targets` directory.\n\nIn the Prometheus server, targets are listed under:\n\n* Status -\u003e Targets -\u003e \"federation-targets\"\n\n# Delete deployment\n\nDelete the prometheus deployment:\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/prometheus.yml\n\nSince the prometheus pod is no longer running, clients connecting to the public\nIP address will try to load but fail. If we also delete the service, then\ntraffic will stop being forwarded from the public IP altogether.\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster name\u003e/services\n\nEven if the prometheus deployment is not running, the persistent volume keeps\nthe data around. If the cluster is destroyed or if the persistent volume claim\nis deleted, the automatically created disk image will be garbage collected and\ndeleted. At that point all data will be lost.\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster-name\u003e/persistentvolumes/persistent-volumes.yml\n\nDelete the storage class.\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster-name\u003e/persistentvolumes/storage-class.yml\n\nConfigMaps are managed explicitly for now:\n\n    kubectl delete configmap prometheus-cluster-config\n    kubectl delete configmap prometheus-federation-config\n\nNow, `kubectl get` should not include any of the above objects.\n\n    kubectl get services,deployments,pods,configmaps,pvc,pv\n\n# Grafana\n\n## Create\n\nCreate ConfigMaps for grafana:\n\n    kubectl create configmap grafana-config \\\n        --from-file=config/federation/grafana\n    kubectl create configmap grafana-env \\\n        --from-literal=domain=\u003cstatic ip address allocated from GCP console\u003e\n\nNote: the domain may be the public IP address, if there is no DNS name yet.\n\nCreate a secret to contain the Grafana admin password:\n\n    kubectl create secret generic grafana-secrets \\\n        --from-literal=admin-password=[redacted text]\n\nFinally, Create the grafana deployment. Like the prometheus deployment, this\nstep starts the grafana server.\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/grafana.yml\n\n## Setup\n\n### Deleting provisioned datasources\n\nDatasources are provisioned through YAML files located in\n`config/federation/grafana/provisioning/datasources/`. Currently, Grafana does\nnot support having a provisioned datasource deleted automatically from the\ndatabase once its YAML file goes away.  The official way to delete a\nprovisioned datasource is to [add a `deleteDatasources` section]\n(\u003chttps://grafana.com/docs/administration/provisioning/#datasources\u003e) to the\nprovisioning YAML. However, this is cumbersome and awkward and somewhat\npollutes the git history. There is another way that is fairly manual, but not\nhard.\n\nIn the configuration section of the Grafana Web interface, there is a section\nfor creating API keys. Create a temporary key that will expire after, say, 1d.\nThen use that key to run a manual API request like the following, being sure to\nreplace `\u003capi-key\u003e` with your API key, `\u003cdatasource-id\u003e` with the ID of the datasource,\n`\u003cdatasource-name\u003e` with the name of the datasource, and `\u003cdatasource-id\u003e` in\nthe API URL to match the datasource ID. Don't forget to also make sure that the\nAPI URL is prointing to the GCP project you intend. This command will flag the\ndatasource as editable:\n\n```\ncurl --request PUT --header \"Authorization: Bearer \u003capi-key\u003e\" \\\n    --header \"Content-Type: application/json\" \\\n    --data '{\"id\": \u003cdatasource-id\u003e, \"name\": \"\u003cdatasource-name\u003e\", \\\n             \"type\": \"prometheus\", \"access\": \"server\", \\\n             \"editable\": \"true\"}' \\\n    https://grafana.mlab-sandbox.measurementlab.net/api/datasources/\u003cdatasource-id\u003e\n```\n\nNow you can delete the datasource with something like the following, replacing\n`\u003capi-key\u003e` with your API key, and changing the API URL to match the ID of the\ndatasource you want to delete:\n\n```\ncurl --request DELETE --header \"Authorization: Bearer \u003capi-key\u003e\" \\\n    https://grafana.mlab-sandbox.measurementlab.net/api/datasources/\u003cdatasource-id\u003e\n```\n\n## Delete\n\nDelete the grafana deployment.\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/grafana.yml\n\nDelete the grafana configmaps:\n\n    kubectl delete configmap grafana-config\n    kubectl delete configmap grafana-env\n\nDelete the grafana secrets:\n\n    kubectl delete secret grafana-secrets\n\n# Alertmanager\n\nThe alertmanager configuration depends on the shared service and persistent\nvolume configurations. If those are already loaded, then continue to the steps\nbelow.\n\nAlso, note: the alertmanager service will do nothing unless the prometheus\ndeployment includes the command line argument that directs alerts to this\ninstance of the alertmanager. i.e. `-alertmanager.url=http://bobloblaw.com:9093`\n\n## Create\n\nIf you're setting up the alertmanager for the first time, copy\n`config.yml.template` to create `config.yml` and update the `api_uri` entries\nwith real values. See the template comments for how to do that.\n\nCreate the configmaps for alertmanager:\n\n    kubectl create configmap alertmanager-config \\\n        --from-file=config/federation/alertmanager\n\n    kubectl create configmap alertmanager-env \\\n        --from-literal=external-url=http://\u003cpublic ip address\u003e:9093\n\nCreate the alertmanager deployment.\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/alertmanager.yml\n\n## Delete\n\nDelete the deployment.\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/alertmanager.yml\n\nDelete the configmaps.\n\n    kubectl delete configmap alertmanager-config\n    kubectl delete configmap alertmanage-env\n\n# Github Receiver\n\nThe github receiver implements the Alertmanager Webhook API. So, Alertmanager\ncan send alerts to the github receiver and they are converted into Github\nissues.\n\nThe github receiver authenticates using Github access tokens. Generate a new one\nat: \u003chttps://github.com/settings/tokens\u003e\n\nActions authenticated using the token will be associated with your account.\n\nNote: only one github receiver should target a given github repository.\n\n## Create\n\nCreate the secrets for the github receiver:\n\n    kubectl create secret generic github-secrets \\\n        --from-literal=auth-token=${AUTH_TOKEN}\n\nCreate the service and deployment:\n\n    kubectl create -f k8s/mlab-oti/\u003ccluster-name\u003e/services/github-receiver-service.yml\n    kubectl create -f k8s/mlab-oti/\u003ccluster-name\u003e/deployments/github-receiver.yml\n\n## Delete\n\nDelete the service and deployment.\n\n    kubectl delete -f k8s/mlab-oti/\u003ccluster-name\u003e/services/github-receiver-service.yml\n    kubectl delete -f k8s/mlab-oti/\u003ccluster-name\u003e/deployments/github-receiver.yml\n\nDelete the secrets:\n\n    kubectl delete secret github-secrets\n\n# Blackbox exporter\n\nThe blackbox exporter allows probes of endpoints over HTTP, HTTPS, DNS, TCP and\nICMP.\n\nTargets are declared in an input file like legacy targets, however\nblackbox targets require specifying a \"module\". The module name must match a\nmodule defined in `config/federation/blackbox/config.yml`. Different modules\nhave different format requirements on the targets (e.g. some with ports, some\nwithout).\n\n```\n[\n    {\n        \"labels\": {\n            \"module\": \"ssh_v4_online\",\n            \"service\": \"sshalt\"\n        },\n        \"targets\": [\n            \"mlab1.den02.measurement-lab.org:806\",\n            \"mlab4.den02.measurement-lab.org:806\"\n        ]\n    }\n]\n```\n\nA single input file can define a list of configurations, each with a different\n\"module\" and independent list of targets.\n\n## Create\n\nCreate the configmaps for the blackbox exporter:\n\n    kubectl create configmap blackbox-config \\\n        --from-file=config/federation/blackbox\n\nCreate the blackbox exporter deployment.\n\n    kubectl create -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/blackbox.yml\n\n## Delete\n\nDelete the deployment.\n\n    kubectl delete -f k8s/mlab-sandbox/\u003ccluster-name\u003e/deployments/blackbox.yml\n\nDelete the configmaps.\n\n    kubectl delete configmap blackbox-config\n\n## Static outbound IP pinning\n\nThere are currently two services running in the cluster which require that\nall outbound requests come from a static source IP: reboot-api and\nscript-exporter. The reboot-api needs a static outbound IP address because it\nmay need to reboot a node via the iDRAC and a firewall rule on the switch\nprevents access to the DRACs, except from a few permitted IPs. The\nscript-exporter needs a static outbound IP because one of the things it does\nis to run e2e NDT tests, and we need a predetermined source IP address so that\ninternal, e2e testing can be easily removed from the data we publish.\n\nWe're going to create a two-node pool, make sure only services that need\nstatic outbound IPs run on it, and then use a service called\n[kubeIP](https://github.com/doitintl/kubeip) to assign static IPs to its\nnodes.\n\n### Create pool\n\nCreate a two-node pool with a specific label (to make sure that any necessary\npods can be selected onto it) and taint (to make sure other services are not\nrun on it). The machine type will differ depending on project as follows:\n\n* mlab-sandbox: n2-standard-8\n* mlab-staging: n2-highcpu-16\n* mlab-oti: n2-highcpu-48\n\n```\ngcloud container node-pools create static-outbound-ip \\\n--cluster=prometheus-federation --machine-type=\u003ctype\u003e --num-nodes=2 \\\n--node-labels=outbound-ip=static --node-taints=outbound-ip=static:NoSchedule \\\n--scopes datastore --zone \u003cprometheus-federation-zone\u003e --project \u003cproject\u003e\n```\n\n### Create kubeIP credentials\n\nCreate gcloud service-account for kubeIP:\n\n```\ngcloud iam service-accounts create kubeip-service-account --display-name \"kubeIP\"\n```\n\nCreate role and attach the service-account to it:\n\n```\ngcloud --project ${GCLOUD_PROJECT} \\\n    iam roles create kubeip --file config/federation/kubeip/roles.yml\ngcloud --project ${GCLOUD_PROJECT} \\\n    projects add-iam-policy-binding $GCLOUD_PROJECT \\\n    --member serviceAccount:kubeip-service-account@$GCLOUD_PROJECT.iam.gserviceaccount.com \\\n    --role projects/$GCLOUD_PROJECT/roles/kubeip\n```\n\nGrab a key file for the service-account:\n\n```\ngcloud --project ${GCLOUD_PROJECT} \\\n    iam service-accounts keys create key.json --iam-account \\\n    kubeip-service-account@${GCLOUD_PROJECT}.iam.gserviceaccount.com\n```\n\nAnd then turn it into a Kubernetes secret, after making sure the right project\nis selected for kubectl:\n\n```\ngcloud container clusters get-credentials prometheus-federation `\n    --zone us-central1-a --project ${GCLOUD_PROJECT}\n\nkubectl create secret generic kubeip-key --from-file=key.json\n```\n\nFinally reserve static IPs for your new nodepool:\n\n```\nfor i in {1..2}; do\n    gcloud --project ${GCLOUD_PROJECT} \\\n        compute addresses create kubeip-ip$i --region=us-central1\ndone\n```\n\nAnd label them for kubeIP (use the same region as the GKE cluster). The\nlabel `kubeip=prometheus-federation` corresponds to the `KUBEIP_LABELKEY`\nand `KUBEIP_LABELVALUE` values in the kubeIP deployment:\n\n```\nfor i in {1..2}; do\n    gcloud --project ${GCLOUD_PROJECT} \\\n        beta compute addresses update kubeip-ip$i \\\n        --update-labels kubeip=prometheus-federation --region=us-central1\ndone\n```\n\nThen continue your deployment as normal.\n\n# Debugging the steps above\n\n## Public IP appears to hang\n\nAfter `kubectl get service prometheus-server` assigns a public IP, you can visit\nthe service at that IP, e.g. \u003chttp://[public-ip\u003e]:9090. If the service appears to\nhang, the docker instance may have failed to start.\n\nCheck using:\n\n```\n$ kubectl get pods\nNAME                               READY     STATUS             RESTARTS   AGE\nprometheus-server-2562116152-9mdrq   0/1       CrashLoopBackOff   9          25m\n```\n\nIn this case, the `READY` status is \"0\", meaning not yet ready. And, the\n`STATUS` gives a clue that the docker instance is crashing immediately after\nstart.\n\n## Viewing logs\n\nIf a docker instance is misbehaving, we can view the logs reported by that\ninstance using `kubectl logs`. For example, in the case above, we can ask for\nthe logs with the pod name.\n\n```\n$ kubectl logs prometheus-server-2562116152-9mdrq\ntime=\"2017-02-01T21:02:36Z\" level=info msg=\"Starting prometheus (version=1.5.0, branch=master, revision=d840f2c400629a846b210cf58d65b9fbae0f1d5c)\" source=\"main.go:75\"\ntime=\"2017-02-01T21:02:36Z\" level=info msg=\"Build context (go=go1.7.4, user=root@a04ed5b536e3, date=20170123-13:56:24)\" source=\"main.go:76\"\ntime=\"2017-02-01T21:02:36Z\" level=info msg=\"Loading configuration file /etc/prometheus/prometheus.yml\" source=\"main.go:248\"\ntime=\"2017-02-01T21:02:36Z\" level=error msg=\"Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): yaml: line 2: mapping values are not allowed in this context\" source=\"main.go:150\"\n```\n\n## Viewing events\n\nKubernetes events are also an good source of recent actions and their status.\nFor example, if we tried to create the grafana deployment before defining the\ngrafana-secrets secret, the pods would fail to start and the events would\ninclude a more specific error message.\n\n```\n$ kubectl get pods,events\nNAME                                    READY     STATUS    RESTARTS   AGE\npo/grafana-server-1476781881-fhr5z      0/1       RunContainerError   0          6m\n\nLASTSEEN   FIRSTSEEN   COUNT     NAME                                 KIND      SUBOBJECT                         TYPE      REASON       SOURCE                                                    MESSAGE\n55s        7m          29        ev/grafana-server-1476781881-fhr5z   Pod                                         Warning   FailedSync   {kubelet gke-soltesz-test-2-default-pool-7151d92d-7dd5}   Error syncing pod, skipping: failed to \"StartContainer\" for \"grafana-server\" with RunContainerError: \"GenerateRunContainerOptions: secrets \\\"grafana-secrets\\\" not found\"\n```\n\n# Google Cloud Build\n\nThe above information describes how to create and bootstrap the clusters. Once\nthe clusters already exists, automated builds in Cloud Build update and modify\nthe running clusters. Pushes to various branches and tagging the repository will\ntrigger these Cloud Builds.\n\nThe builds rely a quite a number of secrets (not kubernetes Secrets) and\ncredentials. This sensitive information is stored in GCP's Secret Manager. The\nsecrets are named like `prometheus-federation-build-\u003cname\u003e`. The secrets have\nalready been created, and the builds configured to use them.\n\nOccasionally, it may be necessary to update a secret value. This can be done\nusing `gcloud`. First, write the secret to a file in your local development\nenvironment. *Be careful*, that the file does not end in a newline character,\nas some of the secrets won't work with a newline there. Once you have the file,\nyou can update the secret with a command like this:\n\n```sh\ngcloud secrets versions add prometheus-federation-build-\u003cname\u003e \\\n  --data-file=\u003csecret file\u003e\n  --project=\u003cproject\u003e\n```\n\nAlternatively, you can pipe the secret value in with something like this:\n\n```sh\necho -n \"\u003csecret\u003e\" | gcloud secrets versions add prometheus-federation-build-\u003cname\u003e \\\n  --data-file=-\n  --project=\u003cproject\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fprometheus-support","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fprometheus-support","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fprometheus-support/lists"}