{"id":30703216,"url":"https://github.com/dfinity/ic-observability-stack","last_synced_at":"2026-06-23T16:31:59.578Z","repository":{"id":241134764,"uuid":"764191837","full_name":"dfinity/ic-observability-stack","owner":"dfinity","description":"Observe the health and operation of the Internet Computer using your own equipment.","archived":false,"fork":false,"pushed_at":"2025-11-24T14:13:00.000Z","size":814,"stargazers_count":3,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-16T08:29:35.020Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dfinity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-02-27T16:35:17.000Z","updated_at":"2025-11-24T12:25:54.000Z","dependencies_parsed_at":"2024-05-22T16:29:06.071Z","dependency_job_id":"a6462302-c84d-4ece-8bb5-02236741bab1","html_url":"https://github.com/dfinity/ic-observability-stack","commit_stats":null,"previous_names":["dfinity/ic-observability-stack"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dfinity/ic-observability-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfinity%2Fic-observability-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfinity%2Fic-observability-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfinity%2Fic-observability-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfinity%2Fic-observability-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dfinity","download_url":"https://codeload.github.com/dfinity/ic-observability-stack/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfinity%2Fic-observability-stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34698687,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-02T16:57:24.994Z","updated_at":"2026-06-23T16:31:59.568Z","avatar_url":"https://github.com/dfinity.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IC observability stack\n\nThis collection of software lets you use your own equipment to observe the health and \noperation of the nodes in your datacenter that are part of the Internet Computer. It\nis a collection of docker containers that, when deployed and configured, will collect\nmetrics about the performance of selected nodes and send alerts if something is off.\n\n## How does this work?\n\nThe stack selects IC nodes and collects metrics from each node, saving them\nin a local Prometheus database every few seconds.  This database is\nqueryable through a Grafana deployed side-to-side with Prometheus.\n\nNodes of the Internet Computer all make available to the public a series\nof metrics (in [Prometheus text format](https://github.com/Prometheus/docs/blob/777846211d502a287ab2b304cb515dc779de3474/content/docs/instrumenting/exposition_formats.md#text-based-format))\nthat can be collected and analyzed by software such as this.\n\nThe stack has to be deployed within the same *network subnet* as the nodes so that \nthe scraping of the nodes can take place. This practically means that the nodes and the \nmachine that is running this stack have to be within the same data center and connected\nto the same router.\n\n## Prerequisites\n\n* IPv6 connectivity.\n* Root-equivalent access on your workstation, to deploy the software this stack \n  needs to be set up.\n* Hardware.\n  * It is recommended to run this stack on a machine with at least 16GB ram and \n    80-100GB storage.\n* SSH access to the machine.\n  * This is needed to later remotely connect to the services and inspect issues.\n  * Setting up ssh server [guide](https://documentation.ubuntu.com/server/how-to/security/openssh-server/).\n\n## Preparation\n\nTo start preparing your workstation for scraping and observing your nodes you have to \n`ssh` into the machine (or be physically next to it in order to run commands on the\nmachine).\n\nInstall [Docker](https://docs.docker.com/engine/install/) for your machine. There\nshouldn't be visible differences across different operating systems. This setup has\nbeen tested on Mac OS and Manjaro OS (Linux flavour) but hasn't been tested on \nWindows machines.\n\nOnce you can run docker commands and have `docker compose` you can proceed to the next step.\n\n## Configuration\n\nNow that you have `docker` you can proceed to configure the stack. \n\n### Ensuring proper user and group\n\nTo properly configure the user and group which will be used for all docker workloads\nrun the following setup script:\n```bash\n# NOTE: you have to be within the same directory as this README!\n./setup.sh\n```\n\n### Scraping targets\n\nFirst thing that you have to configure is your scraping targets. To do that, find your \n[principal id](https://support.dfinity.org/hc/en-us/articles/7365913875988-What-is-a-principal) \nthat correclates to your node provider id. You can find that from the [public dashboard](https://dashboard.internetcomputer.org/network/providers). After that you should find the id of the data center to which\nyou are deploying this stack. You can find that information also on the [public dashboard](https://dashboard.internetcomputer.org/network/providers).\n\nWith that you can run the following command:\n```bash\n# NOTE: you have to be within the same directory as this README!\ndocker compose -f ./docker-compose.tools.yaml run --rm prom-config-builder tools/prom-config-builder/prom_config_builder.py --node-provider-id \u003cnode-provider-id\u003e --dc-id \u003cdc-id\u003e\n```\n\nExample command for node provider Dfinity Stiftung for data center se1:\n```bash\ndocker compose -f ./docker-compose.tools.yaml run --rm prom-config-builder tools/prom-config-builder/prom_config_builder.py --node-provider-id bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe --dc-id se1\n```\n\nOnce that executes, you should be able to see a new file at `./config/prometheus/config.yaml`. \nThis file contains the definitions for the scraping targets. It will be slightly different \nfor each node provider and each data center. It is not versione controlled and you can \nalways recreate it with running the above command if you lose it, or delete it.\n\n### Contact points\n\nThis stack uses [Grafana](https://grafana.com/) to present the dashboards and to send\nalerts. Sending alerts is done using the [contact points](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/contact-points/) which need to be configured. Grafana supports\nvarious contact points, some of them can be used for free, some of them are paid. \nThis stack was tested with the following contact points:\n* Discord [free]\n* Slack\n* Google chat\n\nTo setup your prefered contact point copy over the template with the following command:\n```bash\ncp ./config/grafana/templates/template_contact_points.yaml ./config/grafana/provisioning/alerting/contact_points.yaml\n```\n\nAfter that, edit the new file on path `./config/grafana/provisioning/alerting/contact_points.yaml`\nand uncomment your prefered contact point and configure it (to _uncomment_ you should\nremove the initial `#` from the contact point definition.\n\n## Running the stack\n\nTo deploy the stack run the following command:\n```bash\n# This will spawn the containers \ndocker compose -f ./docker-compose.yaml up -d \n```\n\nIt will take some time for the services to start and to sync between one another. \nYou can monitor if containers are failing by running `docker ps` and see if \nthere are some restarts happening in the containers. To see more about Troubleshooting\nread *Troubleshooting*\n\n## Usage\n\nOnce started, you will see the following applications:\n* Prometheus - http://localhost:9090\n* Grafana - http://localhost:3000 - default creds can be see in `./config/grafana/grafana.ini` \n* Service discovery - http://localhost:8000\n\nAfter 5-10 minutes you should see targets discovered in victoria on the [targets page](http://localhost:9090/targets).\nInitially, they might apear in red and if you keep monitoring they should\nslowly start getting green, which means that the targets are successfuly scraped.\n\nYou should also see some data incoming in the grafana [sample dashboard for the \nnode exporter](http://localhost:3000/d/1/node-exporter?orgId=1\u0026from=now-3h\u0026to=now\u0026timezone=utc\u0026var-datasource=prometheus\u0026var-instance=$__all\u0026var-diskdevice=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B).\nHere, you can see various information about the performance of nodes and their health.\n\n[Alerting tab](http://localhost:3000/alerting/list) will show `obs alert evaluations` \nwhich will contain a list of preconfigured alerts. Here you can see if any of the \nalerts are in in a problematic state. Most of the time they should be in `Normal` \nstate, this means that everything is fine. Some of them may occasionally go into a \n`Pending` state, which means that they crossed the threshold for an alert but still\nisn't happening for long enough to consider this a problem. When a `Pending` alert\nis happening for long enough it will go into `Error` mode and you should receive an\nalert on your preconfigured contact point. \n\n*NOTE*: Not all alerts are configured to send notifications if they fire, usually\nbecause they are just warnings and cannot be acted upon. To see which are and \nwhich aren't you can see `./config/grafana/provisioning/alerting/alerts.yaml` and\nsee which have the lavel `severity: critical` attached to them because only they\nwill send alerts to your contact point. If you wish to send all of them, just replace\nthe other ones that contain `severity: warning` with `severity: critical`.\n\n### Access to the services of the stack\n\nTo access the stack remotely you can do the following:\n```bash\nssh -L 3000:localhost:3000 -L 9090:localhost:9090 -L 8000:localhost:8000 \u003cmachine-with-obs-stack\u003e\n```\n\nExample command with all parameters:\n```bash\nssh -L 3000:localhost:3000 -L 9090:localhost:9090 -i ~/.ssh/priv_key.pem myuser@192.168.15.15\n```\n\n## Extending\n\nExtending this stack usually means adding new alerts or dashboards. Those might\nbe your own modifications or the ones that come from someone else.\n\n### Dashboards\n\nBuilding grafana dashboards is usually done via [grafana ui](http://localhost:3000/dashboard/new?orgId=1\u0026from=now-6h\u0026to=now\u0026timezone=browser). You can follow\n[this tutorial](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/create-dashboard/) from grafana to make your custom dashboards.\n\nAfter creating the dashboard it is suggested to export it and save it\n`./config/grafana/provisioning/dashboards/`. This will make sure that\nif you later need to restore the dashboard or do a fresh deployment \nof grafana it will be persisted.\n\n*NOTE*: Storing a dashboard and restarting can make lead to errors \ndue to overlapping dashboard `uids`. If you export the dashboard and \nsave it you should, be sure to do a clean deployment of grafana.\n\n### Alerts\n\nAdding grafana alerts can also be done via [grafana ui](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/).\n\nSimilarly to dashboards, it is suggested to export alerts and save them \nin `./config/grafana/provisioning/alerting/` which will make sure that\nthey persist after full grafana redeployments.\n\n## Troubleshooting\n\n### Service discovery failing\n\nIt is possible that usually due to networking issues, service discovery component\nmay fail. To debug why you can run the following command to inspect the logs:\n```bash\n# From the same folder of this README\ndocker compose logs multiservice-discovery\n```\n\nYou may see something along the lines of:\n```bash\nmultiservice-discovery-1  | Nov 10 13:25:07.044 WARN Failed to sync registry for mercury @ interval Instant { tv_sec: 9050, tv_nsec: 707832535 }: SyncWithNnsFailed { failures: [(\"targets\", RegistryTransportError { source: UnknownError(\"Failed to query get_certified_changes_since on canister rwlgt-iiaaa-aaaaa-aaaaa-cai: Request failed for http://[2606:fb40:201:1001:6801:2fff:fef5:b129]:8080/api/v2/canister/rwlgt-iiaaa-aaaaa-aaaaa-cai/query: hyper_util::client::legacy::Error(Connect, ConnectError(\\\"tcp connect error\\\", [2606:fb40:201:1001:6801:2fff:fef5:b129]:8080, Os { code: 101, kind: NetworkUnreachable, message: \\\"Network is unreachable\\\" }))\") })] }\n```\n\nThis is usually a transient error and may happen from time to time. What this \nmeans is that the discovery cannot sync with the registry canister and may be \nserving stale targets. Usually this is acceptable but if that happens when \ndeploying and the initial sync fails you may be unable to see any of your \nnodes. As long as you can see your nodes on the following link you can safely\nignore the transient failures.\n```bash\ncurl http://localhost:8000/prom/targets?node_provider_id=\u003cnode-provider-id\u003e\u0026dc_id=\u003cdc-id\u003e\n```\n\n*NOTE*: The initial sync of service discovery may take up to 15 minutes! Syncing \nwill be clearly logged in the multiservice discovery.\n\n### Victoria\n\n#### No targets visible in targets view\n\nIf you don't see anything in the [prometheus targets view](http://localhost:9090/targets?search=), \nthat means that prometheus failed to receive targets from the service discovery.\n\nTo check the logs run:\n```bash\n# From the same folder of this README\ndocker compose logs prometheus\n```\n\nCheck if you can see your nodes by running the following command:\n```bash\ncurl http://localhost:8000/prom/targets?node_provider_id=\u003cnode-provider-id\u003e\u0026dc_id=\u003cdc-id\u003e\n```\n\n#### Targets visible but are being shown in read\n\nYou should now see 4 jobs:\n* `host_node_exporter`\n* `node_exporter`\n* `orchestrator`\n* `replica`\n\nIf any of them are shown in read it means that some of the targets (or all of them)\nare failing to be scraped. You can see that from the logs as well:\n```bash\n# From the same folder of this README\ndocker compose logs prometheus\n```\n\nThis means that the prometheus scraper cannot reach the nodes it is trying\nto scrape. It can be because the workstation for this observability stack\nisn't in the same network subnet as the nodes, or due to other network \nissues.\n\n### Stack restart\n\nTo make a full clean restart (or partial) you can do the following:\n* Ensure that everything you created through grafana ui is exported\n  * Dashboards (save them into `./config/grafana/provisioning/dashboards/`)\n  * Alerts (save them into `./config/grafana/provisioning/alerting/`)\n  * Contact points (save them into `./config/grafana/provisioning/alerting/`)\n  * Message templates (save them into `./config/grafana/provisioning/alerting/`)\n  * Notification policies (save them into `./config/grafana/provisioning/alerting/`)\n* If you haven't created the resources, or don't mind losing them \n  you can proceed.\n* Stop the stack: `docker compose -f ./docker-compose.yaml down`\n* Clean the volumes. You don't have to clean everything, pick just\n  ones that you wish to restart fully:\n  * prometheus: `rm -rf ./volumes/prometheus/`\n  * grafana: `rm -rf ./volumes/grafana/`\n  * multiservice discovery: `rm -rf ./volumes/msd/`\n* Reset the folder structure: `git checkout -- ./volumes/`\n* Run the stack again: `docker compose -f ./docker-compose.yaml up -d`\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfinity%2Fic-observability-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdfinity%2Fic-observability-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfinity%2Fic-observability-stack/lists"}