{"id":18678158,"url":"https://github.com/srl-labs/srl-telemetry-lab","last_synced_at":"2025-04-04T11:07:39.910Z","repository":{"id":38081433,"uuid":"327666729","full_name":"srl-labs/srl-telemetry-lab","owner":"srl-labs","description":"Interactive Streaming Telemetry lab with Nokia SR Linux nodes forming a Clos topology","archived":false,"fork":false,"pushed_at":"2025-01-03T12:35:59.000Z","size":24407,"stargazers_count":162,"open_issues_count":0,"forks_count":42,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-03-28T10:03:09.243Z","etag":null,"topics":["clab-topo","codespaces","gnmi","streaming-telemetry"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/srl-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-01-07T16:28:54.000Z","updated_at":"2025-03-20T08:47:20.000Z","dependencies_parsed_at":"2023-11-10T12:07:36.769Z","dependency_job_id":null,"html_url":"https://github.com/srl-labs/srl-telemetry-lab","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Fsrl-telemetry-lab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Fsrl-telemetry-lab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Fsrl-telemetry-lab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Fsrl-telemetry-lab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/srl-labs","download_url":"https://codeload.github.com/srl-labs/srl-telemetry-lab/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166148,"owners_count":20894652,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clab-topo","codespaces","gnmi","streaming-telemetry"],"created_at":"2024-11-07T09:36:13.216Z","updated_at":"2025-04-04T11:07:39.891Z","avatar_url":"https://github.com/srl-labs.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nokia SR Linux Streaming Telemetry Lab\n\n[![Discord][discord-svg]][discord-url] [![DevPod][devpod-svg]][devpod-url] [![Codespaces][codespaces-svg]][codespaces-url]  \n![w212][w212][Learn more](https://containerlab.dev/macos/#devpod) ![w90][w90][Learn more](https://containerlab.dev/manual/codespaces)\n\n[discord-svg]: https://gitlab.com/rdodin/pics/-/wikis/uploads/b822984bc95d77ba92d50109c66c7afe/join-discord-btn.svg\n[discord-url]: https://discord.gg/tZvgjQ6PZf\n[devpod-svg]: https://gitlab.com/rdodin/pics/-/wikis/uploads/dfc36636ecaa60f3e70340686d5800db/open-in-devpod-btn.svg\n[devpod-url]: https://devpod.sh/open#https://github.com/srl-labs/srl-telemetry-lab\n[codespaces-svg]: https://gitlab.com/rdodin/pics/-/wikis/uploads/80546a8c7cda8bb14aa799d26f55bd83/run-codespaces-btn.svg\n[codespaces-url]: https://codespaces.new/srl-labs/srl-telemetry-lab?quickstart=1\u0026devcontainer_path=.devcontainer%2Fdocker-in-docker%2Fdevcontainer.json\n[w212]: https://gitlab.com/rdodin/pics/-/wikis/uploads/718a32dfa2b375cb07bcac50ae32964a/w212h1.svg\n[w90]: https://gitlab.com/rdodin/pics/-/wikis/uploads/bf1b8ea28b4528eb1b66567355a13c5c/w90h1.svg\n\nSR Linux has first-class Streaming Telemetry support thanks to [100% YANG coverage](https://learn.srlinux.dev/yang/) of state and config data. The holistic coverage enables SR Linux users to stream **any** data off of the NOS with on-change, sample, or target-defined support. A discrepancy in visibility across APIs is not about SR Linux.\n\nThis lab represents a small Clos fabric with [Nokia SR Linux](https://learn.srlinux.dev/) switches running as containers. The lab topology consists of a Clos topology, plus a Streaming Telemetry stack comprised of [gnmic](https://gnmic.openconfig.net), prometheus and grafana applications.\n\n---\n\n![pic1](https://gitlab.com/rdodin/pics/-/wikis/uploads/0784c31d48ec18fd24111ad8d73478b0/image.png)\n\nIn addition to the telemetry stack, the lab also includes a modern logging stack comprised of [promtail](https://grafana.com/docs/loki/latest/clients/promtail/) and [loki](https://grafana.com/oss/loki/).\n\nGoals of this lab:\n\n1. Demonstrate how a telemetry stack can be incorporated into the containerlab topology file.\n2. Explain SR Linux holistic telemetry support.\n3. Provide practical configuration examples for the gnmic collector to subscribe to fabric nodes and export metrics to Prometheus TSDB.\n4. Introduce advanced Grafana dashboarding with [FlowPlugin](https://grafana.com/grafana/plugins/andrewbmchugh-flow-panel/) plugin rendering port speeds and statuses.\n5. Give a sneak peek of the modern logging telemetry stack with Loki and Promtail to consume Syslog data from SR Linux nodes.\n\n## Deploying the lab\n\nThe lab is deployed with the [containerlab](https://containerlab.dev) project, where [`st.clab.yml`](st.clab.yml) file declaratively describes the lab topology.\n\n```bash\n# change into the cloned directory\n# and execute\ncontainerlab deploy --reconfigure\n```\n\nTo remove the lab:\n\n```bash\ncontainerlab destroy --cleanup\n```\n\n## Accessing the network elements\n\nOnce the lab has been deployed, the different SR Linux nodes can be accessed via SSH through their management IP address, given in the summary displayed after the execution of the deploy command. It is also possible to reach those nodes directly via their hostname, defined in the topology file. Linux clients cannot be reached via SSH, as it is not enabled, but it is possible to connect to them with a docker exec command.\n\n```bash\n# reach a SR Linux leaf or a spine via SSH\nssh admin@leaf1\nssh admin@spine1\n\n# reach a Linux client via Docker\ndocker exec -it client1 bash\n```\n\n## Fabric configuration\n\nThe DC fabric used in this lab consists of three leaves and two spines interconnected as shown in the diagram.\n\n![pic](https://gitlab.com/rdodin/pics/-/wikis/uploads/14c768a04fc30e09b0bf5cf0b57b5b63/image.png)\n\nLeaves and spines use Nokia SR Linux IXR-D2 and IXR-D3L chassis respectively. Each network element of this topology is equipped with a [startup configuration file](configs/fabric/) that is applied at the node's startup.\n\nOnce booted, network nodes will come up with interfaces, underlay protocols and overlay service configured. The fabric is running Layer 2 EVPN service between the leaves.\n\n### Verifying the underlay and overlay status\n\nThe underlay network runs eBGP, while iBGP is used for the overlay network. The Layer 2 EVPN service is configured as explained in this comprehensive tutorial: [L2EVPN on Nokia SR Linux](https://learn.srlinux.dev/tutorials/l2evpn/intro/).\n\nBy connecting via SSH to one of the leaves, we can verify the status of those BGP sessions.\n\n```\nA:leaf1# show network-instance default protocols bgp neighbor\n------------------------------------------------------------------------------------------------------------------\nBGP neighbor summary for network-instance \"default\"\nFlags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow\n\n+-----------+---------------+---------------+-------+----------+-------------+--------------+--------------+---------------+\n| Net-Inst  |     Peer      |     Group     | Flags | Peer-AS  |   State     |    Uptime    |   AFI/SAFI   | Rx/Active/Tx] |\n+===========+===============+===============+=======+==========+=============+==============+==============+===============+\n| default   | 10.0.2.1      | iBGP-overlay  | S     | 100      | established | 0d:0h:0m:27s | evpn         | [4/4/2]       |\n| default   | 10.0.2.2      | iBGP-overlay  | S     | 100      | established | 0d:0h:0m:28s | evpn         | [4/0/2]       |\n| default   | 192.168.11.1  | eBGP          | S     | 201      | established | 0d:0h:0m:34s | ipv4-unicast | [3/3/2]       |\n| default   | 192.168.12.1  | eBGP          | S     | 202      | established | 0d:0h:0m:33s | ipv4-unicast | [3/3/4]       |\n+-----------+---------------+---------------+-------+----------+-------------+--------------+--------------+---------------+\n```\n\n## Telemetry stack\n\nAs the lab name suggests, telemetry is at its core. The following telemetry stack is used in this lab:\n\n| Role                | Software                              |\n| ------------------- | ------------------------------------- |\n| Telemetry collector | [gnmic](https://gnmic.openconfig.net) |\n| Time-Series DB      | [prometheus](https://prometheus.io)   |\n| Visualization       | [grafana](https://grafana.com)        |\n\n### gnmic\n\n[gnmic](https://gnmic.openconfig.net) is an Openconfig project that allows to subscribe to streaming telemetry data from network devices and export it to a variety of destinations. In this lab, gnmic is used to subscribe to the telemetry data from the fabric nodes and export it to the prometheus time-series database.\n\nThe gnmic configuration file - [gnmic-config.yml](gnmic-config.yml) - is applied to the gnmic container at the startup and instructs it to subscribe to the telemetry data and export it to the prometheus time-series database.\n\n### Prometheus\n\n[Prometheus](https://prometheus.io) is a popular open-source time-series database. It is used in this lab to store the telemetry data exported by gnmic. The prometheus configuration file - [configs/prometheus/prometheus.yml](configs/prometheus/prometheus.yml) - has a minimal configuration and instructs prometheus to scrape the data from the gnmic collector with a 5s interval.\n\n### Grafana\n\nGrafana is another key component of this lab as it provides the visualisation for the collected telemetry data. Lab's topology file includes grafana node and configuration parameters such as dashboards, datasources and required plugins.\n\nGrafana dashboard provided by this repository provides multiple views on the collected real-time data. Powered by [flow plugin](https://grafana.com/grafana/plugins/andrewbmchugh-flow-panel/) it overlays telemetry sourced data over graphics such as topology and front panel views:\n\n![pic3](https://gitlab.com/rdodin/pics/-/wikis/uploads/919092da83782779b960eeb4b893fb4a/image.png)\n\nUsing the flow plugin and real telemetry data users can create interactive topology maps (aka weathermap) with a visual indication of link rate/utilization.\n\n![pic2](https://gitlab.com/rdodin/pics/-/wikis/uploads/12f154dafca1270f7a1628c1ed3ab77a/image.png)\n\nThe panels for the flow plugin has been autocreated by the [clab-io-draw](https://github.com/srl-labs/clab-io-draw) tool\n\n### Access details\n\nUsing containerlab's ability to expose ports of the containers to the host, the following services are available on the host machine:\n\n* Grafana: \u003chttp://localhost:3000\u003e. Anonymous access is enabled; no credentials are required. If you want to act as an admin, use `admin/admin` credentials.\n* Prometheus: \u003chttp://localhost:9090/graph\u003e\n\n## Traffic generation\n\nWhen the lab is started, there is not traffic running between the nodes as the clients are sending any data. To run traffic between the nodes, leverage `traffic.sh` control script.\n\nTo start the traffic:\n\n* `bash traffic.sh start all` - start traffic between all nodes\n* `bash traffic.sh start 1-2` - start traffic between client1 and client2\n* `bash traffic.sh start 1-3` - start traffic between client1 and client3\n\nTo stop the traffic:\n\n* `bash traffic.sh stop` - stop traffic generation between all nodes\n* `bash traffic.sh stop 1-2` - stop traffic generation between client1 and client2\n* `bash traffic.sh stop 1-3` - stop traffic generation between client1 and client3\n\nAs a result, the traffic will be generated between the clients and the traffic rate will be reflected on the grafana dashboard.\n\n\u003chttps://github.com/srl-labs/srl-telemetry-lab/assets/5679861/158914fc-9100-416b-8b0f-cde932895cec\u003e\n\n## Logging stack\n\nThe logging stack leverages the promtail-\u003eLoki pipeline, where promtail is a log agent that extracts, transforms and ships logs to Loki, a log aggregation system.\n\nThe logging infrastructure logs every message from SR Linux that is above Info level. This includes all the BGP messages, all the system messages, all the interface state changes, etc. The dashboard provides a view on the collected logs and allows filtering on a per-application level.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrl-labs%2Fsrl-telemetry-lab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrl-labs%2Fsrl-telemetry-lab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrl-labs%2Fsrl-telemetry-lab/lists"}