{"id":27944591,"url":"https://github.com/azyablov/sros_bng_observability","last_synced_at":"2026-02-16T02:13:54.164Z","repository":{"id":286709834,"uuid":"962297439","full_name":"azyablov/sros_bng_observability","owner":"azyablov","description":"SROS MAG / BNG observability clab","archived":false,"fork":false,"pushed_at":"2025-04-30T16:35:15.000Z","size":2674,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-19T17:58:59.825Z","etag":null,"topics":["aaa","alert","alerting","alertmanager","bng","clab","dhcp","gnmic","grafana","howto","logs","loki","mag","observability","promtail","telemetry","testbed"],"latest_commit_sha":null,"homepage":"https://www.nokia.com","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/azyablov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-08T00:23:12.000Z","updated_at":"2025-04-30T16:35:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"6ce98fcc-f7a0-4de6-ae47-ea636fbb94a8","html_url":"https://github.com/azyablov/sros_bng_observability","commit_stats":null,"previous_names":["azyablov/sros_bng_observability"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/azyablov/sros_bng_observability","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azyablov%2Fsros_bng_observability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azyablov%2Fsros_bng_observability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azyablov%2Fsros_bng_observability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azyablov%2Fsros_bng_observability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/azyablov","download_url":"https://codeload.github.com/azyablov/sros_bng_observability/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azyablov%2Fsros_bng_observability/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29498398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-16T02:07:14.481Z","status":"online","status_checked_at":"2026-02-16T02:03:22.852Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aaa","alert","alerting","alertmanager","bng","clab","dhcp","gnmic","grafana","howto","logs","loki","mag","observability","promtail","telemetry","testbed"],"created_at":"2025-05-07T12:53:09.374Z","updated_at":"2026-02-16T02:13:54.146Z","avatar_url":"https://github.com/azyablov.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# SROS BNG / MAG-Integrated Observability Lab\n\n# TL;DR VS Quick Start\n\nThe goal of the lab is to spin up a full MAG‑Integrated (BNG) observability stack in less than 10 min, so let's not waste time and get started right away. Go straight to the [Quick Start](#quick-start) section 🚀 🚀 🚀 .\nPlease do not forget to check the [Requirements](#requirements) section before you start, it's important!\n\n\n\u003cdetails\u003e\n\u003csummary\u003e📑 Table of Contents\u003c/summary\u003e\n\n- [SROS MAG-Integrated / BNG Observability Lab](#sros-mag-integrated--bng-observability-lab)\n  - [TL;DR VS Quick Start](#tldr-vs-quick-start)\n  - [Introduction](#introduction)\n    - [Overview](#overview)\n    - [Authors](#authors)\n    - [Software Versions](#software-versions)\n  - [Requirements](#requirements)\n  - [TL;DR Observability for MAG (BNG)](#tldr-observability-for-mag-bng)\n    - [Observability vs Monitoring](#observability-vs-monitoring)\n    - [What Should I Collect?](#what-should-i-collect)\n    - [Why do we need it?](#why-do-we-need-it)\n    - [The three pillars of Observability](#the-three-pillars-of-observability)\n      - [Logs](#logs)\n      - [Metrics and State](#metrics-and-state)\n      - [Traces](#traces)\n    - [Alerting pillars of BNG](#alerting-pillars-of-bng)\n    - [MAG (BNG) Observability HOWTO](#mag-bng-observability-howto)\n      - [Observability Stack](#observability-stack)\n      - [What and Why to collect?](#what-and-why-to-collect)\n  - [MAG (BNG) Setup](#mag-bng-setup)\n    - [Key Components](#key-components)\n  - [Quick Start](#quick-start)\n  - [Dashboards Gallery](#dashboards-gallery)\n\n\u003c/details\u003e\n\n# Introduction\n\n## Overview\n\nThis repository contains `clab` that implements *MAG-Integrated (Multi-Access Gateway) / BNG (Broadband Network Gateway)* test-bed and collects telemetry, events and state data from a *BNG*, aggregation and core network elements via *SROS MDM* interfaces and syslog channels. The lab is designed to be used in a lab environment and can be easily deployed using `Containerlab`. \nFor the sake of simplicity, we are going to refer to the *BNG* as a *MAG* in this document, as the term reflecting better the modern nature of this NE Function in a network. \n\nThe lab is designed to be used for demonstration and testing purposes only. It is not intended to be used in production environments.\n\nOur immediate objectives are to show how *MAG*  telemetry data can be collected and visualized using open-source tools such as `Prometheus`, `Grafana`, `Fluent Bit`, `Loki`, `gNMIc`, `Smokeping` and `Alertmanager`. The lab also includes a *BNG Blaster*, which is a traffic generator that can be used to generate traffic on the *MAG*  devices. \nThe goal is to create a simple observability concept solution for *MAG*  devices that can be used in lab environments and testbeds.\nFor the full-blown solution please refer to the official Nokia representatives.\n\n| \u003cimg src=\"pic/BNG_State_and_Redundancy.png\" height=\"220\" width=\"450\"/\u003e | \u003cimg src=\"pic/Subscriber_Inside.png\" height=\"220\" width=\"450\"/\u003e |\n|----------------------------------------|----------------------------------------|\n| \u003cimg src=\"pic/Core_and_Aggregation_Network.png\" height=\"200\" width=\"450\"/\u003e | \u003cimg src=\"pic/BNG_Resources.png\" height=\"200\" width=\"450\"/\u003e |\n\nLet's move step by step via the principle and pillars, since every endeavor should start with a plan and vision behind.\n\n## Authors\n\n[Kivanc Imer](mailto:kivanc.imer@nokia.com) - IP Solution Architect\n\n[Anton Zyablov](mailto:anton.zyablov@nokia.com) - Solution Architects Lead\n\n## Software Versions\n\n| Software    | Version  |\n|-------------|----------|\n| Nokia SROS  | 24.7.R1  |\n| gnmic       | 0.39.1   |\n| bngblaster  | 0.9.17   |\n| fluent-bit  | 3.2.10   |\n| prometheus  | 2.53.4  |\n| grafana     | 11.5.2   |\n| freeradius-server | 3.2.3 |\n| syslog-ng | 4.8.1-r1-ls153 |\n| promtail |  3.4.3 |\n| loki | 3.4.2 |\n| alertmanager | 0.28.1 |\n| smokeping_prober | 0.9.0 |\n\n\n## Requirements\n\nTo run this clab, you need to have the following installed:\n- [Docker](https://docs.docker.com/get-docker/)\n- [Containerlab](https://containerlab.dev/getting-started/installation/)\n\nAdditional licenses and API tokens:\n- *SROS* images (vr-sros should obtained from official Nokia representatives).\n- *SROS* license is required to run the *SROS* containers (should obtained from official Nokia representatives).\n- Optional: *Telegram API Token* is required to send messages to a Telegram channel [Telegram Bots Tutorial](https://core.telegram.org/bots/tutorial).\n\nGo to 🚀 🚀 🚀  [Quick Start](#quick-start) in order to run it!\n\n# TL;DR Observability for MAG (BNG)\n\n## Observability vs Monitoring\n\nNot the same as monitoring (practically collection and visualization), so the aim of this setup is to provide observability of the *MAG*  devices and the network elements connected to it. Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. In other words, observability is the ability to understand what is happening inside a system based on the data that is available from outside the system. Collection of everything creates a mess and makes observation and troubleshooting hard.\n\nOur magic formula would be the following:\n\n**Observability** = **real-time monitoring of relevant metrics and states** + **context-rich dashboards** + **proactive view**. \n\n## What Should I Collect?\n\nThe observability stack should be able to collect the following data:\n- Metrics or Telemetry: performance, resource usage, and other operational metrics.\n- State Data: very important to create a rich-context, like what's subscriber IP@ now, redundancy state, etc.\n- Logs: *NE* logs to trace the events and errors.\n- Synthetic monitoring: execute *OAM* to measure performance of relevant services.\n\n## Why do we need it?\n\n- Faster Root Cause Analysis: quick identification and resolution performance issues.\n- Higher Productivity: automation of collection and analysis tasks for network operations teams.\n- Improved User Experience: catch and fix problems before it affects users.\n- Better Capacity Planning: understanding of network usage patterns and trends.\n- Enhanced Security: detection of anomalies and potential security threats.\n\nWhile many of them are quite obvious, implementation may be not that easy.\n\n## The three pillars of Observability \n\n### Logs\n\nRecords of events (often human-readable text) with timestamps.\nThey are easy to generate and read. Provide granular details of events or transactions.\nEnable retrospective replay of incidents and can reveal errors/exceptions.\n\n***MAG use-case applicability:*** Practically it's our syslog messages, which are collected from the *NE* devices and sent to the syslog server. \n\n\n### Metrics and State\n\nNumerical measurements (e.g., CPU/memory usage, response times) that track system health over time. They are usually aggregated and stored in time-series databases.\nGreat for detecting trends and setting alert thresholds.\nOffer a high-level view of performance (e.g., error rates, latency).\n\n***MAG use-case applicability:*** In case of network streaming telemetry we should consider sample and on-change depends on a semantic of the metric. Apart from that the state data is also important to collect, since it provides a context for the metrics. For example, if you have a saturation for the particular subscriber, you may want to know what was an IP address or the state of the *MAG*  or link state.\n\n### Traces\n\nEnd-to-end records of a single request’s journey through a distributed system.\nIdeal for pinpointing bottlenecks in complex, microservices-based workflows.\nProvide context-specific details (e.g., which user made a request).\nHelp debug distributed systems by correlating requests across different services.\n\n\n***MAG use-case applicability:*** Has this something to do with *MAG* ? At the first glance no, but if we look deeper and start considering the whole network and distributed system, which is the case, then things like traffic path, appearance of subscriber on the different NEs with relevant state and metrics data could be considered as a trace.\n\n##  Alerting pillars of BNG\n\nWhat are the best practices to alert?\nWell, let's not reinvent a wheel again, since we have quite nice guidelines and best practices from  Niall Richard Murphy, Betsy Beyer, Chris Jones, Jennifer Petoff [Site Reliability Engineering](https://learning.oreilly.com/library/view/site-reliability-engineering/9781491929117/).\nLet's summarize here some key technical and operational aspects of the alerting process:\n\n1. Alert on the data (state and metrics) from *MAG*  devices, CORE and Aggregation layer routers connected to Fixed Access NEs:\n   - High CPU usage and memory usage.\n   - High latency: reach out an *NE*.\n   - Links saturation.\n   - High packet loss: synthetics tests, *OAM*.\n   - High number of error / dropped packets on interfaces.\n   - High number of errors in logs.\n   - Unexpected grow or drop in number sessions.\n   - Approaching capacity / scalability limits.\n   - Multi-chassis state and loss or redundancy between *OLT* and Aggregation routers.\n   - Unexpected changes in number of routes.\n   - Fluctuations on upstream *BGP* peering.\n   - General criteria applicable to modern *IP/MPLS* networks.\n2. Alerts should be real: it should not be a false positive or a false negative.\n3. Alerts should be urgent: it should be sent to the right people at the right time.\n4. Alerts should be actionable: it should provide enough information to help the operator understand the issue and take appropriate action.\n5. Alerts should be clear: it should be easy to understand and interpret.\n\n## MAG (BNG) Observability HOWTO\n\n### Observability Stack\n\nLet's first consider metrics, not the state. The data path is quite standard `gNMIc` =\u003e `Prometheus` =\u003e `Grafana` with sample and on-change *gNMI* telemetry subscriptions.\nThe state data is collected in different ways. If the state is binary and easy to convert to the `(0|1)` metric and passed as time series data via `Prometheus`.\nFor example, the oper-state of the interface can be converted to `0` or `1` via `gnmic` event processors and passed to Prometheus.\n```yaml\nprocessors:\n  oper-state-to-int:\n    event-strings:\n      value-names:\n        - \".*\"\n      transforms:\n        - replace:\n            apply-on: \"value\"\n            old: \"up\"\n            new: \"1\"\n        - replace:\n            apply-on: \"value\"\n            old: \"down\"\n            new: \"0\"\n```\nFull gnmic configuration file to collect telemetry =\u003e [gnmic-metrics.yml](clab/sros_bng_observability/configs/gnmic/gnmic-metrics.yml)\n\nIn case we are talking about something like *SRRP* state or *IPv4/IPv6* address it takes another way: `gNMIc` =\u003e `Fluent Bit` =\u003e `Loki` =\u003e `Grafana`.\nThe corresponding `gnmic` configs can be found below:\n```yaml\n# Definitions of subscriptions\nsubscriptions:\n  sros-sub-mgmt-subscribers-on-change:\n    \u003c\u003c: *mode_stream_on_change\n    paths:\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv4/ip-address/address\"\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv6/dhcp6-lease/client-address\"\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv6/pd/ip-prefix/prefix\"\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv6/slaac/address\"\n      - \"/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=*]/oper-state\"\n      - \"/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=*]/master-since\"\n\n  \n  sros-sub-mgmt-subscribers-sample:\n    \u003c\u003c: *mode_stream_sample\n    paths:\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv4/ip-address/address\"\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv6/dhcp6-lease/client-address\"\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv6/pd/ip-prefix/prefix\"\n      - \"/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[id=*][type=ppp]/ipv6/slaac/address\"\n      - \"/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=*]/oper-state\"\n      - \"/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=*]/master-since\"\n```\nAgain sample and on-change remains relevant too, since we definitely should be aware when the state has been changed. \nSampling is required to and ensure the state is periodically verified for observability purposes and always present on the grafana dashboards in case an interval is a few minutes only.\nThe full `gnmic` configuration file to collect the state data =\u003e [gnmic-state.yml](clab/sros_bng_observability/configs/gnmic/gnmic-state.yml)\n\nThe logs are collected via `syslog-ng` and sent to `Fluent Bit`, which is then sent to `Loki`. Such path is dictated by the need to convert [RFC3164](https://www.rfc-editor.org/rfc/rfc3164) syslog messages into [RFC5424](https://www.rfc-editor.org/rfc/rfc5424) format, which is required by `Fluent Bit`, which is in the next turn feed it toward `Loki`.\n\nAvailability monitoring is implemented via `ICMP`: `Smokeping` =\u003e `Prometheus` =\u003e `Grafana`. For now we are covering *MGMT* interface availability only.\nThe `Smokeping` server is configured to ping the *MAG*  devices and the aggregation device. The results are sent to `Prometheus`, which is then visualized in `Grafana`.\nApart from that additional alerting rules and Heatmap were added to verify if `NEs` are available via *gNMI MDM* interface.\nAlerting rules are configured to send alerts to the `Alertmanager` server, which is then sent to the Telegram channel (check for the telegram token). \nThis is done for the illustrative purposes and could be adjusted per need and requirements relatively easily.\n\nAs *NetOps* and *DevOps* practitioners we should consider limits and scalability of the solution. The current setup is designed to be scalable and can be easily extended to support more devices and services.\nBut we should remember about limits on *NE* side to support number of subscriptions and number of sessions. The telemetry stack scaling should be considered as well as more network elements and data is been collected.\nThis is especially relevant *MAG*  where you can't simply pull everything for hundreds of thousands of subscribers at the same time, but we are going to discuss it in the next session in a bit greater detail.\n\n### What and Why to collect?\n\n| Component                        | XPath                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Notes                                                                                                                                                                                                                                                                                                                                                       |\n|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| CPU                              | - `/state/system/cpu[sample-period=60]/summary/usage`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                                                                                                                                                                                                                                                                                                                                                             |\n| Memory                           | - `/state/system/memory-pools/summary/available-memory`\u003cbr\u003e- `/state/system/memory-pools/summary/current-total-size`\u003cbr\u003e- `/state/system/memory-pools/summary/total-in-use`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Memory usage is calculated as %\u003cbr\u003eMemory usage depends on the application (e.g., *RR* has high memory consumption)\u003cbr\u003eAlert when above 90%                                                                                                                                                                                                                   |\n| *PPP* Session                      | - `/state/subscriber-mgmt/statistics/sessions[counter=ppp-sessions-total-established]/current-value`\u003cbr\u003e- `/state/subscriber-mgmt/statistics/total-hosts[counter=ipv4]`\u003cbr\u003e- `/state/subscriber-mgmt/statistics/total-hosts[counter=ipv6]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Important to monitor. Too much fluctuation would point to a problem.                                                                                                                                                                                                                                                                                        |\n| Hardware\u003cbr\u003eResources            | - `/state/card[slot-number=1]/fp[fp-number=1]/resource-usage/egress-queues`\u003cbr\u003e- `/state/card[slot-number=1]/fp[fp-number=1]/resource-usage/ingress-policers`\u003cbr\u003e- `/state/card[slot-number=1]/mda[mda-slot=1]/resource-usage/sap-instances`\u003cbr\u003e- `/state/system/resource-usage/subscriber-next-hop-entries`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Important to monitor in scaled deployments.\u003cbr\u003eDepleted resource can indicate a configuration error or that *MAG*  has reached its capacity.\u003cbr\u003eSince additional capacity deployment takes time, monitor its trajectory and implications for deployment plans. Resource depletion means no new subscriber sessions.                                             |\n| *SRRP*                             | - `/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=1]/statistics/advertisement-interval-errors`\u003cbr\u003e- `/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=1]/statistics/advertisement-interval-discards`\u003cbr\u003e- `/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=1]/statistics/master-changes`\u003cbr\u003e- `/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=1]/statistics/become-master`\u003cbr\u003e- `/state/service/vprn[service-name=BNG-INTERNET]/subscriber-interface[interface-name=SUB]/group-interface[group-interface-name=GRP_1]/srrp[srrp-id=1]/statistics/become-non-master` | It is important to know which *MAG*  is active for each *Group Interface*, especially when each *MAG*  acts as master for half of the *Group Interfaces* (active/active redundancy).                                                                                                                                                                                  |\n| Port and Interface\u003cbr\u003eStatistics | - `/state/router[router-name=*]/interface[interface-name=*]/statistics/ip/in-octets`\u003cbr\u003e- `/state/router[router-name=*]/interface[interface-name=*]/statistics/ip/out-octets`\u003cbr\u003e- `/state/router[router-name=*]/interface[interface-name=*]/statistics/mpls/in-octets`\u003cbr\u003e- `/state/router[router-name=*]/interface[interface-name=*]/statistics/mpls/out-octets`\u003cbr\u003e- `/state/port[port-id=*]/ethernet/statistics/in-octets`\u003cbr\u003e- `/state/port[port-id=*]/ethernet/statistics/out-octets`\u003cbr\u003e- `/state/port[port-id=*]/ethernet/statistics/in-utilization`\u003cbr\u003e- `/state/port[port-id=*]/ethernet/statistics/out-utilization`\u003cbr\u003e- `/state/port[port-id=*]/oper-state`\u003cbr\u003e- `/state/port[port-id=*]/ethernet/statistics/in-errors`\u003cbr\u003e- `/state/port[port-id=*]/ethernet/statistics/out-errors`                                                                                                                                                                | Port and interface statistics are critical for monitoring, troubleshooting, and capacity planning.                                                                                                                                                                                                                                                           |\n| Per Subscriber\u003cbr\u003eInfo           | - `/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[type=ppp][id=*]/up-time`\u003cbr\u003e- `/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[type=ppp][id=*]/ipv4/forwarding-state`\u003cbr\u003e- `/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[type=ppp][id=*]/ipv6/pd/forwarding-state`\u003cbr\u003e- `/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/session[type=ppp][id=*]/ipv6/slaac/forwarding-state`\u003cbr\u003e- `/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/ingress/qos/statistics/out-profile-forwarded-octets`\u003cbr\u003e- `/state/subscriber-mgmt/subscriber[subscriber-id=*]/sla-profile-instance[id=*]/egress/qos/statistics/out-exceed-profile-forwarded-octets`                                                                                                                                                 | Obtain detailed subscriber session and *QoS* information.                                                                                                                                                                                                                                                                                                    |\n\n# MAG (BNG) Setup\n\nThe topology below should represent the *MAG*  testbed and it's key functional elements without observability stack since it's all living in the MGMT network.\n\n![Topology and Services](pic/Topology_and_Services.png)\n\nThe setup comprises two redundant **MAGs** serving a small number of simulated users in a straightforward network topology. The **AGG**, **MAGs**, and **CORE** routers all use **ISIS** for routing and **SR-MPLS** for tunneling.\n\n**BNG Blaster** simulates subscribers by initiating dual-stack (**IPv4**/**IPv6**) **PPPoE** sessions. Subscriber traffic is transmitted to the **AGG** router via an **Access Interface**.\n\nA service topology is depicted at the top of the diagram, employing a combination of **EVPN** and **L3VPN** services interconnected via **PXC**. On the **AGG** router, an **EVPN-VPLS** instance receives subscriber traffic and broadcasts **PPP** session initiation (**PADI**) requests to the **MAGs**.\n\nThe two **MAGs** function in an active/standby redundancy model. Only the active **MAG** responds to **PADI** messages. The redundancy mechanism is managed using the **SRRP** protocol, which elects the active **MAG**. Both **MAGs** are interconnected through a **Redundant Interface** running over the **MPLS** network. The **Redundant Interface** ensures that traffic can always reach the active **MAG**, even if the standby **MAG** receives downstream traffic due to transient network conditions or misconfigurations.\n\nAll session-related information (such as subscriber **Profiles**, **QoS settings**, and **DHCP leases**) is synchronized continuously to the standby **MAG**.\n\nA **RADIUS** server provides the **AAA** (Authentication, Authorization, and Accounting) services. The active **MAG** authenticates incoming access requests with the **RADIUS** server, which responds with necessary attributes defining subscriber profiles and associated **DHCP Pool Names**. Upon successful authentication, the active **MAG** allocates session resources and coordinates with the **PPP client** to establish the session. IP address assignment is managed locally by the **DHCP Server** on the **MAG**.\n\nBoth **MAGs** advertise aggregated subscriber prefixes to the **CORE**. Prefix states are tracked using **SRRP**, enabling each **MAG** to determine its state (active or standby) for a given prefix. Routes advertised toward the **CORE** from the active **MAG** have a higher **BGP Local Preference** (150). In contrast, the standby **MAG** advertises routes with a standard local preference (100).\n\nThe **CORE** router manages upstream subscriber traffic and maintains a connection back to **BNG Blaster** to complete the loop. **BNG Blaster** receives traffic initially transmitted from the **Access Interface**.\n\n## Key Components\n\nThe following components represent the primary functional and observability elements in the **MAG** testbed setup:\n- **2 SROS MAG / BNG devices** We are using `bng1` and `bng2` as traditional naming conventions here.\n- **1 Aggregation device** (`agg`): Subscriber-side connectivity\n- **1 Core device**: Manages upstream peering (e.g., High-Speed Internet services)\n- **1 BNG Blaster**: Traffic generator simulating 10 dual-stack PPPoE IPv4/IPv6 subscriber sessions\n- **1 Syslog server**: Aggregates logs and forwards them to Promtail\n- **1 Promtail server**: Collects logs from Syslog and forwards to Loki\n- **1 Loki server**: Aggregates log and state data\n- **1 Prometheus server**: Stores and manages collected metrics\n- **1 Grafana server**: Provides visualizations for monitoring and observability\n- **1 Alertmanager server**: Manages alerts from Prometheus, capable of integration with Telegram for alert notifications\n- **1 Fluent Bit server**: Collects device state data from gNMIc and forwards it to Loki\n- **1 Smokeping server**: Gathers latency data and forwards it to Prometheus\n- **1 gNMIc server**: Collects telemetry data from BNG devices and forwards it to Prometheus\n- **1 gNMIc server**: Collects state data from BNG devices and forwards it to Fluent Bit\n- **1 FreeRadius server**: Provides AAA (Authentication, Authorization, and Accounting) functionalities\n\n# Quick Start\n\nIn order to bring up the lab execute `sudo clab deploy`. Use `sudo -E` to preserve your environment variables, such as `HTTP_PROXY`. \nPlease check for a proxy server if you have no direct connection to the Internet!\n\nThe lab setup assumes prometheus and loki data is kept persistent across lab runs, so in order to make it correctly you should adjust permissions for the directories where prometheus and loki data is stored. \n```bash\nsudo chmod o+w promdata\nsudo chmod o+w lokidata\n```\n\nThe lab is designed to be run with `sudo` privileges:\n\n```bash\n❯ sudo -E clab deploy\n15:23:47 INFO Containerlab started version=0.67.0\n15:23:47 INFO Parsing \u0026 checking topology file=bngt.clab.yaml\n15:23:47 INFO Pulling quay.io/prometheus/prometheus:v2.53.4 Docker image\n15:23:52 INFO Done pulling quay.io/prometheus/prometheus:v2.53.4\n15:23:52 INFO Pulling docker.io/grafana/promtail:3.4.3 Docker image\n15:23:53 INFO Done pulling docker.io/grafana/promtail:3.4.3\n15:23:53 INFO Pulling quay.io/prometheus/alertmanager:v0.28.1 Docker image\n15:23:54 INFO Done pulling quay.io/prometheus/alertmanager:v0.28.1\n15:23:54 INFO Pulling quay.io/superq/smokeping-prober:v0.9.0 Docker image\n15:23:56 INFO Done pulling quay.io/superq/smokeping-prober:v0.9.0\n15:23:56 INFO Creating lab directory path=/home/azyablov/clab/sros_bng_observability/clab-sros_bngt\n15:23:56 INFO unable to adjust Labdir file ACLs: operation not supported\n15:23:56 INFO node \"tel-fb\" is being delayed for 20 seconds\n15:23:56 INFO node \"gnmic-metrics\" is being delayed for 40 seconds\n15:23:56 INFO node \"tel-am\" is being delayed for 30 seconds\n15:23:56 INFO node \"tel-sping\" is being delayed for 20 seconds\n15:23:56 INFO node \"tel-syslog\" is being delayed for 10 seconds\n15:23:56 INFO node \"radius\" is being delayed for 10 seconds\n15:23:56 INFO node \"tel-prom\" is being delayed for 40 seconds\n15:23:56 INFO node \"tel-loki\" is being delayed for 20 seconds\n15:23:56 INFO node \"tel-pt\" is being delayed for 20 seconds\n15:23:56 INFO node \"tel-graf\" is being delayed for 60 seconds\n15:23:56 INFO node \"gnmic-state\" is being delayed for 30 seconds\n15:23:56 INFO Using existing config file (/home/azyablov/clab/sros_bng_observability/clab-sros_bngt/bng1/tftpboot/config.txt) instead of applying a new one\n15:23:56 INFO Using existing config file (/home/azyablov/clab/sros_bng_observability/clab-sros_bngt/bng2/tftpboot/config.txt) instead of applying a new one\n15:23:56 INFO Using existing config file (/home/azyablov/clab/sros_bng_observability/clab-sros_bngt/agg/tftpboot/config.txt) instead of applying a new one\n15:23:56 INFO Using existing config file (/home/azyablov/clab/sros_bng_observability/clab-sros_bngt/core/tftpboot/config.txt) instead of applying a new one\n15:23:56 INFO Creating container name=core\n15:23:56 INFO Creating container name=bng1\n15:23:56 INFO Creating container name=bng2\n15:23:56 INFO Creating container name=agg\n15:23:56 INFO Adding public keys configuration node=clab-sros_bngt-bng2\n15:23:56 INFO Created link: bng2:eth2 (1/1/2) ▪┄┄▪ core:eth2 (1/1/2)\n15:23:56 INFO Waiting for node to be ready. This may take a while node=clab-sros_bngt-bng2 log=\"docker logs -f clab-sros_bngt-bng2\"\n15:23:57 INFO Created link: bng1:eth1 (1/1/1) ▪┄┄▪ agg:eth2 (1/1/2)\n15:23:57 INFO Created link: bng1:eth2 (1/1/2) ▪┄┄▪ core:eth1 (1/1/1)\n15:23:57 INFO Adding public keys configuration node=clab-sros_bngt-core\n15:23:57 INFO Waiting for node to be ready. This may take a while node=clab-sros_bngt-core log=\"docker logs -f clab-sros_bngt-core\"\n15:23:57 INFO Created link: bng1:eth3 (1/1/3) ▪┄┄▪ bng2:eth3 (1/1/3)\n15:23:57 INFO Created link: bng2:eth1 (1/1/1) ▪┄┄▪ agg:eth3 (1/1/3)\n15:23:57 INFO Adding public keys configuration node=clab-sros_bngt-agg\n15:23:57 INFO Adding public keys configuration node=clab-sros_bngt-bng1\n15:23:57 INFO Waiting for node to be ready. This may take a while node=clab-sros_bngt-agg log=\"docker logs -f clab-sros_bngt-agg\"\n15:23:57 INFO Waiting for node to be ready. This may take a while node=clab-sros_bngt-bng1 log=\"docker logs -f clab-sros_bngt-bng1\"\n15:24:06 INFO Creating container name=radius\n15:24:06 INFO Creating container name=tel-syslog\n15:24:06 INFO Created link: radius:eth1 ▪┄┄▪ core:eth4 (1/1/4)\n15:24:16 INFO Creating container name=tel-sping\n15:24:16 INFO Creating container name=tel-fb\n15:24:16 INFO Creating container name=tel-pt\n15:24:16 INFO Creating container name=tel-loki\n15:24:26 INFO Creating container name=tel-am\n15:24:26 INFO Creating container name=gnmic-state\n15:24:36 INFO Creating container name=gnmic-metrics\n15:24:36 INFO Creating container name=tel-prom\n15:24:56 INFO Creating container name=tel-graf\n15:24:56 INFO node \"bngblaster\" is being delayed for 10 seconds\n15:25:06 INFO Creating container name=bngblaster\n15:25:06 INFO Created link: bngblaster:eth1 ▪┄┄▪ agg:eth1 (1/1/1)\n15:25:06 INFO Created link: bngblaster:eth2 ▪┄┄▪ core:eth3 (1/1/3)\n15:25:08 INFO Adding host entries path=/etc/hosts\n15:25:08 INFO Adding SSH config for nodes path=/etc/ssh/ssh_config.d/clab-sros_bngt.conf\n╭──────────────────────────────┬───────────────────────────────────────────────────────────────────┬─────────┬────────────────╮\n│             Name             │                             Kind/Image                            │  State  │ IPv4/6 Address │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-agg           │ nokia_sros                                                        │ running │ 172.20.20.21   │\n│                              │ registry.srlinux.dev/pub/vr-sros:24.7.R1                          │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-bng1          │ nokia_sros                                                        │ running │ 172.20.20.11   │\n│                              │ registry.srlinux.dev/pub/vr-sros:24.7.R1                          │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-bng2          │ nokia_sros                                                        │ running │ 172.20.20.12   │\n│                              │ registry.srlinux.dev/pub/vr-sros:24.7.R1                          │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-bngblaster    │ linux                                                             │ running │ 172.20.20.2    │\n│                              │ azyablov/bng-blaster:latest                                       │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-core          │ nokia_sros                                                        │ running │ 172.20.20.22   │\n│                              │ registry.srlinux.dev/pub/vr-sros:24.7.R1                          │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-gnmic-metrics │ linux                                                             │ running │ 172.20.20.42   │\n│                              │ harbor.nice.nokia.net/ghcr/openconfig/gnmic:0.39.1                │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-gnmic-state   │ linux                                                             │ running │ 172.20.20.41   │\n│                              │ harbor.nice.nokia.net/ghcr/openconfig/gnmic:0.39.1                │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-radius        │ linux                                                             │ running │ 172.20.20.111  │\n│                              │ harbor.nice.nokia.net/docker/freeradius/freeradius-server:3.2.3   │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-am        │ linux                                                             │ running │ 172.20.20.35   │\n│                              │ quay.io/prometheus/alertmanager:v0.28.1                           │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-fb        │ linux                                                             │ running │ 172.20.20.33   │\n│                              │ cr.fluentbit.io/fluent/fluent-bit:3.2.10                          │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-graf      │ linux                                                             │ running │ 172.20.20.38   │\n│                              │ grafana/grafana:11.5.2                                            │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-loki      │ linux                                                             │ running │ 172.20.20.34   │\n│                              │ grafana/loki:3.4.2                                                │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-prom      │ linux                                                             │ running │ 172.20.20.37   │\n│                              │ quay.io/prometheus/prometheus:v2.53.4                             │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-pt        │ linux                                                             │ running │ 172.20.20.32   │\n│                              │ grafana/promtail:3.4.3                                            │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-sping     │ linux                                                             │ running │ 172.20.20.36   │\n│                              │ quay.io/superq/smokeping-prober:v0.9.0                            │         │ N/A            │\n├──────────────────────────────┼───────────────────────────────────────────────────────────────────┼─────────┼────────────────┤\n│ clab-sros_bngt-tel-syslog    │ linux                                                             │ running │ 172.20.20.31   │\n│                              │ harbor.nice.nokia.net/docker/linuxserver/syslog-ng:4.8.1-r1-ls153 │         │ N/A            │\n╰──────────────────────────────┴───────────────────────────────────────────────────────────────────┴─────────┴────────────────╯\n```\n\n![bng_topology](pic/bngt.clab.png)\n\nThen it's a time to let in your subscribers by running *BNG Blaster*, just execute following command:\n\n```bash\nsudo docker exec -it clab-sros_bngt-bngblaster bash\n```\n\nThen, you can run the following command to generate traffic:\n```bash\nbngblaster -C pppoe.json -I -l dhcp\n```\n\nOR do it in one go:\n\n```bash\nsudo docker exec -it clab-sros_bngt-bngblaster bash -c 'bngblaster -C pppoe.json -I -l dhcp'\n```\n\nThen you can perform SSH port forwarding to your remote machine toward ro connect locally to `clab-sros_bngt-tel-graf:3000` and voilà... your nice dashboards are ready to use.\n\nGrafana: \u003chttp://localhost:3000\u003e\n\n# Dashboards Gallery\n\n***_BNG State and Redundancy_***\n\n![BNG State and Redundancy](pic/BNG_State_and_Redundancy.png)\n\n***_BNG Resources_***\n\n![BNG Resrouces](pic/BNG_Resources.png)\n\n***_Core and Aggregation Network_***\n\n![Core and Aggregation Network](pic/Core_and_Aggregation_Network.png)\n\n***__Subscriber Inside__***\n\n![Subscriber Inside](pic/Subscriber_Inside.png)\n\n***__Traffic__***\n\n![Traffict](pic/Traffic.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazyablov%2Fsros_bng_observability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazyablov%2Fsros_bng_observability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazyablov%2Fsros_bng_observability/lists"}