{"id":18678184,"url":"https://github.com/srl-labs/logging-with-loki-lab","last_synced_at":"2026-03-09T17:37:05.238Z","repository":{"id":241272408,"uuid":"803659326","full_name":"srl-labs/logging-with-loki-lab","owner":"srl-labs","description":"A modern logging with SR Linux and Loki using Syslog","archived":false,"fork":false,"pushed_at":"2024-05-24T09:45:40.000Z","size":9992,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-22T19:54:06.958Z","etag":null,"topics":["clab-topo","loki","srlinux","syslog"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/srl-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-21T06:36:40.000Z","updated_at":"2025-04-11T14:56:58.000Z","dependencies_parsed_at":"2025-05-19T02:46:11.119Z","dependency_job_id":null,"html_url":"https://github.com/srl-labs/logging-with-loki-lab","commit_stats":null,"previous_names":["srl-labs/logging-with-loki-lab"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/srl-labs/logging-with-loki-lab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Flogging-with-loki-lab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Flogging-with-loki-lab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Flogging-with-loki-lab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Flogging-with-loki-lab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/srl-labs","download_url":"https://codeload.github.com/srl-labs/logging-with-loki-lab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/srl-labs%2Flogging-with-loki-lab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30304741,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T17:35:44.120Z","status":"ssl_error","status_checked_at":"2026-03-09T17:35:43.707Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clab-topo","loki","srlinux","syslog"],"created_at":"2024-11-07T09:36:18.189Z","updated_at":"2026-03-09T17:37:05.217Z","avatar_url":"https://github.com/srl-labs.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SR Linux remote logging with Loki\n\nIn today's rapidly evolving network environments, maintaining robust and efficient operations is more critical than ever. Remote logging plays a pivotal role in achieving this by providing a comprehensive and centralized method for monitoring, analyzing, and troubleshooting network activities across diverse and distributed systems. This can be achieved by using a logging stack consisting of a syslog collector [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/), a database for central log storage [Loki](https://grafana.com/docs/loki/latest/) and a visualization [Grafana](https://grafana.com/docs/grafana/latest/) tool for querying and analyzing data.\n\nOn a high level, SR Linux will send syslog messages according to [RFC5424](https://datatracker.ietf.org/doc/html/rfc5424) to Promtail which is repsonsible for collecting the logs. Promtail will create additional labels assotiated to the logs and pass it through to the database Loki. Grafana can then query the logs based on the labels assiciated to them.\n\n![stack](./img/stack.JPG)\n\n## Deploying the lab\n\nThe lab is deployed with the [containerlab](https://containerlab.dev) project, where [`st.clab.yml`](st.clab.yml) file declaratively describes the lab topology.\n\n```bash\n# change into the cloned directory\n# and execute\ncontainerlab deploy --reconfigure\n```\n\nTo remove the lab:\n\n```bash\ncontainerlab destroy --cleanup\n```\n\n## Accessing the network elements\n\nOnce the lab has been deployed, the different SR Linux nodes can be accessed via SSH through their management IP address, given in the summary displayed after the execution of the deploy command. It is also possible to reach those nodes directly via their hostname, defined in the topology file. Linux clients cannot be reached via SSH, as it is not enabled, but it is possible to connect to them with a docker exec command.\n\n```bash\n# reach a SR Linux leaf or a spine via SSH\nssh admin@clab-loki-leaf1\nssh admin@clab-loki-spine1\n\n# reach a Linux client via Docker\ndocker exec -it clab-loki-client1 bash\n```\n\n## Configure logging on SR Linux\nLet's start from the beginning of our data pipeline and configure logging on SR Linux nodes.\n\nNokia SR Linux Network OS design employs a high level of granularity where each service is represented as a standalone process with a messaging bus enabling inter-process communication. These processes implement logging via the standard Linux syslog interface. In particular, SR Linux uses a well-known rsyslog server to manage syslog messages in the underlying Linux OS.\n\nBasic logging configuration consists of specifying a source for input log messages, filtering the log messages, and specifying an output destination for the filtered log messages.\n\nMessages logged to Linux syslog [facilities](https://documentation.nokia.com/srlinux/24-3/books/config-basics/logg.html#input-sources-log-messages__ai9ep6mg6y) and messages generated by SR Linux [subsystems](https://documentation.nokia.com/srlinux/24-3/books/config-basics/logg.html#input-sources-log-messages__ai9ep6mg6z) can be used as input sources for log messages. You can find a list of those facilities and subsystems in the documentation.\nWhen defining a Linux OS facility or SR Linux subsystem as a log source, users can provide a priority param that narrows the capture to a given severity, a range, or a specific set of severities.\n\nA destination for the ingested and filtered log messages can be one of the following:\n\n* **local log file**: a file on disk that can be configured with retention policies.\n* **buffer**: an in-memory file.\n* **console**: console output.\n* **remote**: remote server.\n\nIn the course of this lab exercise, we will work with a `remote-destination` output type, as we intend to send the log messages over to Promtail.\n\nWith a basic understanding of how to configure logging on SR Linux, let's see what does logging configuration look like. Here is an example:\n\n```sh\n--{ + running }--[  ]--\nA:leaf11# system logging\n\n--{ + running }--[ system logging ]--\nA:leaf11# info\n    network-instance mgmt\n    #\u003csnippet\u003e\n    remote-server 10.128.1.75 {\n        transport udp\n        remote-port 1514\n        subsystem aaa {\n            priority {\n                match-above informational\n            }\n        }\n        subsystem acl {\n            priority {\n                match-above informational\n            }\n        }\n        # other subsystems snipped here for brevity\n        subsystem vxlan {\n            priority {\n                match-above informational\n            }\n        }\n    }\n```\n\nNow log into `clab-srexperts-leaf11` and navigate to the `system logging remote-server 10.128.1.75` context. With the `info` command you can see what is configured in this conext. Compare the configuration with the example above. What do you notice?\n\n```\n--{ + running }--[  ]--\nA:leaf11# system logging remote-server 10.128.1.75\n\n--{ + running }--[ system logging remote-server 10.128.1.75 ]--\nA:leaf11# info detail\n    transport udp\n    remote-port 1514\n    facility local6 {\n        priority {\n            match-above informational\n        }\n    }\n```\n\nYou should notice we don't define individual [subsystems](https://documentation.nokia.com/srlinux/24-3/books/config-basics/logg.html#input-sources-log-messages__ai9ep6mg6z) here, but only define a facility `local6` matching informational level priority and above. By default, SR Linux subsystem messages are logged to Linux syslog facility `local6`, which gives us a configuration efficiency if we want to capture all.\n\n### Log format\nConsider the following example of syslog-formatted messages of the `chassis` subsystem that can be seen on SR Linux:\n```bash\n2024-05-17T20:27:21.252868+00:00 leaf11 local6|NOTI sr_chassis_mgr: chassis|1476|1476|00122|N: The subinterface ethernet-1/50.0 is now up\n2024-05-17T20:27:23.630370+00:00 leaf11 local6|NOTI sr_chassis_mgr: chassis|1476|1476|00123|N: Interface lag1 is now up\n2024-05-17T20:27:23.630388+00:00 leaf11 local6|NOTI sr_chassis_mgr: chassis|1476|1476|00124|N: The subinterface lag1.1 is now up\n2024-05-17T20:27:23.630494+00:00 leaf11 local6|NOTI sr_chassis_mgr: chassis|1476|1476|00125|N: The subinterface lag1.101 is now up\n```\n\nLog message format that is used by rsyslog when sending to remote destination has the following signature:\n```\n\u003cTIMESTAMP\u003e \u003cHOSTNAME\u003e \u003cFACILITY\u003e|\u003cSEVERITY\u003e \u003cAPPLICATION\u003e: \u003cSUBSYSTEM\u003e|\u003cPID\u003e|THREAD_ID|SEQUENCE|\u003cSEVERITY\u003e: \u003cMESSAGE\u003e \n```\nwhere\n```\n\u003cTIMESTAMP\u003e   - Time in format - MMM DD HH:MM:SS.\n\u003cHOSTNAME\u003e    - SR Linux hostname.\n\u003cAPPLICATION\u003e - SR Linux application name, in the context of Syslog this is the Message ID.\n\u003cSUBSYSTEM\u003e   - SR Linux subsystem name, which is configured under /system/logging/remote-server \n\u003cPID\u003e         - Process ID.\n\u003cTHREAD_ID\u003e   - Thread ID.\n\u003cSEQUENCE\u003e    - Sequence number, which allows to reproduce order of the messages sent by SR Linux.\n\u003cSEVERITY\u003e    - A singe char indicating criticality of the message (I - informational, W - warning, etc.)\n\u003cMESSAGE\u003e     - Application free-form message that provides information about the event, that could contain network-instance name, like \"Network-instance default\".\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eDumping syslog messages sent to the remote-server?\u003c/summary\u003e\n\u003cbr\u003e\nThe format that rsyslog uses to send log messages to the remote destination differs from the default format used for buffer and file destination.\n\nTo see the messages on the wire as they are being sent towards a remote syslog collector users can leverage tcpdump tool available on SR Linux:\n```\n--{ running }--[  ]--\nA:leaf1# bash \n[admin@leaf11 ~]$ tcpdump -vAnni any dst 10.128.1.75\ntcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes\n\n07:50:36.937551 mgmt0.0 Out IP (tos 0x0, ttl 64, id 10656, offset 0, flags [DF], proto UDP (17), length 182)\n    10.128.1.33.48501 \u003e 10.128.1.75.1514: UDP, length 154\nE...).@.@..+..!..K.u....zj\u003c182\u003e1 2024-05-21T07:50:36.937263+00:00 leaf11 sr_cli - - -  debug|896501|896501|00003|I: common    |admin|45|srl running / | info interface ethernet-1/1\n```\n\u003c/details\u003e\n\n## Promatil\nNow we are at a point when a raw syslog message has been generated by SR Linux and sent towards its first stop - Promtail. Promtail will take the raw syslog and generate additional labels associated to the log entry. You can find the Promtail configuration file [here](../../clab/configs/promtail/promtail-config.yml). Let's have a closer look and break it down a bit. \n\nThis section specifies where Promtail should send the logs. In this case, it's sending logs to a Loki instance.\n```yaml\nclients:\n  - url: http://loki:3100/loki/api/v1/push\n```\n\nHere we define Promtail to be used as a syslog collector. It listens to any IP address on UDP port 1514 and generates a stattic label `syslog` which will me applied to all log entries. \n```yaml\nscrape_configs:\n  - job_name: syslog\n    syslog:\n      listen_address: 0.0.0.0:1514\n      listen_protocol: udp\n      idle_timeout: 300s\n      label_structured_data: yes\n      labels:\n        job: \"syslog\"\n```\n\nRelabel configs is used to manipulate or add labels to the log entries. These labels can be very useful when query them from the Loki database. In the example below we manipulate only one label - the `source` label. `source_label` extracts a value from the specified field in the syslog message, in this case the `hostname`, and replaces it with `clab-srexperts-$1` and applies it to the `target_label` `source`. \n```yaml\nscrape_configs:\n    #\u003csnippet\u003e\n    relabel_configs:\n      - source_labels: [\"__syslog_message_hostname\"]\n        replacement: clab-srexperts-$1\n        target_label: \"source\"\n      - source_labels: [\"__syslog_message_severity\"]\n        target_label: \"level\"\n      - source_labels: [\"__syslog_message_app_name\"]\n        target_label: \"application\"\n      - source_labels: [\"__syslog_message_facility\"]\n        target_label: \"facility\"\n      - source_labels: [\"__syslog_message_proc_id\"]\n        target_label: \"procid\"\n```\n\n![promtail](./img/promtail.JPG)\n\n## Loki\nTime to navigate loki and query some logs based on the label created by Promtail. Log into the Grafana dashboard with `admin`/`SReXperts2024` credentials. \n\n(\\*_connect using the instructions below before moving on_)\n\n\u003e To be able to view the [grafana](https://grafana.com) dashboard on your laptop, use ssh forwarding: \\\n\u003e `ssh -NL 3000:10.128.\u003clab id\u003e.73:3000 \u003cuser\u003e@\u003clab server\u003e`\\\n\u003e Now you can access the [grafana](https://grafana.com) dashboard at: [http://127.0.0.1:3000](http://127.0.0.1:3000)\n\n1. Go to the `burger menu` top left corner\n2. Click on Explore\n3. Select Loki as data source\n4. In `Label filter` select label `source` with value `clab-srexperts-leaf11`\n5. Click `Run query`\n6. You should now see all the log related to `clab-srexperts-leaf11`\n7. Add a extra search label by clicking on `+`\n8. In `Label filter` select label `application` with value `sr_cli`\n9. You should now see all cli activity on `clab-srexperts-leaf11`\n\n![promtail](./img/loki-nav.gif)\n\n## Grafana\nUsers have the posibility to implement Loki logs into Grafana dashboards. Have a look into the `SR Linux Telemetry` dashboard. \n\n![promtail](./img/grafana.JPG)\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrl-labs%2Flogging-with-loki-lab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrl-labs%2Flogging-with-loki-lab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrl-labs%2Flogging-with-loki-lab/lists"}