{"id":23080120,"url":"https://github.com/opsdis/aci-streamer","last_synced_at":"2025-04-30T12:35:26.033Z","repository":{"id":116137365,"uuid":"284009562","full_name":"opsdis/aci-streamer","owner":"opsdis","description":"A Cisco ACI streaming telemetry server written in Go","archived":false,"fork":false,"pushed_at":"2023-12-02T08:01:12.000Z","size":285,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-16T13:04:44.397Z","etag":null,"topics":["aci","apic","cisco","elasticsearch","golang","loki","streaming","telemetry","websocket"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opsdis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-07-31T10:33:08.000Z","updated_at":"2024-07-16T20:35:15.000Z","dependencies_parsed_at":"2023-12-01T04:26:40.734Z","dependency_job_id":"908be6f6-0523-45b4-92c1-b2fdfaabf514","html_url":"https://github.com/opsdis/aci-streamer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Faci-streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Faci-streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Faci-streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Faci-streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opsdis","download_url":"https://codeload.github.com/opsdis/aci-streamer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237896984,"owners_count":19383626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aci","apic","cisco","elasticsearch","golang","loki","streaming","telemetry","websocket"],"created_at":"2024-12-16T13:04:54.944Z","updated_at":"2025-02-09T02:33:17.479Z","avatar_url":"https://github.com/opsdis.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"aci-streamer - A Cisco ACI log streamer using ACI REST API subscription mechanism.\n--------------------\n\n[![published](https://static.production.devnetcloud.com/codeexchange/assets/images/devnet-published.svg)](https://developer.cisco.com/codeexchange/github/repo/opsdis/aci-streamer)\n\n\n# Overview\nThe aci-streamer use the Cisco ACI API subscription to provide streaming on events on a ACI classes. \nAccording to the Cisco [documentation](https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-rest-api-configuration-guide/cisco-apic-rest-api-configuration-guide-42x-and-later/m_using_the_rest_api.html)\nsubscription works for:\n\n*\"The REST API supports the subscription to one or more MOs during your active API session. \nWhen any MO is created, changed, or deleted because of a user- or system-initiated action, an event is \ngenerated. If the event changes the data on any of the active subscribed queries, \nthe APIC will send out a notification to the API client that created the subscription.\"*\n\nThe most basic example is subscribing to events on the ACI class `faultInst` to get continues stream of fault events.\n\nThe streamed events are by default written to stdout and the format is json, so it's easy to consume by any \nsystems like [Loki](https://github.com/grafana/loki) and [Elastic](https://www.elastic.co/).\n\n![Dashboard example](images/streamer_example.png)\n\nIn the above screenshot we have combined Prometheus and Loki data sources in the same dashboard. \nIn the middle row we have the Loki logs based on a “stream” from the aci-streamer called faults. \n\nIn the left graph we have the log panel for Loki logs. The query is based on the upper left Grafana variable \nfilters that are applied for the whole dashboard.\n\n    {fabric=~\"$Aci\",stream=\"faults\",podid=~\"$Podid\",nodeid=~\"$Nodeid\",severity=~\"$Severity\"}\n\nOn the right graph we create a simple fault rate metric based on the log data.\n\n    sum by (nodeid,severity) (rate({stream=\"faults\",fabric=~\"$Aci\",podid=~\"$Podid\",nodeid=~\"$Nodeid\",severity=~\"$Severity\"}[5m]))\n\nThe rest of the graphs are based on the [aci-exporter](https://github.com/opsdis/aci-exporter) \nPrometheus data using the same label filters, so we can drill down both on pod, node and severity.\n\nThe events streams are configured by a definition of `streams`. The below example create a stream of ACI created sessions.\n\n```yaml\nstreams:\n  sessions:\n    # The ACI class to subscribe to\n    class_name: aaaActiveUserSession\n    # The query parameters to use for the subscription\n    query_parameter: \"?query-target=self\u0026query-target-filter=and(wcard(aaaActiveUserSession.status,\\\"created\\\"))\"\n    # The json root of where data is collected\n    root: imdata.0.aaaActiveUserSession.attributes    \n    # A json key called mesg that has a format, Go fmt.Sprintf, where source property names is inserted\n    # This is typical a field that do not exist in the original json\n    message:\n      name: mesg\n      format: \"%s - %s\"\n      property_names:\n        - name\n        - ipAddress\n```\nOriginal message from ACI looks like:\n```json\n{\n\t\"subscriptionId\": [\"72066106679951361\"],\n\t\"imdata\": [{\n\t\t\"aaaActiveUserSession\": {\n\t\t\t\"attributes\": {\n\t\t\t\t\"annotation\": \"\",\n\t\t\t\t\"childAction\": \"\",\n\t\t\t\t\"descr\": \"\",\n\t\t\t\t\"dn\": \"uni/usersessext/actsession-CwqMYooyRDKiftKyopDgGw==\",\n\t\t\t\t\"expiryTime\": \"2020-08-04T14:14:09.692+00:00\",\n\t\t\t\t\"extMngdBy\": \"\",\n\t\t\t\t\"hashToken\": \"CwqMYooyRDKiftKyopDgGw==\",\n\t\t\t\t\"ipAddress\": \"10.10.20.141\",\n\t\t\t\t\"lcOwn\": \"local\",\n\t\t\t\t\"loginTime\": \"2020-08-04T14:04:09.692+00:00\",\n\t\t\t\t\"modTs\": \"2020-08-04T14:04:09.691+00:00\",\n\t\t\t\t\"name\": \"admin\",\n\t\t\t\t\"nameAlias\": \"\",\n\t\t\t\t\"ownerKey\": \"\",\n\t\t\t\t\"ownerTag\": \"\",\n\t\t\t\t\"rn\": \"\",\n\t\t\t\t\"status\": \"created\",\n\t\t\t\t\"uType\": \"local\",\n\t\t\t\t\"uid\": \"0\"\n\t\t\t}\n\t\t}\n\t}]\n}\n```\nAfter processed by the above streams configuration:\n```json\n{\n\t\"annotation\": \"\",\n\t\"childAction\": \"\",\n\t\"descr\": \"\",\n\t\"dn\": \"uni/usersessext/actsession-CwqMYooyRDKiftKyopDgGw==\",\n\t\"expiryTime\": \"2020-08-04T14:14:09.692+00:00\",\n\t\"extMngdBy\": \"\",\n\t\"hashToken\": \"CwqMYooyRDKiftKyopDgGw==\",\n\t\"ipAddress\": \"10.10.20.141\",\n\t\"lcOwn\": \"local\",\n\t\"loginTime\": \"2020-08-04T14:04:09.692+00:00\",\n\t\"modTs\": \"2020-08-04T14:04:09.691+00:00\",\n\t\"name\": \"admin\",\n\t\"nameAlias\": \"\",\n\t\"ownerKey\": \"\",\n\t\"ownerTag\": \"\",\n\t\"rn\": \"\",\n\t\"status\": \"created\",\n\t\"uType\": \"local\",\n\t\"uid\": \"0\",\n\t\"mesg\": \"admin - 10.10.20.141\",\n\t\"fabric\": \"ACI Fabric1\",\n\t\"stream\": \"sessions\"\n}\n```\nThe structure has been flattened from the `root` definition. Three additional keys has been added:\n- `fabric` the name of the ACI fabric\n- `stream` then name of the stream configuration\n- `mesg` the created new field\n\n\u003e The main reason for the use of the `mesg` definition is to create a combined log key to use when saved to the log \n\u003e system.\n\nA typical Loki configuration for the above example would be:\n\n```yaml\n  pipeline_stages:\n  - json:\n     expressions:\n       # Get the stream name\n       stream: stream\n  - labels:\n      stream:\n  - match: \n      # Use selector for each stream\n      selector: '{stream=\"sessions\"}'\n      stages:\n      - json:\n         # Define the json keys to use\n         expressions:\n           message: mesg\n           status: status\n           fabric: fabric\n      - labels:\n           # Define which keys to be used as labels \n           status:\n           fabric:\n      - output:\n          # The key to use to write as Loki data\n          source: message \n```\n\nWith the Loki `selector` different stream's that are logged to the same file descriptor that the \naci-streamer writes to can be handled in unique ways.\n\nWith aci-streamer we can for a stream also use a configuration directive called `labels`. We use the example to stream \nACI faults.\n\n```yaml\nstreams:\n  faults:\n    class_name: faultInst\n    root: imdata.0.faultInst.attributes\n    query_parameter: \"\"\n    labels:\n      - property_name: dn\n        regex: \"^topology/pod-(?P\u003cpodid\u003e[1-9][0-9]*)/node-(?P\u003cnodeid\u003e[1-9][0-9]*)/sys/.*\"\n    message:\n      name: message\n      format: \"%s - %s [dn: %s]\"\n      # Names are add to the format in the order they are written\n      property_names:\n        - descr\n        - dn\n```\nIn the above example we define that the property key `dn` should be parsed with a regular named expression. The above\nwill result in two, if matched, new keys called `podid` and `nodeid` with value.\n\nThis will result in the following stream data:\n\n```json\n{\n\t\"ack\": \"no\",\n\t\"cause\": \"threshold-crossed\",\n\t\"changeSet\": \"crcLast:4\",\n\t\"childAction\": \"\",\n\t\"code\": \"F381328\",\n\t\"created\": \"2020-08-03T23:09:28.395+00:00\",\n\t\"delegated\": \"no\",\n\t\"descr\": \"TCA: CRC Align Errors current value(eqptIngrErrPkts5min:crcLast) value 4% raised above threshold 1%\",\n\t\"dn\": \"topology/pod-1/node-102/sys/phys-[eth1/3]/fault-F381328\",\n\t\"domain\": \"infra\",\n\t\"highestSeverity\": \"warning\",\n\t\"lastTransition\": \"2020-08-04T16:14:27.106+00:00\",\n\t\"lc\": \"raised\",\n\t\"occur\": \"951\",\n\t\"origSeverity\": \"warning\",\n\t\"prevSeverity\": \"cleared\",\n\t\"rn\": \"\",\n\t\"rule\": \"tca-eqpt-ingr-err-pkts5min-crc-last\",\n\t\"severity\": \"warning\",\n\t\"status\": \"modified\",\n\t\"subject\": \"counter\",\n\t\"type\": \"operational\",\n\t\"podid\": \"1\",\n\t\"nodeid\": \"102\",\n\t\"message\": \"2020-08-04T16:14:27.106+00:00 - TCA: CRC Align Errors current value(eqptIngrErrPkts5min:crcLast) value 4% raised above threshold 1% [dn: topology/pod-1/node-102/sys/phys-[eth1/3]/fault-F381328]\",\n\t\"fabric\": \"ACI Fabric1\",\n\t\"timestamp\": \"2020-08-04T16:14:27.106000000Z\",\n\t\"stream\": \"faults\"\n}\n```\nEven if `podid` or `nodeid` was not part of the original event, except part of the `dn` key, they are now separate keys.\n\nFor more information about configuration please see the `example-config.yaml`\n\n# Configuration\n\n\u003e For configuration options please see the `example-config.yml` file.\n\nAll attributes in the configuration has default values, except for the fabric and the different streams sections.\nA fabric profile include the information specific to an ACI fabrics, like authentication and apic(s) url.\n\nIf there is multiple apic urls configured the aci-streamer will use the first apic it can log in to starting with the first\nin the list.\n\nAll configuration properties can be set by using environment variables. The prefix is `ACI_STREAMER_` and property \nmust be in uppercase. So to set the property `port` with an environment variable `ACI_STREAMER_PORT=7121`. \n\n# Installation\n\n## Build \n    go build -o build/aci-streamer  *.go\n\n## Run\nBy default, the aci-streamer will look for a configuration file called `config.yaml`. The directory search paths are:\n\n- Current directory\n- $HOME/.aci-streamer\n- usr/local/etc/aci-streamer\n- etc/aci-streamer\n\n```\n    ./build/aci-streamer -fabric XYZ\n```\nWhere XYZ is a named entry in the fabrics section of the configuration file.\n\nTo run against the Cisco ACI sandbox:\n```\n    ./build/aci-streamer -config example-config.yaml -fabric cisco_sandbox \n```\n\u003e Make sure that the sandbox url and authentication is correct. Check out Cisco sandboxes on \n\u003e https://devnetsandbox.cisco.com/RM/Topology - \"ACI Simulator AlwaysOn\"\n\n# Output\nBy default, the output is sent to standard out. The output kan also be directed to a file using the \n`-output` flag. When written to a file aci-streamer use the log4go package to write the output. \nThe main reason for this to get file rotation on the output, which is good if aci-streamer is deployed \nwith a sidecar in kubernetes, like promtail.\n\n# Internal metrics\nInternal metrics are exposed in Prometheus exposition format on the endpoint `/metrics`.\nTo get the metrics in openmetrics format use header `Accept: application/openmetrics-text`\n     \n# Loki promtail configuration\n\nPlease see the example file loggers/promtail.yaml.\n\n# Elastic filebeat configuration\n\nPlease see the example file loggers/filebeat.yaml.\n\n# License\nThis work is licensed under the GNU GENERAL PUBLIC LICENSE Version 3.\n ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopsdis%2Faci-streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopsdis%2Faci-streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopsdis%2Faci-streamer/lists"}