Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/coroot/coroot-node-agent

A Prometheus exporter based on eBPF that gathers comprehensive container metrics
https://github.com/coroot/coroot-node-agent

ebpf logs monitoring network-metrics node-metrics observability prometheus prometheus-exporter prometheus-metrics

Last synced: 24 days ago
JSON representation

A Prometheus exporter based on eBPF that gathers comprehensive container metrics

Host: GitHub
URL: https://github.com/coroot/coroot-node-agent
Owner: coroot
License: apache-2.0
Created: 2021-09-15T11:46:46.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2024-08-21T13:39:39.000Z (3 months ago)
Last Synced: 2024-08-30T09:32:32.092Z (2 months ago)
Topics: ebpf, logs, monitoring, network-metrics, node-metrics, observability, prometheus, prometheus-exporter, prometheus-metrics
Language: Go
Homepage: https://coroot.com/docs/metrics/node-agent
Size: 19.8 MB
Stars: 313
Watchers: 9
Forks: 55
Open Issues: 19
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

# Coroot-node-agent

[![Go Report Card](https://goreportcard.com/badge/github.com/coroot/coroot-node-agent)](https://goreportcard.com/report/github.com/coroot/coroot-node-agent)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

The agent gathers metrics related to a node and the containers running on it, and it exposes them in the Prometheus format.

It uses eBPF to track container related events such as TCP connects, so the minimum supported Linux kernel version is 4.16.

## Features

### TCP connection tracing

To provide visibility into the relationships between services, the agent traces containers TCP events, such as *connect()* and *listen()*.

Exported metrics are useful for:
* Obtaining an actual map of inter-service communications. It doesn't require integration of distributed tracing frameworks into your code.
* Detecting connections errors from one service to another.
* Measuring network latency between containers, nodes and availability zones.

Related blog posts:
* [Building a service map using eBPF](https://coroot.com/blog/building-a-service-map-using-ebpf)
* [How ping measures network round-trip time accurately using SO_TIMESTAMPING](https://coroot.com/blog/how-to-ping)
* [The current state of eBPF portability](https://coroot.com/blog/ebpf-portability)
### Log patterns extraction

Log management is usually quite expensive. In most cases, you do not need to analyze each event individually.
It is enough to extract recurring patterns and the number of the related events.

This approach drastically reduces the amount of data required for express log analysis.

The agent discovers container logs and parses them right on the node.

At the moment the following sources are supported:
* Direct logging to files in */var/log/*
* Journald
* Dockerd (JSON file driver)
* Containerd (CRI logs)

To learn more about automated log clustering, check out the blog post "[Mining metrics from unstructured logs](https://coroot.com/blog/mining-logs-from-unstructured-logs)".

### Delay accounting

[Delay accounting](https://www.kernel.org/doc/html/latest/accounting/delay-accounting.html) allows engineers to accurately
identify situations where a container is experiencing a lack of CPU time or waiting for I/O.

The agent gathers per-process counters through [Netlink](https://man7.org/linux/man-pages/man7/netlink.7.html) and aggregates them into per-container metrics:
* [container_resources_cpu_delay_seconds_total](https://coroot.com/docs/metrics/node-agent#container_resources_cpu_delay_seconds_total)
* [container_resources_disk_delay_seconds_total](https://coroot.com/docs/metrics/node-agent#container_resources_disk_delay_seconds_total)

Related blog posts:
* [Delay accounting: an underrated feature of the Linux kernel](https://coroot.com/blog/linux-delay-accounting)

### Out-of-memory events tracing

The [container_oom_kills_total](https://coroot.com/docs/metrics/node-agent#container_oom_kills_total) metric shows that a container has been terminated by the OOM killer.

### Instance meta information

If a node is a cloud instance, the agent identifies a cloud provider and collects additional information using the related metadata services.

Supported cloud providers: [AWS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html), [GCP](https://cloud.google.com/compute/docs/metadata/overview), [Azure](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service?tabs=linux), [Hetzner](https://docs.hetzner.cloud/#server-metadata)

Collected info:
* AccountID
* InstanceID
* Instance/machine type
* Region
* AvailabilityZone
* AvailabilityZoneId (AWS only)
* LifeCycle: on-demand/spot (AWS and GCP only)
* Private & Public IP addresses

Related blog posts:
* [Gathering cloud instance metadata in AWS, GCP and Azure](https://coroot.com/blog/cloud-metadata)

## Installation

The documentation is available at [coroot.com/docs/metric-exporters/node-agent](https://coroot.com/docs/metric-exporters/node-agent/installation).

## Metrics

The collected metrics are described [here](https://coroot.com/docs/metrics/node-agent).

## Coroot

The best way to turn metrics to answers about app issues is to use [Coroot](https://github.com/coroot/coroot) - a zero-instrumentation observability tool for microservice architectures.

A live demo of Coroot is available at [community-demo.coroot.com](https://community-demo.coroot.com)

## Contributing
To start contributing, check out our [Contributing Guide](https://github.com/coroot/coroot-node-agent/blob/main/CONTRIBUTING.md).

## License

Coroot-node-agent is licensed under the [Apache License, Version 2.0](https://github.com/coroot/coroot-node-agent/blob/main/LICENSE).

The BPF code is licensed under the General Public License, Version 2.0.