Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/monogon-dev/NetMeta

NetMeta is a scalable network observability toolkit optimized for performance.
https://github.com/monogon-dev/NetMeta

collector hacktoberfest ipfix netflow network observability portmirror sflow

Last synced: 4 days ago
JSON representation

NetMeta is a scalable network observability toolkit optimized for performance.

Host: GitHub
URL: https://github.com/monogon-dev/NetMeta
Owner: monogon-dev
License: apache-2.0
Created: 2020-04-15T09:51:59.000Z (over 4 years ago)
Default Branch: main
Last Pushed: 2024-06-13T10:28:50.000Z (5 months ago)
Last Synced: 2024-08-01T12:15:22.863Z (3 months ago)
Topics: collector, hacktoberfest, ipfix, netflow, network, observability, portmirror, sflow
Language: CUE
Homepage: https://netmeta.demo.monogon.dev/
Size: 888 KB
Stars: 133
Watchers: 10
Forks: 8
Open Issues: 55
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-cue - netmeta - NetMeta is a scalable network observability toolkit optimized for performance. (Projects)

README

# NetMeta

| ⚠️ **PRE-RELEASE**: This is a work in progress - please watch this repo for news. |
|-------------------------------------------------------------------------------------------|

NetMeta is a scalable network observability toolkit optimized for performance.

Flows are not pre-aggregated and stored with one second resolution. This allows for
queries in arbitrary dimensions with high-fidelity graphs.

## ⚡️ Features

NetMeta captures, aggregates and analyzes events from a variety of data sources:

* sFlow
* NetFlow/IPFIX
* Port Mirror (AF_PACKET)
* Linux NFLOG (soon)
* Linux conntrack (soon)
* Scamper traces (soon)
* GCP VPC Flow Logs (soon)
* AWS VPC Flow Logs (soon)

The different pros/cons between these and a more informations about the ingest of events can be found [here](doc/ingest.md)

NetMeta also allows to integrate FastNetMon Attack notifications on the Dashboards.
The docs to setup this and other Grafana related settings can be found [here](doc/grafana.md)

Sampling rate is detected automatically. Different devices with different sampling rates can be mixed.
IPv6 is fully supported throughout the stack.

## 👀 Demo

## 💥 API Stability

NetMeta is **beta software** and subject to change. It exposes the following APIs:

* The cluster configuration file for single-node deployments.
* ClickHouse SQL schema for raw database access.
* Protobuf schemas for ingestion for writing custom processors.

One NetMeta has stabilized, these APIs will be stable and backwards compatible.

## 🛠 Deployment
### [Single-node deployment](deploy/single-node/README.md)

NetMeta includes a production-ready single node deployment that scales to up to ~100k events/s and billions of database
rows. More infos can be found [here](deploy/single-node/README.md)

Ingestion performance is limited by CPU performance and disk bandwidth.
Query performance is limited by disk and memory bandwidth, as well as total amount of available memory for larger
in-memory aggregations.

Most reads/writes are sequential due to heavy use of batching in all parts of the stack,
and it works fine even on network storage or spinning disks. We recommend local NVMe drives for best performance.

NetMeta can scale to millions of events per seconds in multi-node deployments.

### Multi-node deployment

We are currently finalizing the design for multi-node deployments. Please contact us if you're interested in
large-scale deployments - we want your feedback!

### Monogon OS

NetMeta will be a first-class citizen on [Monogon OS](https://monogon.tech/monogon_os.html) - stay tuned!

### ☸️ Kubernetes

NetMeta works on any Kubernetes cluster that supports LoadBalancer and Ingress objects and can provision storage.
It's up to you to carefully read the deployment code and cluster role assigments to make sure it works with your
cluster.
Note that we use two operators, which require cluster-admin permissions since CRDs are global
([Strimzi](https://strimzi.io/docs/master) for Kafka
and [clickhouse-operator](https://github.com/Altinity/clickhouse-operator)).

All pieces of NetMeta are installed into a single namespace. By default, this is ``default``, which is
probably not what
you want.
You can change the target namespace in the deployment config.

Please contact us if you need help porting NetMeta to an existing k8s cluster.

## 💼 Support

[Please contact us](https://monogon.tech/pricing.html)
for support and consulting. If you are using NetMeta in production, we'd love to hear from you!

## 🧩 Related

NetMeta is powered by a number of great open source projects, we use:

- [ClickHouse](https://clickhouse.tech) as the main database
- [Kafka](https://kafka.apache.org) as a queue in front of ClickHouse
- [Grafana](https://grafana.com/) with
- [clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) as frontend
- [goflow](https://github.com/cloudflare/goflow) as the sFlow/Netflow collector
- [Strimzi](https://strimzi.io/) to deploy Kafka,
- [clickhouse-operator](https://github.com/Altinity/clickhouse-operator) to deploy ClickHouse, as well as
- [Kubernetes](https://kubernetes.io/) and Rancher's [k3s](https://k3s.io/).

## 🏰 Architecture

```mermaid
flowchart TD;
sFlow --> goflow
IPFIX --> goflow
Netflow --> goflow

kafka[Kafka Broker]
clickhouse["ClickHouse ServerMergeTree(hourly partitions)"]

goflow --> kafka
ntm-agent --> kafka
kafka --> clickhouse
clickhouse --> |SQL| grafana[Grafana]
asmap --> clickhouse
nexthop --> clickhouse
resolver --> clickhouse

```

---

This software is provided "as-is" and
without any express or implied warranties, including, without limitation, the implied warranties of
merchantability and fitness for a particular purpose.