{"id":19561247,"url":"https://github.com/monogon-dev/netmeta","last_synced_at":"2025-02-26T08:43:55.958Z","repository":{"id":39853321,"uuid":"255873819","full_name":"monogon-dev/NetMeta","owner":"monogon-dev","description":"NetMeta is a scalable network observability toolkit optimized for performance.","archived":false,"fork":false,"pushed_at":"2025-01-05T02:33:31.000Z","size":1001,"stargazers_count":137,"open_issues_count":54,"forks_count":6,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-01-08T22:49:26.180Z","etag":null,"topics":["collector","hacktoberfest","ipfix","netflow","network","observability","portmirror","sflow"],"latest_commit_sha":null,"homepage":"https://netmeta.demo.monogon.dev/","language":"CUE","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monogon-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-15T09:51:59.000Z","updated_at":"2024-11-17T18:27:58.000Z","dependencies_parsed_at":"2023-02-19T02:30:50.663Z","dependency_job_id":"30c96b31-4438-47a6-8420-3cfa9a3ba1d3","html_url":"https://github.com/monogon-dev/NetMeta","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monogon-dev%2FNetMeta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monogon-dev%2FNetMeta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monogon-dev%2FNetMeta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monogon-dev%2FNetMeta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monogon-dev","download_url":"https://codeload.github.com/monogon-dev/NetMeta/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240822608,"owners_count":19863302,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collector","hacktoberfest","ipfix","netflow","network","observability","portmirror","sflow"],"created_at":"2024-11-11T05:10:15.545Z","updated_at":"2025-02-26T08:43:55.893Z","avatar_url":"https://github.com/monogon-dev.png","language":"CUE","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NetMeta\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/monogon-dev/NetMeta/issues\" \u003e\n        \u003cimg alt=\"github issues\" src=\"https://img.shields.io/github/issues/monogon-dev/NetMeta?style=for-the-badge\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/monogon-dev/NetMeta/network\" \u003e\n        \u003cimg alt=\"github forks\" src=\"https://img.shields.io/github/forks/monogon-dev/NetMeta?style=for-the-badge\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/monogon-dev/NetMeta/stargazers\" \u003e\n        \u003cimg alt=\"github stars\" src=\"https://img.shields.io/github/stars/monogon-dev/NetMeta?style=for-the-badge\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/monogon-dev/NetMeta/blob/main/LICENSE\" \u003e\n        \u003cimg alt=\"github license\" src=\"https://img.shields.io/github/license/monogon-dev/NetMeta?style=for-the-badge\" /\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n| ⚠️         **PRE-RELEASE**: This is a work in progress - please watch this repo for news. |\n|-------------------------------------------------------------------------------------------|\n\nNetMeta is a scalable network observability toolkit optimized for performance.\n\nFlows are not pre-aggregated and stored with one second resolution. This allows for\nqueries in arbitrary dimensions with high-fidelity graphs.\n\n## ⚡️ Features\n\nNetMeta captures, aggregates and analyzes events from a variety of data sources:\n\n* sFlow\n* NetFlow/IPFIX\n* Port Mirror (AF_PACKET)\n* Linux NFLOG (soon)\n* Linux conntrack (soon)\n* Scamper traces (soon)\n* GCP VPC Flow Logs (soon)\n* AWS VPC Flow Logs (soon)\n\nThe different pros/cons between these and a more informations about the ingest of events can be found [here](doc/ingest.md)\n\nNetMeta also allows to integrate FastNetMon Attack notifications on the Dashboards.\nThe docs to setup this and other Grafana related settings can be found [here](doc/grafana.md)\n\nSampling rate is detected automatically. Different devices with different sampling rates can be mixed.\nIPv6 is fully supported throughout the stack.\n\n## 👀 Demo\n\n\u003cimg src=\"https://i.imgur.com/kl2ThBc.png\" width=\"550px\" alt=\"\" /\u003e\n\u003cimg src=\"https://i.imgur.com/lt0LFqh.png\" width=\"550px\" alt=\"\" /\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/859697/103421113-10df0f00-4b9b-11eb-8747-7ccd9c8af76c.png\" width=\"100%\" alt=\"\" /\u003e\n\n## 💥 API Stability\n\nNetMeta is **beta software** and subject to change. It exposes the following APIs:\n\n* The cluster configuration file for single-node deployments.\n* ClickHouse SQL schema for raw database access.\n* Protobuf schemas for ingestion for writing custom processors.\n\nOne NetMeta has stabilized, these APIs will be stable and backwards compatible.\n\n## 🛠 Deployment\n### [Single-node deployment](deploy/single-node/README.md)\n\nNetMeta includes a production-ready single node deployment that scales to up to ~100k events/s and billions of database\nrows. More infos can be found [here](deploy/single-node/README.md)\n\nIngestion performance is limited by CPU performance and disk bandwidth.\nQuery performance is limited by disk and memory bandwidth, as well as total amount of available memory for larger\nin-memory aggregations.\n\nMost reads/writes are sequential due to heavy use of batching in all parts of the stack,\nand it works fine even on network storage or spinning disks. We recommend local NVMe drives for best performance.\n\nNetMeta can scale to millions of events per seconds in multi-node deployments.\n\n### Multi-node deployment\n\nWe are currently finalizing the design for multi-node deployments. Please contact us if you're interested in\nlarge-scale deployments - we want your feedback!\n\n### Monogon OS\n\nNetMeta will be a first-class citizen on [Monogon OS](https://monogon.tech/monogon_os.html) - stay tuned!\n\n### ☸️ Kubernetes\n\nNetMeta works on any Kubernetes cluster that supports LoadBalancer and Ingress objects and can provision storage.\nIt's up to you to carefully read the deployment code and cluster role assigments to make sure it works with your\ncluster.\nNote that we use two operators, which require cluster-admin permissions since CRDs are global\n([Strimzi](https://strimzi.io/docs/master) for Kafka\nand [clickhouse-operator](https://github.com/Altinity/clickhouse-operator)).\n\nAll pieces of NetMeta are installed into a single namespace. By default, this is ``default``, which is\nprobably not what\nyou want.\nYou can change the target namespace in the deployment config.\n\nPlease contact us if you need help porting NetMeta to an existing k8s cluster.\n\n## 💼 Support\n\n[Please contact us](https://monogon.tech/pricing.html)\nfor support and consulting. If you are using NetMeta in production, we'd love to hear from you!\n\n## 🧩 Related\n\nNetMeta is powered by a number of great open source projects, we use:\n\n- [ClickHouse](https://clickhouse.tech) as the main database\n- [Kafka](https://kafka.apache.org) as a queue in front of ClickHouse\n- [Grafana](https://grafana.com/) with\n- [clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) as frontend\n- [goflow](https://github.com/cloudflare/goflow) as the sFlow/Netflow collector\n- [Strimzi](https://strimzi.io/) to deploy Kafka,\n- [clickhouse-operator](https://github.com/Altinity/clickhouse-operator) to deploy ClickHouse, as well as\n- [Kubernetes](https://kubernetes.io/) and Rancher's [k3s](https://k3s.io/).\n\n## 🏰 Architecture\n\n```mermaid\nflowchart TD;\n    sFlow --\u003e goflow\n    IPFIX --\u003e goflow\n    Netflow --\u003e goflow\n\n    kafka[Kafka Broker]\n    clickhouse[\"ClickHouse Server\u003c/br\u003eMergeTree\u003c/br\u003e(hourly partitions)\"]\n    \n    goflow --\u003e kafka\n    ntm-agent --\u003e kafka\n    kafka --\u003e clickhouse\n    clickhouse --\u003e |SQL| grafana[Grafana]\n    asmap --\u003e clickhouse\n    nexthop --\u003e clickhouse\n    resolver --\u003e clickhouse\n\n```\n\n---\n\n(C) 2022 [Monogon SE](https://monogon.tech).\n\nThis software is provided \"as-is\" and\nwithout any express or implied warranties, including, without limitation, the implied warranties of\nmerchantability and fitness for a particular purpose.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonogon-dev%2Fnetmeta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonogon-dev%2Fnetmeta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonogon-dev%2Fnetmeta/lists"}