{"id":27294501,"url":"https://github.com/nerdalert/cloud-bandwidth","last_synced_at":"2025-04-11T22:53:17.687Z","repository":{"id":33015149,"uuid":"36649055","full_name":"nerdalert/cloud-bandwidth","owner":"nerdalert","description":"Measure and Graph Network Bandwidth","archived":false,"fork":false,"pushed_at":"2022-04-30T05:46:31.000Z","size":28776,"stargazers_count":178,"open_issues_count":2,"forks_count":35,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-11T22:53:08.777Z","etag":null,"topics":["docker","grafana","graphite","kubernetes","monitoring","networking","time-series"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nerdalert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-01T08:53:08.000Z","updated_at":"2025-03-11T08:46:22.000Z","dependencies_parsed_at":"2022-07-16T05:00:37.199Z","dependency_job_id":null,"html_url":"https://github.com/nerdalert/cloud-bandwidth","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fcloud-bandwidth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fcloud-bandwidth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fcloud-bandwidth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fcloud-bandwidth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nerdalert","download_url":"https://codeload.github.com/nerdalert/cloud-bandwidth/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248492951,"owners_count":21113162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","grafana","graphite","kubernetes","monitoring","networking","time-series"],"created_at":"2025-04-11T22:53:16.112Z","updated_at":"2025-04-11T22:53:17.677Z","avatar_url":"https://github.com/nerdalert.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bandwidth Performance Monitoring\n\n[![cloud-bandwidth CI](https://github.com/nerdalert/cloud-bandwidth/actions/workflows/build-image.yml/badge.svg)](https://github.com/nerdalert/cloud-bandwidth/actions/workflows/build-image.yml)\n\n### Overview\n\nNetwork visibility is one of the most important assets in a network engineer's toolkit. This is a tool measuring bandwidth \nthat can be useful for capacity planning, SLAs, troubleshooting or any other scenario that having a realtime and historical \nmeasurements of bandwidth is useful. Multiple cloud environments, coupled with data gravity driving clusters of compute at \nthe edge is increasing the network sprawl and piling on the ever-growing challenges for network ops/architecture/engineering.\n\nThis project is designed to measure bandwidth to distributed endpoints across networks and clouds. Having visibility to \nedge and multi-cloud resources can get overly complicated and/or expensive with many solutions. Heavyweight agents can be \nproblematic.\n\nThis is a simple method of:\n- Setting up iperf servers on your edge/clouds/enterprise as listeners\n- Polling the listeners\n- Graphing the results into a TSDB \n- Visualizing the results into dashboards with Grafana\n- Netperf/Netserver support is optional instead of Iperf3\n\nFor Kubernetes support, see [README-KUBERNETES](kubernetes/README-KUBERNETES.md) for experimental usage.\n\n### New Opportunities to Manage Networks\n\n- Building tools has never been easier. Open source software can be leveraged to customize tools for your network or customers network \n\n- Historical records of network data. Many of the traditional problems with networking and uptime can be significantly reduced\nwith post-mortem reviews with relevant data to determine root causes of events. \n\n- Rather than using traditional RDBMS approaches, here we are using Time Series Databases (TSDB) that are designed to ingest \nlarge amounts of timestamped metrics. That is perfect for collecting all the different metrics we are interested in as network nerds.\n\n### QuickStart Demo\n\nStart the TSDB and Grafana in your preferred container runtime, below is a nice and simple grafan/graphite stack (you can also setup \nyour own TSDB stack as long as it supports the formatting exported shown below):\n\n```sh\ndocker run -d \\\n   --name graphite-grafana \\\n   --restart=always \\\n   -p 80:80 \\\n   -p 2003-2004:2003-2004 \\\n   quay.io/networkstatic/graphite-grafana\n \n # or using Podman\n sudo sysctl net.ipv4.ip_unprivileged_port_start=80\n podman run -d \\\n    --name graphite-grafana \\\n    --restart=always \\\n    -p 80:80 \\\n    -p 2003-2004:2003-2004 \\\n   quay.io/networkstatic/graphite-grafana\n```\n\nThis maps the following ports:\n\nHost | Container | Service\n---- | --------- | -------------------------------------------------------------------------------------------------------------------\n  80 |        80 | [grafana](http://docs.grafana.org/)\n2003 |      2003 | [carbon receiver - plaintext](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol)\n2004 |      2004 | [carbon receiver - pickle](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol)\n\nverify you can reach the grafana/graphite server running by pointing your browser to the container IP. If you're running Docker for \ndesktop on a Mac, [http://localhost](http://localhost). On a Linux host (easiest) just point to the host IP since the port is getting mapped with `-p 80:80`. \nThe default login is `username: admin` and `password: admin`\n\n```sh\ngit clone https://github.com/nerdalert/cloud-bandwidth.git\ncd cloud-bandwidth/\n```\n\nNow you can run a simple script that will generate some metrics to simulate iperf polls and push them to the graphite/grafana server. \nI recommend starting here in order to make sure your stack is working properly.\n\n```sh\n./scripts/generate_test_grafana_data.sh \u003cinsert_ip_address of the server\u003e 2003\n```\n\nWhile some generic data is getting populated, next log into the Grafana server and import a dashboard that will match the sample data the script is pushing.\n\n- Copy the json in [default-grafana.json](./default-grafana.json) and paste it into a new dashboard by clicking the + and then import in the Grafana splash page:\n\n![](docs/images/grafana-import-sm.png)\n\n- After a couple of minutes you should start seeing data getting populated like so: \n\n![](docs/images/grafana-demo.png)\n\nIf you do not see data appearing in the graphs, it will possibly be a connectivity issue. Testing the port is a good first step with something like netcat or your favorite tool:\n\n```sh\nnc -zv \u003caddress_of_graphite/grafana container\u003e 2003\nfound 0 associations\nfound 1 connections:\n     1:\tflags=82\u003cCONNECTED,PREFERRED\u003e\n\toutif lo0\n\tsrc ::1 port 49493\n\tdst ::1 port 2003\n\trank info not available\n\tTCP aux info available\n\nConnection to localhost port 2003 [tcp/*] succeeded!\n```\n\nYou can also send any data you want to the Graphite/Grafana/Carbon stack for testing with:\n\n```sh\necho \"bandwidth.download.azure $RANDOM `date +%s`\" | nc \u003cinsert_ip_of_the_graphite_server_here\u003e 2003\n```\n\nIt may be helpful in getting started with Grafana to use the template included here with names of the server endpoints you \nsetup, or start from scratch with your endpoints. The name of the endpoint being polled is what will show up in Grafana that \nyou then make a graph for. You can view all endpoints in the time-series database by entering `bandwidth.download.*` \nor `bandwidth.upload.*` from within the Grafana UI. It's pretty easy to get the hang of by playing around with the demo data \nand grafana template included that you can import.\n\n## Quickstart with Real Bandwidth\n\nOnce you have some test data being feed into the carbon receiver, next let's measure some real bandwidth. First, if you haven't \nused iperf before, I recommend running it manually to get a feel of what's going on under the hood to get the measurements \n[nerdalert/iperf3](https://github.com/nerdalert/iperf3). The remote nodes can be simple hardware/VMs/Rasberry Pi's or even network \ndevices that are flexible enough to install software on them with a little CPU allocated.\n\n```sh\ngit clone https://github.com/nerdalert/cloud-bandwidth.git\ncd cloud-bandwidth/\n```\n\nOn your edge devices that you are measuring bandwidth to, start the iperf servers. You can either run them in a container \nor in the default namespace outside a container:\n\n- In a container run:\n\n```sh\n# running the container in daemon mode with `-d` is the best idea but for \n# getting started to view the output for debugging the setup with stdout, I recommend:\ndocker run -it --restart=always --name=iperf-svr networkstatic/iperf3 -s\n# or with podman \npodman run -it --restart=always --name=iperf-svr networkstatic/iperf3 -s\n```\nAs you go beyond testing on your localhost, you will also likely want to map the iperf listening port 5201 to be exposed \nfor remote connections like so using the flag `-p 5201:5201`:\n\n```sh\ndocker run -it --restart=always --name=iperf3-server2 -p 5201:5201  networkstatic/iperf3 -s\n\n# Or using podman and quay if you prefer:\npodman run -it --restart=always --name=iperf3-server2 -p 5201:5201  quay.io/networkstatic/iperf3 -s\n```\n\nJust as valid, if you dont feel like using a container, you can simply install iperf3 and run it in the default namespace:\n\n```sh\niperf3 -s\n```\n\nAt any time you can delete the container with:\n\n```sh\ndocker stop iperf-svr \u0026\u0026 docker rm iperf-svr\n```\n\n### COnfiguration File config.yaml\n\nThe program can be configured in three different ways, configuration file `config.yaml`, CLI arguments or ENV variables. \nRun the binary with `--help` to view the CLI/ENV configuration options. To start with, I would probably recommend the \nconfiguration file as discussed next.\n\n- For testing this out, I recommend starting some iperf instances in containers on your dev machine to make troubleshooting \n- easier. The poller is a container so there is no need to map ports if you are running your test iperf server on the same \nmachine as where you are running the `cloud-bandwidth` binary. \n\n- The `iperf-servers` in the config are the addresses of the `iperf -s` instances you are going to poll. The name after the \naddress is not required, but only there for better labeling of the data e.g. `172.17.0.6: ibm-cloud`.\n\n- `tsdb-download-prefix` and `tsdb-upload-prefix` are essentially the paths that the data is getting mapped to in the tsdb. \nGiving them meaningful names can be useful for organizing data viewing in grafana.\n\n- `test-interval` is the time between polls to all  the nodes listed under `iperf-servers`.\n\n- `test-length` is the time the `iperf -c` client poll will run. The longer the test the more accurate the results up to \na certain point, but it also consumes more bandwidth so it is left to a short period in the example.\n\n```yaml\n---\n# the length of the iperf test in seconds\ntest-length: 5\n# the time between polls, defaults to 5 minutes (300 sec)\ntest-interval: 300\n# iperf server port\nserver-port: 5201\n# Address of the graphite/grafana stack running in a container (docker for mac uses localhost).\n# For a setup beyond a dev environment, grafana-address will be a routable/reachable address\n# that the polling host can connect to in order to run the client/server test.\ngrafana-address: localhost\ngrafana-port: 2003\n# The prefix of the stored tsdb data in graphite\ntsdb-download-prefix: bandwidth.download\ntsdb-upload-prefix: bandwidth.upload\niperf-servers: \n  - 172.17.0.3: azure\n  - 172.17.0.4: digitalocean\n  - 172.17.0.5: colo_datacenter_xyz\n# iperf-servers are the remote iperf servers getting polled\n# the key is the address getting polled\n# The value (after the colon )is the name that will show up in grafana \n```\n\nThe above example [config.yaml](config.yaml) file is included. The `iperf-servers:` in the config can also be DNS entries \nif using `-nocontainer` (name resolution not supported in the containers, happy to add the support if anyone wants it). \nThe `config.yaml` file either needs to be in the same directory as the binary or referenced with the flag `-config=path/config.yaml`.\n\nIf you prefer the CLI for configuration, here is an example doing so. **Note:** if there is a configuration file in the same directory,\nthe app will merge the `iperf-servers` endpoints between the CLI/ENVs and `config.yaml`, the rest of the configuration will default to the\nconfiguration file and then to the CLI and CLI defaults:\n\n**Note** -perf-servers is a pair `ip:name` (the name is for more meaningful descriptions\nin grafana/tsdb and mapping IPs to grafana with dots can make the prefix wonky). Multiple entries\nneed to be seperated by a colon ',' as exemplified below. DNS names instead of IP are\nalso perfectly valid to use as well. Alternatively, the name value `ip:name` you supply could\nbe the source node running the test. In some scenarios you could have a bunch of endpoints\ntesting to a single node you would want to record them to grafana as the source.\nRegardless, the name should be something meaningful to the data visualization in Grafana and\ndoesn't need to be the name of the host you are testing to.\n\n```shell\ncloud-bandwidth \\\n  -perf-servers 172.17.0.3:azure,172.17.0.4:digitalocean,172.17.0.5 \\\n   -test-length 3 -test-interval 30 \\\n   -grafana-address 192.168.1.100 \\\n   -grafana-port 2003 \\\n   -nocontainer \\\n   -debug\n```\n\n### Start polling\n\n- Now start the poller by dropping into the binaries directory and running the binary if on Linux. See the build section for compiling for other machine archs.\n\n- Docker or Podman are supported, but not required as long as iperf3 is installed you can pass the `-nocontainer` flag. \n\n- The app provides a sample of the bi-drectional bandwidth by testing both upload and download speeds between the server and poller.\n\nLinux:\n\n```sh\ncd ./binaries/linux/\n./cloud-bandwidth -config=config.yaml -debug\n```\n\nThe output will look something like this using the default `config.yaml` (if you started some iperf3 instances to test to):\n\n```shell\n./cloud-bandwidth -config=config.yml\nINFO[0005] Download results for endpoint 172.17.0.3 [azure] -\u003e 5020388 bps\nINFO[0010] Upload results for endpoint 172.17.0.3 [azure] -\u003e 4643267 bps\nINFO[0016] Download results for endpoint 172.17.0.4 [digitalocean] -\u003e 4353795 bps\nINFO[0021] Upload results for endpoint 172.17.0.4 [digitalocean] -\u003e 4619377 bps\nINFO[0026] Download results for endpoint 172.17.0.2 [colo_datacenter_xyz] -\u003e 4737081 bps\nINFO[0032] Upload results for endpoint 172.17.0.2 [colo_datacenter_xyz] -\u003e 4649928 bps\n```\n\nTo view the underlying commands being run such as the iperf polling and the writes to Carbon, simply add the `./cloud-bandwidth -debug` flag.\n\nHere is some example output of the app polling three endpoints:\n\n![](docs/images/cbandwidth-800.gif)\n\n### Run without containers\n\n- If you don't want to use containers at all, simply pass `-nocontainer`\n\n```shell\n./cloud-bandwidth -config=config.yml -nocontainer -debug\nINFO[0000] Running shell command -\u003e  [-c iperf3 -P 1 -t 5 -f K -p 5201 -c 172.17.0.3 | tail -n 3 | head -n1 | awk '{print $7}']\n```\n### Building the Binary\n\nYou can also of course run this using `go run cloud-bandwidth.go` directly. \n\nYou can also build binaries for your machine type. For other hardware architectures you can cross-compile in go. \nSupported types can be seen with `go tool dist list` and is as simple as the following example \n`GOOS=linux GOARCH=amd64 go build -o cloud-bandwidth -v cbandwidth.go` which will leave a binary in the working \ndirectory named `cloud-bandwidth`.\n\n- Build it yourself with [go](https://go.dev/doc/install):\n\n```shell\ngit clone https://github.com/nerdalert/cloud-bandwidth.git\ncd cloud-bandwidth\ngo build\n./cloud-bandwidth -h\n```\n\n### Custom Iperf3 image repo\n\nYou can also use your own iperf3 image with `-image`\n```shell\n./cloud-bandwidth -config=config.yml -image quay.io/networkstatic/iperf3 -debug\nINFO[0000] Running shell command -\u003e  [-c docker run -i --rm quay.io/networkstatic/iperf3 -P 1 -t 5 -f K -p 5201 -c 172.17.0.3 | tail -n 3 | head -n1 | awk '{print $7}']\n```\n\n### Netperf and Netserver\n\nNetperf/Netserver is an alternative bandwidth measuring tool. While the CLI output has always been\na little rough, it has interesting capabilities that Iperf does not. Specifically, netperf can measure \nCPU usage on the client and server side as well as run tests from multiple clients\nconcurrently. Iperf servers block any new tests if a test is in progress, while netserver/netperf will\nallow for the tests to run concurrently. This is especially attractive when looking to measure aggregate \nbandwidth through gateways/aggregation points.\n\nHere are the basics of the command:\n\n- Server side\n\n```shell\ndocker run  -it --rm --name=netserver -p 12865:12865 quay.io/networkstatic/netserver -D\n```\n\n- Client side\n```shell\n# Grab the IP address of the netserver started above\ndocker inspect --format \"{{ .NetworkSettings.IPAddress }}\" netserver\n172.17.0.5\n\n# Run a test against the netserver address\ndocker run  -it --rm quay.io/networkstatic/netperf -l 5 -H 172.17.0.5\n\nlatest: Pulling from networkstatic/netperf\nDigest: sha256:1bbcae9884ad9397b190e5a9d818efe7266c18cdcbb0abc44ba5086f062b4519\nStatus: Image is up to date for quay.io/networkstatic/netperf:latest\nMIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.17.0.5 () port 0 AF_INET : demo\nenable_enobufs failed: getprotobyname\nRecv   Send    Send\nSocket Socket  Message  Elapsed\nSize   Size    Size     Time     Throughput\nbytes  bytes   bytes    secs.    10^6bits/sec\n\n131072  16384  16384    10.00    11299.58\n```\n\nMore details on the container usage can be viewed at [nerdalert/netperf-netserver](https://github.com/nerdalert/netperf-netserver).\n\nTo run this with cloud-bandwidth, the following is an example (Netserver IP is `172.17.0.5` and the grafana display name would be `netserver-host`):\n\n```shell\n./cloud-bandwidth -perf-servers 172.17.0.5:netserver-host \\\n    -test-length 3 \\\n    -test-interval 30 \\\n    -grafana-address x.x.x.x \\\n    -grafana-port 2003 \\\n    -netperf \\\n    -debug \n```\n\n### Feedback!\n\n\nThats it! Patch, Fork, do whatever you want with it. Thanks to all the various open source projects used for this. \nKudos to [ESnet](http://software.es.net/iperf/) for re-rolling iperf into iperf3. It is really nice how the initialized \nchannel from client -\u003e server is reused for the reverse. It gives you bi-directional measurements without having to expose \n(or NAT) both endpoints, just the channel initializer. Feel free to open any issues if you run into anything. I will improve \nthings as time permits. Thanks!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnerdalert%2Fcloud-bandwidth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnerdalert%2Fcloud-bandwidth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnerdalert%2Fcloud-bandwidth/lists"}