{"id":27294499,"url":"https://github.com/nerdalert/ts-net","last_synced_at":"2025-04-11T22:53:16.593Z","repository":{"id":34054110,"uuid":"37839032","full_name":"nerdalert/ts-net","owner":"nerdalert","description":"Time Series Network Metric Collection and Visualization Example Using Docker, InfluxDB 0.9+ and Grafana v2+","archived":false,"fork":false,"pushed_at":"2016-09-02T18:01:17.000Z","size":25,"stargazers_count":6,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T22:53:12.013Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nerdalert.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-22T05:31:39.000Z","updated_at":"2023-03-25T17:30:25.000Z","dependencies_parsed_at":"2022-07-14T03:10:41.569Z","dependency_job_id":null,"html_url":"https://github.com/nerdalert/ts-net","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fts-net","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fts-net/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fts-net/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nerdalert%2Fts-net/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nerdalert","download_url":"https://codeload.github.com/nerdalert/ts-net/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248492951,"owners_count":21113162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-11T22:53:15.961Z","updated_at":"2025-04-11T22:53:16.587Z","avatar_url":"https://github.com/nerdalert.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Time Series Network Metric Collection and Visualization Example Using Docker, InfluxDB and Grafana\n\n### Overview\n\nThis is the same functionality as [networkstatic/cloud-bandwidth](https://github.com/nerdalert/cloud-bandwidth) but using InfluxDB and Grafana only (not whisper, carbon etc).\n\n* InfluxDB time series DB and Grafana. The advantage is less moving parts and some powerful aggragate functions from InfluxDB along with data rollups and some other key differentiators from other TSDBs. It uses the Graphite plugin to enable simple time series entries. \n* [Grafana 2.0+](http://docs.grafana.org/v2.0/guides/whats-new-in-v2/) which has recently been re-rolled in Go. InfluxDB 0.9+ changed the API significantly so Grafana required significant work also. The combination still misses a few things but overall is close to being super usable.\n\nIf this all looks foreign:\n\n* The Docker free training site is great for all levels of users at [Docker Training](https://training.docker.com)\n* If unfamiliar with Grafana, check out [Grafana Play](http://play.grafana.org/?theme=dark).\n* If you prefer a GUI to get to know Docker, check out [Kitematic by Docker](https://www.docker.com/docker-kitematic)\n\nI only have the integration tests migrated. I will add the `run.sh` script to use it for real bandwidth next weekend.\n\n### QuickStart Demo\n\n```\ngit clone https://github.com/nerdalert/ts-net.git\ncd ts-net/\ndocker-compose -f run_demo.yml up\n```\n\nThen point a browser to `http://\u003cDOCKER_IP\u003e:3000`\n\nTo stop and remove the demo containers running with compose simply run the following in the same directory as the .yml file:\n\n```\ndocker-compose -f run_demo.yml kill\ndocker-compose -f run_demo.yml rm -f\n```\n\n### Longer Instructions\n\nTo run the tests from a Docker Machine VM use the following instructions. If you have a `native` instance running you simply use the native keyword with the `run.sh` wrapper shown later in the readme. In a new terminal, your docker machine should have at least a virtualbox machine defined.\n\n*Note*: The machine (VM) naming needs to match the pattern of driver-foo. 'driver' being one of the dozen or so docker machine drivers and foo being whatever you want to end it with. Both of those with a `-` delimiting them. For example, vmwarefusion-machine, \n\nHere are the driver names as defined by docker machine:\n\n* amazonec2\n* azure\n* digitalocean\n* google\n* openstack\n* rackspace\n* softlayer\n* virtualbox\n* vmwarefusion\n* vmwarevcloudair\n* vmwarevsphere\n\nYou can test both source and destination polling for testing on the same machine. Here we will use the virtualbox driver as an example:\n\n*This is only required if you aren't running a native instance of Docker on Linux for example. For the remote Cloud test you would use Docker-Machine unless the provider offers native Docker instances via the Docker remote API (e.g. Joyent).*\n\nIf an instance is not already running, start one with a `docker-machine create`:\n```\n$ docker-machine create \\\n    --driver virtualbox \\\n    virtualbox-machine\n```\n \nList the machines:\n\n```\n$ docker-machine ls\nNAME                   ACTIVE   DRIVER         STATE     URL                         SWARM\nvirtualbox-machine     *        virtualbox     Running   tcp://192.168.99.101:2376\n```\n\nA simple way to get the external address is:\n\n```\n$ docker-machine ip vmwarefusion-machine\n172.16.166.157\n```\n\nThen point a browser to `http://\u003cDOCKER_IP\u003e:3000`\n\nIf you dont see data in the browser but instead see `datapoints outside time range` then change your time to the past 12 hours and you should see the data. Javascript is notoriously finicky with timezones. You can explicitly set one or use UTC. I have run into this a couple of times with grafana but havent figured out the pattern yet so will patch the config when I do.\n\n\n### Clone the repo\n\n```\ngit clone https://github.com/nerdalert/ts-net.git\ncd ts-net\n```\n\n### Create a Docker Machine\n\n```\ndocker-machine create --driver virtualbox virtualbox-machine\neval \"$(docker-machine env virtualbox-machine)\"\n\ndocker-machine ls\n# NAME                   ACTIVE   DRIVER         STATE     URL                         SWARM\n# virtualbox-machine     *        virtualbox     Running   tcp://192.168.99.101:2376\n```\n\n### Run the demo\n\nPass `-f` and the demo yml file to docker-compose which tells it to use `run_demo.yml` configurations rather then the default `docker-compose.yml` that is in the same directory. Again, the `-f run_demo.yml` is **only** for running the demo. Later when running with real data you simply use the defaults with `docker-compose up`. Note you have to use `-d` since the `init` container that creates the initial InfluxDB database will exit once it completes its API operations. If you dont use daemon mode it will shutdown the rest of the containers in the docker-compose group.\n \n```\ndocker-compose -f run_demo.yml up -d\n```\n\nSince in `-d` daemon mode, the containers will be run in the background. To see logs of the ops run:\n\n```\n$ docker-compose logs\n\n```\n\nTo attach to a container and open a bash shell (like ssh but cooler imo :) first get the container IDs from a `docker ps`:\n\n```\n$ docker ps\nCONTAINER ID        IMAGE                  COMMAND                CREATED              STATUS              PORTS                                                                                                                               NAMES\nbefe91ae530d        tsnet_demo             \"./generate_test_dat   About a minute ago   Up About a minute                                                                                                                                       tsnet_demo_1\nb58938c88098        tsnet_grafana          \"/usr/sbin/grafana-s   About a minute ago   Up About a minute   0.0.0.0:3000-\u003e3000/tcp                                                                                                              tsnet_grafana_1\n52c3bcb00e50        tsnet_influxdb         \"/opt/influxdb/influ   About a minute ago   Up About a minute   0.0.0.0:2003-\u003e2003/tcp, 0.0.0.0:8083-\u003e8083/tcp, 0.0.0.0:8086-\u003e8086/tcp, 0.0.0.0:8125-8126-\u003e8125-8126/tcp, 0.0.0.0:32947-\u003e8125/udp   tsnet_influxdb_1\nca959836fb92        networkstatic/iperf3   \"iperf3 -s\"            12 hours ago         Up 12 hours         0.0.0.0:5201-\u003e5201/tcp                                                                                                              bandwidth_agent\n```\n\nPass the CID (Container ID)\n```\ndocker exec -i -t b58938c88098 bash\n# Which drops you into a bash shell inside of the container namespsace that looks and feels like a standalone Linux system and associated commands for the distro\nroot@grafana:/usr/share/grafana# ps -eaf\nUID        PID  PPID  C STIME TTY          TIME CMD\nroot         1     0  0 09:43 ?        00:00:00 /usr/sbin/grafana-server --config /opt/conf/grafana.ini\nroot        13     0  0 09:46 ?        00:00:00 bash\nroot        19    13  0 09:46 ?        00:00:00 ps -eaf\n```\n\n\n### View The Grafana Dashboard\n\nNow point your browser to the grafana UI and see the data being graphed. Get the ip address with:\n\n```\ndocker-machine ip virtualbox-machine\n# 192.168.99.101\n\n# or docker-machine ls and see the API ip:port '192.168.99.101:2376'\n\ndocker-machine ls\n# NAME                   ACTIVE   DRIVER         STATE     URL                         SWARM\n# virtualbox-machine     *        virtualbox     Running   tcp://192.168.99.101:2376\n```\n\nThe docker-compose and dockerfile instruct grafana to use port 3000 `3000`\n\n`http://\u003cMACHINE_IP\u003e:3000`\n\nYou will begin to see the following be generated:\n\n![](http://networkstatic.net/wp-content/uploads/2015/06/Grafana-InfluxDB-Time-Series-Dashboard-Sm.jpg)\n\nTo reiterate these are mock values being written to the TSDB using `docker-compose -f run_demo.yml up`. \n\nOnce done with the demo stop and recreate the containers using the default yml file.\n\n```\ndocker-compose -f run_demo.yml kill\ndocker-compose -f run_demo.yml rm -f\n```\n\nTo dispose of the current state of the containers (remove container/volumes and recreate them) simply paste:\n \n```\ndocker-compose -f run_demo.yml rm -v -f  \\\n    \u0026\u0026 docker-compose -f run_demo.yml  build \\\n    \u0026\u0026 docker-compose -f run_demo.yml up -d\n```\n\n### View InfluxDB Raw Series Data\n\nYou can go to the InfluxDB admin page whose port is exposed by the compose `yml` file.\n \n`Example url: http://192.168.99.100:8083/`\n\nThe IP is the docker host/machines IP.\n\nThen login -\u003e explore data (a DB named graphite should be there if the compose demo is running):\n\nIn the query box use `select * from /.*/ limit 10` to query the last 10 data points in all of the series.\n\nIt should look something like this:\n\n![](http://networkstatic.net/wp-content/uploads/2015/06/InfluxDB-Query.jpg)\n\n*note* I need to rename the time series as upload/download are not unique. I think its just a different opinion between whisper db and influx.\n\n### Measuring Real Bandwidth\n\nI need to change the run.sh script around to use InfluxDB. I will one night this week :) \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnerdalert%2Fts-net","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnerdalert%2Fts-net","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnerdalert%2Fts-net/lists"}