{"id":19756679,"url":"https://github.com/esnet/netgraf","last_synced_at":"2025-04-30T12:30:28.795Z","repository":{"id":41135999,"uuid":"423694472","full_name":"esnet/netgraf","owner":"esnet","description":"Real-time network monitoring tool that uses machine learning","archived":false,"fork":false,"pushed_at":"2022-05-16T16:31:56.000Z","size":16979,"stargazers_count":4,"open_issues_count":0,"forks_count":4,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-05-09T12:11:47.974Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/esnet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-02T03:26:40.000Z","updated_at":"2023-11-30T14:31:20.000Z","dependencies_parsed_at":"2022-09-09T01:21:23.052Z","dependency_job_id":null,"html_url":"https://github.com/esnet/netgraf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2Fnetgraf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2Fnetgraf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2Fnetgraf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2Fnetgraf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/esnet","download_url":"https://codeload.github.com/esnet/netgraf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224207899,"owners_count":17273674,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T03:16:36.577Z","updated_at":"2024-11-12T03:16:37.141Z","avatar_url":"https://github.com/esnet.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/esnet/netgraf/blob/main/logo/logo.png\" width=\"1000%\" height=\"100%\" title=\"NetGraf logo\"\u003e\n\u003cp\u003e\n\n# NetGraf: Real-time Network Monitoring Tool that uses Machine Learning\n\nNetwork performance monitoring collects heterogeneous data such as network flow data to give an overview of network performance, and other metrics, necessary for diagnosing and optimizing service quality. However, due to disparate and heterogeneity, to obtain metrics and visualize entire data from several devices, engineers have to log into multiple dashboards.\nHere we present NetGraf: An end-to-end learning monitoring system that utilizes current monitoring tools and libraries to analyze the data and perform real-time anomaly finding. It can learn network performance baselines,  understand important data features, and can capture a holistic view of the networking infrastructure from a packet,flow, and device-level data by tapping into multiple  opensource solutions. NetGraf uses automated deployment with Ansible to provide real-time visualizations of various network health metrics from different multiple monitoring sources into a single dashboard for valuable insight on the network. With its machine learning libraries, NetGraf can learn baseline performance, and eventually the ability to optimize in a self-learning network. \n    \n\n## Guide and Documentation\n\n- NetGraf API documentations: \n- NetGraf Example Tutorials: https://github.com/esnet/daphne/tree/master/NetGraf-Ansible\n\n\n## Explanations\n- Website\n- Blog\n- Video\n\n## Installation Guide\nPlease ensure you have the IP addresses of the network devices, switches, routers ,servers, systems or hosts you intend to monitor. Netgraf has been tested on Chameleon cloud and Digital ocean instances. To get started, please feel free to use any enviroment or provider of your choice.\n\n\n## Installation Pre-requisite\n\n* Create a set of Virtual Machine(VM) Instances or network devices that you intend to monitor, and ensure they can communicate and pingable to each other.\n* For example you can spin up VMs Using Chameleon Cloud, Amazon EC2, Digital Ocean or any other cloud provider of your choice. \n* To reserve a node and lauch an instance on Chameleon, follow the steps provided [here](https://chameleoncloud.readthedocs.io/en/latest/getting-started/index.html#reserving-a-node).\n* Make one of your host or network devices your Control Node, and then others your Target Nodes.\n* Modify your hosts file to match the number of intended Target hosts by specifying the IP addresses.\n* Depending on your Controller node OS, Install ansible using the steps provided [here](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-with-pip).\n\nEnsure the devices or host machines are pingable to each other by creating a public key using the instruction [here](https://www.ssh.com/academy/ssh/keygen#what-is-ssh-keygen?), and then copy over your keys to your target nodes using:\n\n\tssh-copy-id user@0.0.0.0 \n\nOnce the installtion is complete and your environment is all set up, check the version of your ansible:\n\n    ansible --version\n\n\nClone the repository accordingly from a terminal whilst in your home directory with the following command -:\n\n    git clone https://github.com/esnet/netgraf.git\n\n    cd netgraf-main\n\n\nTest the connectivity between your nodes(Control Node and Target Nodes):\n\n    ansible all -m ping \n\n    ansible all -m ping -o\n\n\nInstall NetGraf using one push button:\n\n    ansible-playbook playbook.yml \n\n    time ansible-playbook playbook.yml -vvv\n\n\nOnce the NetGraf installtion is complete, view the PLAY RECAP for any errors and check the prometheus and grafana status.\n\n    sudo systemctl status prometheus\n\n \tsudo systemctl status grafana\n\n\nTo view all the active target nodes and metrics, type the following below. Please note that our controller IP address is 159.65.60.19. Please refer to our environment setup and IP address assignment [here](https://github.com/esnet/netgraf/blob/main/hosts):\n\n     http://159.65.60.19:9090/targets\n\n     http://159.65.60.19:9090/graph\n\nTo view the all-in-one NetGraf Dashboard on the controller node:\n\n     http://159.65.60.19:3000\n\n\nTo check the log if the collected metrics is streaming into the central database:\n\n    cd /var/log/promscale/\n  \n    tail -n 30 -f promscale.log promscale.log \n\nTo check login to the Central database:\n\n\tsudo su postgres -c psql\n\n\t\\l+\n\n\t\\c timescaledb_db\n\n\tselect * from  metric;\n\n\t\\q\n\n\t\\exit\n\nTo extract specific network related metrics and store them into the DB for analysis:\n\n\n\t bash monitoring_script.sh\n\nTo export collected data to a remote location using [rlcone](https://rclone.org/remote_setup/) - Google\n\n\trclone copy /opt/monitor_metrics netgraf_metrics:/metrics_data/ -v\n\n\n\nTo run the Machine learning component, change to the ml-pipepine directory and follow the instructions [here](https://github.com/esnet/netgraf/tree/main/ml-pipeline/)\n\n    cd ml-pipeline\n\n\n\n## Network and System Monitoring tools\n\nCurrently, NetGraf library supports the following Monitoring tools: \n\n**Supported Network and System Monitoring tools:**\n\n* ntopng\n* netdata\n* collectl\n* prometheus\n* perfSONAR\n* Confluo\n* zabbix\n* node_exporter\n* grafana\n\n\n\n\n## Features\n\nCurrently, NetGraf library contains the following features: \n\n**Machine Learning Models:**\n\n* LSTM,\n* SARIMA,\n* Exponential smoothing,\n* ARIMA,\n* Facebook Prophet,\n* FFT (Fast Fourier Transform),\n* DDCRNN,\n\n\n\n### Tests\n\nA [gradle](https://gradle.org/) setup works best when used in a python environment, but the only requirement is to have `pip` installed for Python 3+\n\nTo run all tests at once just run\n```bash\n./gradlew test_all\n```\n\nalternatively you can run\n```bash\n./gradlew unitTest_all # to run only unittests\n./gradlew coverageTest # to run coverage\n./gradlew lint         # to run linter\n```\n\n\n### Documentation\n\nTo build documantation locally just run\n```bash\n./gradlew buildDocs\n```\nAfter that docs will be available in `./docs/build/html` directory. You can just open `./docs/build/html/index.html` using your favourite browser.\n\n## Important Links\n- Docs: \n- Example Notebooks: \n- Tutorials: \n- pypi package: \n- Release notes:\n- Issue Logs: \n- Contribute:\n- Join Slack Community:\n- Other Resources: \n\n## Contact Us\nSee attached Licence to Lawrence Berkeley National Laboratory\nEmail: Mariam Kiran \u003cmkiran@es.net\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesnet%2Fnetgraf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesnet%2Fnetgraf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesnet%2Fnetgraf/lists"}