{"id":18254283,"url":"https://github.com/realvnf/coord-sim","last_synced_at":"2025-04-04T17:30:41.561Z","repository":{"id":38052194,"uuid":"172504885","full_name":"RealVNF/coord-sim","owner":"RealVNF","description":"Lightweight flow-level simulator for inter-node network and service coordination (e.g., in cloud/edge computing or NFV).","archived":false,"fork":false,"pushed_at":"2023-05-04T20:26:04.000Z","size":1309,"stargazers_count":57,"open_issues_count":0,"forks_count":27,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-07T13:06:13.720Z","etag":null,"topics":["coordination","edge-computing","flow","load-balancing","network","networkx","nfv","openai-gym","placement","python","scaling","scheduling","simpy","simulation","simulator"],"latest_commit_sha":null,"homepage":"https://coordination-simulation.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RealVNF.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null}},"created_at":"2019-02-25T12:49:02.000Z","updated_at":"2024-09-10T14:35:30.000Z","dependencies_parsed_at":"2022-09-20T04:14:54.331Z","dependency_job_id":null,"html_url":"https://github.com/RealVNF/coord-sim","commit_stats":{"total_commits":463,"total_committers":13,"mean_commits":35.61538461538461,"dds":0.5291576673866091,"last_synced_commit":"281f77f9f273cfd41536a66f3c0180aabc94f4b1"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealVNF%2Fcoord-sim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealVNF%2Fcoord-sim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealVNF%2Fcoord-sim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealVNF%2Fcoord-sim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RealVNF","download_url":"https://codeload.github.com/RealVNF/coord-sim/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223150742,"owners_count":17095959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coordination","edge-computing","flow","load-balancing","network","networkx","nfv","openai-gym","placement","python","scaling","scheduling","simpy","simulation","simulator"],"created_at":"2024-11-05T10:11:27.287Z","updated_at":"2024-11-05T10:11:29.371Z","avatar_url":"https://github.com/RealVNF.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/RealVNF/coord-sim.svg?branch=master)](https://travis-ci.com/RealVNF/coord-sim)\n\n# Simulation: Inter-node service coordination and flow scheduling\n\nSimulate flow-level, inter-node network coordination including scaling and placement of services and scheduling/balancing traffic between them.\n\n**Features**:\n\n- Simulate any given network topology with arbitrary node and link capacities and link delays\n- Simulate any given network service consisting of linearly chained SFs/VNFs\n- VNFs can specify arbitrary resource consumption as function of their load using Python modules. Also VNF delay can be specified individually and may be normally distributed.\n- Simulate network traffic in the form of flow arrivals at various ingress nodes with varying arrival rate, flow length, volume, etc according to stochastic distributions\n- Simple and clear interface to run algorithms for scaling, placement, and scheduling/load balancing of these incoming flows across the nodes in the network. Coordination within each node is out of scope.\n- Interface allows easy integration with OpenAI Gym to enable training and evaluating reinforcement learning algorithms\n- Collection of metrics like successful/dropped flows, end-to-end delay, resource consumption, etc over time. Easily extensible.\n- Discrete event simulation to evaluate coordination over time with SimPy\n- Graceful adjustment of placements: When VNFs are removed from a placement by an algorithm. Currently processing flows are allowed to finish processing before the VNF is completely removed (see PR [#78](https://github.com/RealVNF/coordination-simulation/pull/78) and [#81](https://github.com/RealVNF/coordination-simulation/pull/81)).\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/RealVNF/coord-sim/master/docs/realvnf_logo.png\" height=\"150\" hspace=\"30\"/\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/RealVNF/coord-sim/master/docs/upb.png\" width=\"200\" hspace=\"30\"/\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/RealVNF/coord-sim/master/docs/huawei_horizontal.png\" width=\"250\" hspace=\"30\"/\u003e\n\u003c/p\u003e\n\n## Projects Using coord-sim\n\n* [DeepCoord: Centralized Deep Reinforcement Learning (DRL) Approach](https://github.com/RealVNF/DeepCoord)\n* [Distributed DRL Approach](https://github.com/RealVNF/distributed-drl-coordination)\n* [B-JointSP: Centralized Heuristic Approach](https://github.com/CN-UPB/B-JointSP) (corresponding [adapter code](https://github.com/RealVNF/bjointsp-adapter))\n* [Two Fully Distributed Heuristic Approaches](https://github.com/CN-UPB/distributed-coordination)\n* [Simple baselines for comparison](https://github.com/RealVNF/baseline-algorithms)\n\nFeel free to open a pull request and add your own project if you use coord-sim!\n\n## Setup\n\nInstall with (ideally using [virtualenv](https://virtualenv.pypa.io/en/stable/)):\n\n```bash\npip install .\n# For dev install: pip install -e .\n```\n\n\n## Usage\n\nType `coord-sim -h` for help using the simulator. For now, this should print\n\n```\n$ coord-sim -h\nusage: coord-sim [-h] -d DURATION -sf SF [-sfr SFR] -n NETWORK -c CONFIG\n                 [-t TRACE] [-s SEED]\n\nCoordination-Simulation tool\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -d DURATION, --duration DURATION\n                        The duration of the simulation (simulates\n                        milliseconds).\n  -sf SF, --sf SF       VNF file which contains the SFCs and their respective\n                        SFs and their properties.\n  -sfr SFR, --sfr SFR   Path which contains the SF resource consumption\n                        functions.\n  -n NETWORK, --network NETWORK\n                        The GraphML network file that specifies the nodes and\n                        edges of the network.\n  -c CONFIG, --config CONFIG\n                        Path to the simulator config file.\n  -t TRACE, --trace TRACE\n                        Provide a CSV trace file to configure the traffic the\n                        simulator is generating.\n  -s SEED, --seed SEED  Random seed\n```\n\nYou can use the following command as an example (run from the root project folder)\n\n```bash\ncoord-sim -d 20 -n params/networks/triangle.graphml -sf params/services/abc.yaml -sfr params/services/resource_functions -c params/config/sim_config.yaml\n```\n\nThis will run a simulation on a provided GraphML network file and a YAML placement file for a duration of 20 timesteps.\n\n### Dynamic SF resource consumption\n\nBy default, all SFs have a node resource consumption, which exactly equals the aggregated traffic that they have to handle.\n\nIt is possible to specify arbitrary other resource consumption models simply by implementing a python module with a\nfunction `resource_function(load)` (see examples [here](https://github.com/RealVNF/coordination-simulation/tree/master/params/services/resource_functions)).\n\nTo use these modules, they need to be referenced in the service file:\n\n```\nsf_list:\n    a:\n      processing_delay_mean: 5.0\n      processing_delay_stdev: 0.0\n      resource_function_id: A\n```\n\nAnd the path to the folder with the Python modules needs to be passed via the `-sfr` argument.\n\nSee PR https://github.com/RealVNF/coordination-simulation/pull/78 for details.\n\n### Egress nodes\n\n- A node can be set to be a `Egress` node in the `NodeType` attribute of the network file\n- If some nodes are set as `Egress` then only the simulator will randomly choose one of them as the Egress node for each flow in the network\n- If some nodes are set to be Egress then once the flow is processed we check if for the flow, `current node == egress node` . If Yes then we depart , otherwise we forward the flow to the egress_node using the shortest_path routing.\n- **Todo**: Ideally the coordination algorithms should keep the path(Ingress to Egress) of the flow in view while creating the schedule/placement.\n\nSee [PR 137](https://github.com/RealVNF/coord-sim/pull/137) for details.\n\n### Conversion of real world traffic traces  \n\nReal World traffic traces are available at [sndlib](http://sndlib.zib.de/) under 'Dynamic traffic' at the left. They contain the data rate for every pair of node in a network for every 5 minutes for a timespan of six months. Available data formats are xml and another \"native sndlib format\". For usage in the simulator this data has to be converted into inter_arrival_mean. A script for that (which works with the xml files) you find here `coord-sim/params/convert_traces/convert_traces.py`. In the same folder you also find an example configuration for the script and an example data set for the first try.\n```sh\ncoord-sim/params/convert_traces$ tree\n.\n├── abilene_node_name_map.yaml\n├── convert_traces.py\n├── directed-abilene-zhang-5min-over-6months-ALL\n│   ├── demandMatrix-abilene-zhang-5min-20040302-1830.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040305-0150.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040411-0520.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040626-0345.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040630-2150.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040704-2020.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040808-0140.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040812-1415.xml\n│   ├── demandMatrix-abilene-zhang-5min-20040819-2305.xml\n│   └── demandMatrix-abilene-zhang-5min-20040907-0905.xml\n└── trace_xml_reader_config.yaml\n```\n\nThe folder `directed-abilene-zhang-5min-over-6months-ALL` contains 10 xml files from sndlib each standing for traffic in one 5min timespan.\nThe configuration you find in `trace_xml_reader_config.yaml`. It contains:  \n```yaml\nsource: \"directed-abilene-zhang-5min-over-6months-ALL\"\n# result_trace_filename: \u003c\u003e  # default  = f'{directory}_{_from}-{to}_trace.csv'\n# intermediate_result_filename: \u003c\u003e  # default  = result_trace_filename + \"_intermediate\n# _from: 0 # default 0\n# to: 100  # default None, means slice is [_from:]\nnode_name_map: abilene_node_name_map.yaml  # default None, means leave the names\nrun_duration: 100  # default 100\nscale_factor: 0.001  # default 0.001\nchange_rate: 2  # default 2\n#ingress_nodes:  # default None, means choose all nodes\n#  - pop0\n#  - pop1\n```\nParameter `source` points to the folder with the xml files. Execute:  \n```sh\ncoord-sim/params/convert_traces$ python3 convert_traces.py --config_file trace_xml_reader_config.yaml\n[...]\n23:20:54: 10  files in directory\n23:20:54: Chosen files: os.listdir(directed-abilene-zhang-5min-over-6months-ALL)[0:]\n[...]\n23:21:00: Written to directed-abilene-zhang-5min-over-6months-ALL_0-None_trace.csv. Last time step 1800\n[...]\n23:21:00: inter_arrival_mean range: 0.323815912931867, 36.447214465141315\n23:21:00: ... mean:  7.381947818616778\n23:21:00: ... median:  5.646327364198291\n23:21:00: ... std:  5.661472126922231\n```\nThe converted trace is written to `directed-abilene-zhang-5min-over-6months-ALL_0-None_trace.csv`. Reading the files takes most of the time. That's why the script writes some intermediate data to another csv file (in this case it is named `directed-abilene-zhang-5min-over-6months-ALL_0-None_intermediate.csv`). You can reuse it with different parameters by setting the `source` parameter to the filename of the intermediate. For example we want to include not all ingress nodes:\n`trace_xml_reader_config.yaml`:\n```yaml\nsource: directed-abilene-zhang-5min-over-6months-ALL_0-None_intermediate.csv\nresult_trace_filename: ing_pop0_pop1.csv\n[...]\ningress_nodes:  # default None, means choose all nodes\n  - pop0\n  - pop1\n```\nWe also give the resulting trace_file another filename to avoid overwriting:\n```sh\ncoord-sim/params/convert_traces$ python3 convert_traces.py --config_file trace_xml_reader_config.yaml\n```\nThe script will work on the data from the intermediate file. By default filenames are constructed from the directory the arguments `_from` and `to`. You also can assign filenames to to the intermediate and the results file:\n`trace_xml_reader_config.yaml`:\n```yaml\nsource: \"directed-abilene-zhang-5min-over-6months-ALL\"\nintermediate_result_filename: directed-abilene-zhang-5min-over-6months-ALL_0-None_intermediate.csv\nresult_trace_filename: ing_pop0_pop1.csv\n[...]\n```  \nSince a batch from sndlib contains so many files you can choose a sample of them with arguments `_from` and `to`, which defines a slice. The script calls: `os.listdir(source)[_from:to]` if source is a directory. That way you can limit the number of files to read. If `source` is set to an intermediate file it will be also sliced according to those parameters.  \nThe node names in our network files differ from those in sndlib. To change them a yaml file is assigned. In the above config example parameter `node_name_map` was set to `abilene_node_name_map.yaml`, which looks like this:\n```yaml\n# defines how to rename nodes (from keys to values). If a node is set to null it will be removed from the\n# dataframe. If a node is not mentioned in the yaml it will be ignored, the name will be kept.\nATLAM5: null  # this node is removed and does not appear even in the intermediate\nATLAng: pop9  # renamed from ATLAng to pop9\nCHINng: pop1\nDNVRng: pop6\nHSTNng: pop8\nIPLSng: pop10\nKSCYng: pop7\nLOSAng: pop5\nNYCMng: pop0\nSNVAng: pop4\nSTTLng: pop3\nWASHng: pop2\n```\nSave plots of the `data_rate` or the `inter_arrival_mean`:\n```sh\ncoord-sim/params/convert_traces$ python3 convert_traces.py --config_file trace_xml_reader_config.yaml --save_plots data_rate inter_arrival_mean\ncoord-sim/params/convert_traces$ tree\n.\n├── abilene_node_name_map.yaml\n├── convert_traces.py\n├── directed-abilene-zhang-5min-over-6months-ALL\n│   ├── demandMatrix-abilene-zhang-5min-20040302-1830.xml\n│   ├── [...]\n│   └── demandMatrix-abilene-zhang-5min-20040907-0905.xml\n├── directed-abilene-zhang-5min-over-6months-ALL_0-None_intermediate.csv\n├── directed-abilene-zhang-5min-over-6months-ALL_0-None_trace.csv\n├── directed-abilene-zhang-5min-over-6months-ALL_0-None_trace_data_rate.png            \u003c---\n├── directed-abilene-zhang-5min-over-6months-ALL_0-None_trace_inter_arrival_mean.png   \u003c---\n├── directed-abilene-zhang-5min-over-6months-ALL_0-None_trace_meta.yaml\n└── trace_xml_reader_config.yaml\n```\nSave plots as pdf:\n```sh\ncoord-sim/params/convert_traces$ python3 convert_traces.py --config_file trace_xml_reader_config.yaml --save_plots data_rate inter_arrival_mean --plot_format pdf\n```\nShow plots in the end of the script by calling plt.show():\n```sh\ncoord-sim/params/convert_traces$ python3 convert_traces.py --config_file trace_xml_reader_config.yaml --plot data_rate inter_arrival_mean\n```\nFor more information look at the doctrings in the script or the comments in the example config.   \n\n\n#### Overall abilene intermediate file\n\nWe have an intermediate csv-file [overall_abilene_intermediate.csv](https://github.com/RealVNF/coord-sim/blob/master/params/convert_traces/overall_abilene_intermediate.csv), which contains the whole abilene batch from sndlib with 48 thousand time step. It is recommended to use it for producing traces for the abilenme network by setting the `source` parameter in config to it. \n\n## Create Episode Animations  \n  \nAnother way to analyse results is to create animation, which shows the placement in a single test episode changing over time. Use command `animation` for that:  \n  \nCreate animation from the first test directory from a results directory: `animation --results_dir \u003c\u003e`\nTo show all available test directories in a results directory: `animation --results_dir \u003c\u003e --show_tests`  \nCreate animation from test directory: `animation --test_dir \u003c\u003e`  \nShow animation in the end (by calling plt.show()): `animation --test_dir \u003c\u003e --show`  \nSave animation as html video: `animation --test_dir \u003c\u003e --save \u003cpossible values: html, git, both\u003e`  \nYou can limit the amount of data to process by setting `--sample_rate`: `animation --test_dir \u003c\u003e --sample_rate 5`  Default is 1\nThus for example every fifth point in time will be taken into the animation. Resolution will be worse, of course. But it is useful if the scenario is to big\nSet interval between frames: `animation --test_dir \u003c\u003e --interval 100`  Default is 100\n  \nCreate an animation not from cli, but as python code with the PlacementAnime class:  \n```py\npa = PlacementAnime(test_dir)\npa.create_animation()\npa.animation # animation object\npa.fig # figure object\npa.ax # Axis object (network, placement etc)\npa.ingress_traffic_ax # Axis object (ingress_traffic)\n```\n\nMaybe you will need the tkinter module installed for that: `sudo apt install python3-tk`\n\n## LSTM Traffic Prediction\n\nThe simulator has an LSTM module to predict traffic based on the traffic traces mentioned above. \n\nThe LSTM module must be trained separately, to do so, `lstm_prediction` must be enabled in the simulator config file. Additionally, `lstm_weights` must be set to the directory where the desired location to save the weights within the current working directory. This will also be used during the running of the simulator to load the weights.\n\nTo train the LSTM nerual network:\n\nUse the `lstm-predict` module similar to the example below. The module takes one argument — the simulator config file that will be used during the actual simulation.\n```\nlstm-predict -c \u003cPATH-TO-CONFIG-FILE\u003e\n```\nThis will train the LSTM network based on the trace specified in the `trace_path` in the config file, then save the weights to the weights directory specified in the config file. \n\nAfterwards, use the same configuration file when using the simulator to make sure that weights are correctly loaded in the simulator later on.\n\n\n## Tests\n\n```bash\n# style check\nflake8 src\n\n# tests\nnose2\n```\n\n## Acknowledgement\n\nThis project has received funding from German Federal Ministry of Education and Research ([BMBF](https://www.bmbf.de/)) through Software Campus grant 01IS17046 ([RealVNF](https://realvnf.github.io/)).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/software_campus.png\" width=\"200\"/\u003e\n  \u003cimg src=\"docs/BMBF_sponsored_by.jpg\" width=\"250\"/\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frealvnf%2Fcoord-sim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frealvnf%2Fcoord-sim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frealvnf%2Fcoord-sim/lists"}