{"id":18513017,"url":"https://github.com/vmware/network-insight-sdk-generic-datasources","last_synced_at":"2025-08-16T05:34:03.291Z","repository":{"id":44148579,"uuid":"190723591","full_name":"vmware/network-insight-sdk-generic-datasources","owner":"vmware","description":"SDK to help add support for generic switches and routers not supported in Network Insight","archived":false,"fork":false,"pushed_at":"2023-10-31T08:32:46.000Z","size":390,"stargazers_count":15,"open_issues_count":7,"forks_count":11,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-09T22:16:24.499Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-07T10:08:14.000Z","updated_at":"2023-12-28T03:41:17.000Z","dependencies_parsed_at":"2024-11-06T15:43:48.704Z","dependency_job_id":"ebff53bf-72d0-480f-905a-134a2abf04af","html_url":"https://github.com/vmware/network-insight-sdk-generic-datasources","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vmware/network-insight-sdk-generic-datasources","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fnetwork-insight-sdk-generic-datasources","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fnetwork-insight-sdk-generic-datasources/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fnetwork-insight-sdk-generic-datasources/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fnetwork-insight-sdk-generic-datasources/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmware","download_url":"https://codeload.github.com/vmware/network-insight-sdk-generic-datasources/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fnetwork-insight-sdk-generic-datasources/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270670990,"owners_count":24625671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T15:36:11.954Z","updated_at":"2025-08-16T05:34:03.265Z","avatar_url":"https://github.com/vmware.png","language":"Python","readme":"\nnetwork-insight-sdk-generic-datasources\n=======================================\n\nSDK is written to support as many physical devices so that vRNI can consume\nnetwork information in defined format. Network information includes Router Interfaces, VRFs, Switch Ports,\nRoutes, etc. SDK can connect to physical device and execute commands. Output of command is then used to create\nfiles in specific format (eg. CSV). Files generated are then bundled in ZIP format which can then be\nfed into vRNI. SDK generates files compatible to vRNI version 4.2.0 onwards. Driver in SDK is governed by YAML configuration file.\n\n1. [ Environment Requirements ](#environment)\n2. [ Dependencies ](#dependencies)\n3. [ Launch SDK ](#launch-sdk)\n4. [ Device Configuration File ](#zipfile)\n5. [ YAML Configuration ](#yaml-configuration)\n6. [ Parser Definition ](#parser-definition)\n7. [ Table Joiner ](#table-joiner)\n8. [ Example ](#example)\n9. [ Uploading Output Zipfile ](#vrni-api-ref)\n10. [ Running SDK and Uploading ](#run-and-upload)\n\n\u003ca name=\"environment\"\u003e\u003c/a\u003e\n## 1. Environment Requirements\n* Linux distributions like Ubuntu, CentOS, Fedora, etc.\n* Pre-requisite packages/tools should be installed:\n    * git\n    * python 3.5 onwards\n\n\u003ca name=\"dependencies\"\u003e\u003c/a\u003e\n## 2. Dependencies\n* netmiko\n* requests\n* pyyaml\n\nInstall dependencies using following command.\n```\npip install --src . -r requirements.txt\n```\n\n\n\u003ca name=\"launch-sdk\"\u003e\u003c/a\u003e\n## 3. Launch SDK\n\nAfter cloning this project and add project folder to PYTHONPATH. Then run following command to run SDK.\n```shell\n$ python ./network_insight_sdk_generic_datasources/main.py -d \u003cdevice\u003e -m \u003cmodel\u003e -s \u003cdevice_type\u003e -i \u003cip-address\u003e -u \u003cusername\u003e -p \u003cpassword\u003e -o \u003coutput.zip\u003e\n```\n\n\u003e Command parameter explanation\n- -d = the device which you want to run\n- -m = model of device\n- -s = List of device type as defined in network-insight-sdk-generic-datasources/connection/device_type.py\n- -i = ip address or fqdn\n- -u = username\n- -p = password\n- -o = Output zip file, for upload to vRNI\n\nExample\n```\n$ git clone https://github.com/vmware/network-insight-sdk-generic-datasources.git\n$ cd network-insight-sdk-generic-datasources\n$ pip install --src . -r requirements.txt\n$ export PYTHONPATH=\"$PYTHONPATH:../network-insight-sdk-generic-datasources\"\n$ python ./network_insight_sdk_generic_datasources/main.py -d cisco -m n5k -s CISCO_IOS -i 10.1.1.1 -u test -p test -o cisco-n5k-10.1.1.1.zip\n\n```\n\n\u003ca name=\"zipfile\"\u003e\u003c/a\u003e\n## 4. Device Configuration File\nColumns in each CSV file represents attributes of entity.\n\nNOTE: General Guideline is to use double quotes for each value in a cell.\nSpecial characters allowed for any data of type string except defined values. Accepted special character are\nas follows.\n* _ Underscore\n* \\- Hyphen\n* : Colon\n* . Period\n* \\ Back Slash\n* / Forward Slash\n\n* switch.csv - contains switch information. MANDATORY\n\nColumn Name    | Mandatory / Optional | Description                    | Accepted Value\n---------------| -------------------- | ------------------------------ | --------------\nipAddress/fqdn | mandatory            | ipAddress or FQDN of switch    |\nname           | mandatory            | name of the switch             |\nserial         | optional             | serial of the switch           |\nos             | optional             | operating system of the switch |\nmodel          | optional             | model of the switch            |\nvendor         | optional             | vendor of the switch           |\nhostname       | mandatory            | hostname of the switch         |\nhaState        | optional             | redundant state of the switch  | ACTIVE, STANDBY\n\n* switch-ports.csv - contains all the switch ports and their attributes. MANDATORY\n\nColumn Name           | Mandatory / Optional | Description                    | Accepted Value\n----------------------| -------------------- | ------------------------------ | --------------\nname                  | mandatory            | name of switch port            |\nvlans                 | optional             | vlans                          | Comma separated Integer values (Note: use double quotes)\naccessVlan            | optional             | accessVlan of switch port      | Integer Value\nmtu                   | optional             | mtu                            | Integer Value\ninterfaceSpeed        | optional             | interface speed                | Integer Value in bits per second\noperationalSpeed      | optional             | operational speed              | Integer Value in bits per second\nadministrativeStatus  | mandatory            | administrative status          | UP, DOWN\noperationalStatus     | mandatory            | operational status             | UP, DOWN\nhardwareAddress       | optional             | physical / mac address         | Mac Address for format(MM:MM:MM:SS:SS:SS or MM-MM-MM-SS-SS-SS or MMM.MMM.SSS.SSS)\nduplex                | optional             | duplex                         | FULL, HALF, AUTO\nconnected             | mandatory            | connected                      | TRUE, FALSE\nswitchPortMode        | mandatory            | switch port mode               | ACCESS, TRUNK, OTHER\n\n* port-channels.csv - contains all the port channel (bundled switch ports) and their attributes. OPTIONAL\n\nColumn Name           | Mandatory / Optional | Description                    | Accepted Value\n----------------------| -------------------- | ------------------------------ | --------------\nname                  | mandatory            | name of the port channel       |\nvlans                 | optional             | vlans for Port Channel         | Comma separated Integer values (Note: use double quotes)\nmtu                   | optional             | mtu                            | Integer Value\ninterfaceSpeed        | optional             | interface speed                | Integer Value in bits per second\noperationalSpeed      | optional             | operational speed              | Integer Value in bits per second\nadministrativeStatus  | mandatory            | administrative status          | UP, DOWN\noperationalStatus     | mandatory            | operational status             | UP, DOWN\nhardwareAddress       | optional             | physical / mac address         | Mac Address for format(MM:MM:MM:SS:SS:SS or MM-MM-MM-SS-SS-SS or MMM.MMM.SSS.SSS)\nduplex                | optional             | duplex                         | FULL, HALF, AUTO\nconnected             | mandatory            | connected                      | TRUE, FALSE\nswitchPortMode        | mandatory            | switch port mode               | ACCESS, TRUNK, OTHER\nactivePorts           | optional             | active switch ports            | Interfaces defined in switch-ports.csv\npassivePorts          | optional             | passive switch ports           | Interfaces defined in switch-ports.csv\n\n* vrfs.csv - contains all vrfs of the switch/router. MANDATORY\n\nColumn Name           | Mandatory / Optional  | Description                    | Accepted Value\n----------------------| --------------------- | ------------------------------ | --------------\nname                  | mandatory             | name of the vrf                |\n\n* router-interfaces.csv - contains all the router interfaces and their attributes. MANDATORY\n\nColumn Name           | Mandatory / Optional | Description                    | Accepted Value\n----------------------| -------------------- | ------------------------------ | --------------\nname                  | mandatory            | name of the router interface   |\nvrf                   | mandatory            | vrf                            | VRFs defined in vrfs.csv\nvlan                  | optional             | vlan                           | Integer Value\nipAddress             | mandatory            | ip address of router interface | IP address in CIDR format\nmtu                   | optional             | Mtu                            | Integer Value\ninterfaceSpeed        | optional             | Interface Speed                | Integer Value in bits per second\noperationalSpeed      | optional             | Operational Speed              | Integer Value in bits per second\nadministrativeStatus  | mandatory            | Administrative Status          | UP, DOWN\noperationalStatus     | mandatory            | Operational Status             | UP, DOWN\nhardwareAddress       | optional             | physical / mac address         | Mac Address for format(MM:MM:MM:SS:SS:SS or MM-MM-MM-SS-SS-SS or MMM.MMM.SSS.SSS)\nduplex                | optional             | Duplex                         | FULL, HALF, AUTO\nconnected             | mandatory            | connected                      | TRUE, FALSE\nloadBalancedProtocol  | optional             | Load Balanced Protocol         | VRRP, GLBP, HSRP, VARP, OTHER\nloadBalancedStatus    | optional             | Load Balanced Status           | 1. VRRP -\u003e INITIALIZE, MASTER, BACKUP. 2. GLBP -\u003e ACTIVE, DISABLED, LISTEN, SPEAK, STANDBY, INITIAL. 3. HSRP -\u003e ACTIVE, INITIAL, LEARN, LISTEN, STANDBY, SPEAK. 4.VARP -\u003e ACTIVE. 5. OTHER -\u003e  ACTIVE, STANDBY\nloadBalancedIpAddress | optional             | Load balanced IP Address       | Load Balanced Protocol IP Address\n\n* routes.csv - contains all routes. MANDATORY\n\nColumn Name           | Mandatory / Optional  | Description                    | Accepted Value\n----------------------| --------------------- | ------------------------------ | --------------\nname                  | mandatory             | name of route                  | Generally, name is same as network\nnetwork               | mandatory             | network of route               | Network IP in CIDR Format\nnextHop               | mandatory             | next hop of route              | Use IP Address or DIRECT. If there is no nextHop ip address for DIRECT routeType then use DIRECT.\nrouteType             | mandatory             | route type eg. static, dynamic | Use Direct or any other routing protocol like OSPF, BGP, Static, etc.\ninterfaceName         | mandatory             | interface name                 |\nvrf                   | mandatory             | vrf                            | VRFs defined in vrfs.csv\n\n* peer-devices.csv - contains list of redundant devices with respect the switch. OPTIONAL\n\nColumn Name           | Mandatory / Optional  | Description                    | Accepted Value\n----------------------| --------------------- | ------------------------------ | --------------\npeerIpAddress         | mandatory             | peer device ip address         |\npeerHostname          | mandatory             | peer device hostname           |\n\n* neighbors.csv - contains list of LLDP/CDP neighbors. OPTIONAL\n\nColumn Name           | Mandatory / Optional  | Description                    | Accepted Value\n----------------------| --------------------- | ------------------------------ | --------------\nlocalInterface        | mandatory             | local switch port name         | Interfaces defined in switch-ports.csv\nremoteDevice          | mandatory             | remote device ip/fqdn          |\nremoteInterface       | mandatory             | remote device interface        |\n\n* mac-address-table.csv - contains mac address table. OPTIONAL\n\nColumn Name           | Mandatory / Optional  | Description                    | Accepted Value\n----------------------| --------------------- | ------------------------------ | --------------\nmacAddress            | mandatory             | mac address                    | Mac Address for format(MM:MM:MM:SS:SS:SS or MM-MM-MM-SS-SS-SS or MMM.MMM.SSS.SSS)\nvlan                  | mandatory             | vlan                           | VLAN-ID in Integer value\nswitchPort            | mandatory             | local switch port name         | Interfaces defined in switch-ports.csv\n\n* l2bridges.csv - contains list of layer 2 (vlan) bridges. OPTIONAL\n\nColumn Name           | Mandatory / Optional  | Description                    | Accepted Value\n----------------------| --------------------- | ------------------------------ | --------------\nname                  | mandatory             | name of layer 2 bridge         |\nvlans                 | mandatory             | all the vlans of bridge        | Comma separated Integer values (Note: use double quotes)\n\n\u003ca name=\"yaml-configuration\"\u003e\u003c/a\u003e\n## 5. YAML Configuration\n\nYAML configuration is defined in sections.\n1. Workloads - Contains workload definition which needs to be run. Workloads defined in top are executed first and one\ndefined in bottom is executed last.\n* Each workload definition has following items.\n    * table_id - refer result with an id.\n    * command - command to execute\n    * block parser - Parser to parse blocks\n    * parser - to parse each block, in any, otherwise full command output\n    * pre_post_processor - If there is any custom handling required before and after parsing then it can be defined.\n    * reuse_command - If command output needs to be processed differently then instead of re-executing the same command\n      we can reuse already stored command output.\n    * reuse_tables - Already created tables can be reused as per requirement.\n    * reuse_table_processor - Define class which can operate on listed reuse_tables.\n2. Table Joiner - Used for joining table. More information below.\n3. Result writer - Used to write table, reference with table_id, to csv file.\n    * Result writer accepts generation directory parameter which tells in which directory csv files to generate.\n4. Package Handler - Used to package files into a zip format.\n    * Package Handler accepts path parameter which tells which directory to compress as zip.\n\n\n\u003ca name=\"parser-definition\"\u003e\u003c/a\u003e\n## 6. Parser Definition\n\nThere are several kinds of parsers defined broadly defined in two categories\n* Block Parsers - To parse command output where there is definite pattern of blocks reappearing every time.\n    * SimpleBlockParser - Parses block if newline is present\n    * LineBasedBlockParser - Similar to SimpleBlockParser but here you can specify line pattern (regex) reappearing.\n        * Arguments\n            * line_pattern\n    * PatternBasedBlockParser - Similar to LineBasedBlockParser but here you can specify start_pattern and end_pattern\n        * Arguments\n            * start_pattern\n            * end_pattern\n    * GenericBlockParser - It is wrapper of all other Block Parsers. Accepts parameter same as other block parsers.\n        * Arguments -\n            * line_pattern\n            * start_pattern and end_pattern\n\n* Text Parsers - Each blocks has fields which needs to be parsed.\n    * Horizontal Table parser - Parses block in tabular format and creates list of key values.\n        * Arguments\n            * skip_head - Number of lines to skip from start(top) of block\n            * skip_tail - Number of lines to skip from end(bottom) of block\n            * header_keys - Define Column headers names\n    * Vertical Table parser - Parses data in vertical data format and creates list of key values.\n        * Arguments\n            * skip_head - Number of lines to skip from start(top) of block\n            * skip_tail - Number of lines to skip from end(bottom) of block\n            * delimiter - Field delimiter. Default is colon\n    * Generic Text Parser - Can parse block using regex pattern. Regex pattern must have group to select values.\n        * Arguments - Contains variable arguments where argument name is the key to regex group value.\n        For example, if we want to parse mtu value from text like `MTU 1500 Bytes`. In that case, we would define\n        argument as `mtu: MTU (.*) Bytes` where `mtu` is key and regex would be `MTU (.*) Bytes`. Regex group value is\n        surrounded with parenthesis which will be parsed.\n\nNOTE: All the parsers produce output as list of dictionaries.\n\n\n\n\u003ca name=\"table-joiner\"\u003e\u003c/a\u003e\n## 7. Table Joiner\n\nParsers create list of key value pairs which is logically in tabular format.\nThere are scenarios where we need to join two table having same column values.\nTable joiner accept parameter as source table, source column, destination table and destination column.\nSuch table join can be considered as RIGHT join denoted by source_table RIGHT JOIN destination_table.\n\n* SimpleTableJoiner - a table joiner which is able to join two tables (list for key value pairs) into a single table.\n\nTable Joiner configuration accept following parameters.\n  * name: Name of table joiner to be used.\n    For example, joiner.table_joiner.SimpleTableJoiner\n  * source_table: Source table for joining. Source table can be referred using table_id which is already defined in\n    command list\n  * destination_table: Destination table for joining. Destination table can be referred using table_id which is already defined in\n    command list\n  * source_column: Column from source table\n  * destination_column: Column from destination table\n  * joined_table_id: table id for referencing final joined table\n\n\u003ca name=\"example\"\u003e\u003c/a\u003e\n## 8. Example\nExample depicts the thought process on implementing a new unsupported device.\nSee example [here.](EXAMPLE.md)\n\n\u003ca name=\"vrni-api-ref\"\u003e\u003c/a\u003e\n## 9. Uploading Output Zipfile\nThis SDK also includes API integration with vRNI. The script `network_insight_sdk_generic_datasources/common/vrni_uani_ops.py` can be run using the following parameters:\n\n```\nexport PYTHONPATH=$PYTHONPATH:./network-insight-sdk-python/swagger_client-py2.7.egg\npython ./network-insight-sdk-python/examples/add_generic_switch_router.py --platform_ip my-platform-hostname \\\n --proxy_ip my-proxy-hostname --username test@local --password 'test' \\\n --device_ip_or_fqdn device_ip_or_fqdn --zip_file_path path_of_output-sdk-generic-ds.zip\n```\n\n\u003ca name=\"run-and-upload\"\u003e\u003c/a\u003e\n## 10. Running SDK and Uploading\n\nCombining the SDK and the upload script would go as follows:\n\n```\nexport PYTHONPATH=$PYTHONPATH:~/network-insight-sdk-generic-datasources\npython network_insight_sdk_generic_datasources/main.py -d juniper -m srx -s JUNIPER_JUNOS -i abc.srx.lab -u test -p 'test' -o fw1.srx.lab.zip\n\nexport PYTHONPATH=$PYTHONPATH:./network-insight-sdk-python/swagger_client-py2.7.egg\npython ./network-insight-sdk-python/examples/add_generic_switch_router.py --platform_ip my-platform-hostname \\\n --proxy_ip my-proxy-hostname --username test@local --password 'test' \\\n --device_ip_or_fqdn device_ip_or_fqdn --zip_file_path path_of_output-sdk-generic-ds.zip\n```\n\n\nContributing\n============\n\nFeel free to raise issues and send pull requests, we'll be happy to look at them!\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmware%2Fnetwork-insight-sdk-generic-datasources","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmware%2Fnetwork-insight-sdk-generic-datasources","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmware%2Fnetwork-insight-sdk-generic-datasources/lists"}