{"id":13710762,"url":"https://github.com/bb-Ricardo/netbox-sync","last_synced_at":"2025-05-06T19:32:29.902Z","repository":{"id":37691803,"uuid":"307420604","full_name":"bb-Ricardo/netbox-sync","owner":"bb-Ricardo","description":"Sync objects from VMware or redfish sources to NetBox","archived":false,"fork":false,"pushed_at":"2024-05-14T23:22:16.000Z","size":596,"stargazers_count":264,"open_issues_count":24,"forks_count":61,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-05-15T18:25:15.745Z","etag":null,"topics":["netbox","netbox-automation","python","redfish","sync","vcenter","vmware"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bb-Ricardo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-26T15:37:31.000Z","updated_at":"2024-05-29T21:55:04.138Z","dependencies_parsed_at":"2023-02-19T09:46:05.054Z","dependency_job_id":"6f66a40e-2d32-4f12-b244-4e964573bfd8","html_url":"https://github.com/bb-Ricardo/netbox-sync","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bb-Ricardo%2Fnetbox-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bb-Ricardo%2Fnetbox-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bb-Ricardo%2Fnetbox-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bb-Ricardo%2Fnetbox-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bb-Ricardo","download_url":"https://codeload.github.com/bb-Ricardo/netbox-sync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252753720,"owners_count":21798999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["netbox","netbox-automation","python","redfish","sync","vcenter","vmware"],"created_at":"2024-08-02T23:01:00.648Z","updated_at":"2025-05-06T19:32:24.874Z","avatar_url":"https://github.com/bb-Ricardo.png","language":"Python","funding_links":[],"categories":["Synchronization"],"sub_categories":[],"readme":"\n# NetBox-Sync\n\nThis is a tool to sync data from different sources to a NetBox instance.\n\nAvailable source types:\n* VMware vCenter Server\n* [bb-ricardo/check_redfish](https://github.com/bb-Ricardo/check_redfish) inventory files\n\n**IMPORTANT: READ INSTRUCTIONS CAREFULLY BEFORE RUNNING THIS PROGRAM**\n\n## Thanks\nA BIG thank-you goes out to [Raymond Beaudoin](https://github.com/synackray) for creating\n[vcenter-netbox-sync](https://github.com/synackray/vcenter-netbox-sync) which served as source of a lot\nof ideas for this project.\n\n## Principles\n\n\u003e copied from [Raymond Beaudoin](https://github.com/synackray)\n\nThe [NetBox documentation](https://netbox.readthedocs.io/en/stable/#serve-as-a-source-of-truth) makes it clear\nthe tool is intended to act as a \"Source of Truth\". The automated import of live network state is\nstrongly discouraged. While this is sound logic we've aimed to provide a middle-ground\nsolution for those who desire the functionality.\n\nAll objects collected from vCenter have a \"lifecycle\". Upon import, for supported object types,\nthey are tagged `NetBox-synced` to note their origin and distinguish them from other objects.\nUsing this tagging system also allows for the orphaning of objects which are no longer detected in vCenter.\nThis ensures stale objects are removed from NetBox keeping an accurate current state.\n\n## Requirements\n### Software\n* python \u003e= 3.6\n* packaging\n* urllib3==2.2.1\n* wheel\n* requests==2.31.0\n* pyvmomi==8.0.2.0.1\n* aiodns==3.0.0\n* pyyaml==6.0.1\n\n### Environment\n* NetBox \u003e= 2.9\n#### Source: VMWare (if used)\n* VMWare vCenter \u003e= 6.0\n#### Source: check_redfish (if used)\n* check_redfish \u003e= 1.2.0\n\n# Installing\n* here we assume we install in ```/opt```\n\n## RedHat based OS\n* on RedHat/CentOS 7 you need to install python3.6 and pip from EPEL first\n* on RedHat/CentOS 8 systems the package name changed to `python3-pip`\n```shell\nyum install python36-pip\n```\n\n## Ubuntu 18.04 \u0026 20.04 \u0026\u0026 22.04\n```shell\napt-get update \u0026\u0026 apt-get install python3-venv\n```\n\n## Clone repo and install dependencies\n* If you need to use python 3.6 then you would need `requirements_3.6.txt` to install requirements\n* download and setup of virtual environment\n```shell\ncd /opt\ngit clone https://github.com/bb-Ricardo/netbox-sync.git\ncd netbox-sync\npython3 -m venv .venv\n. .venv/bin/activate\npip3 install --upgrade pip || pip install --upgrade pip\npip3 install wheel || pip install wheel\npip3 install -r requirements.txt || pip install -r requirements.txt\n```\n\n### VMware tag sync (if necessary)\nThe `vsphere-automation-sdk` must be installed if tags should be synced from vCenter to NetBox\n* assuming we are still in an activated virtual env\n```shell\npip install --upgrade git+https://github.com/vmware/vsphere-automation-sdk-python.git\n```\n\n## NetBox API token\nIn order to updated data in NetBox you need a NetBox API token.\n* API token with all permissions (read, write) except:\n  * auth\n  * secrets\n  * users\n\nA short description can be found [here](https://docs.netbox.dev/en/stable/integrations/rest-api/#authentication)\n\n# Running the script\n\n```\nusage: netbox-sync.py [-h] [-c settings.ini [settings.ini ...]] [-g]\n                      [-l {DEBUG3,DEBUG2,DEBUG,INFO,WARNING,ERROR}] [-n] [-p]\n\nSync objects from various sources to NetBox\n\nVersion: 1.7.0 (2024-10-21)\nProject URL: https://github.com/bb-ricardo/netbox-sync\n\noptions:\n  -h, --help            show this help message and exit\n  -c settings.ini [settings.ini ...], --config settings.ini [settings.ini ...]\n                        points to the config file to read config data from\n                        which is not installed under the default path\n                        './settings.ini'\n  -g, --generate_config\n                        generates default config file.\n  -l {DEBUG3,DEBUG2,DEBUG,INFO,WARNING,ERROR}, --log_level {DEBUG3,DEBUG2,DEBUG,INFO,WARNING,ERROR}\n                        set log level (overrides config)\n  -n, --dry_run         Operate as usual but don't change anything in NetBox.\n                        Great if you want to test and see what would be\n                        changed.\n  -p, --purge           Remove (almost) all synced objects which were create\n                        by this script. This is helpful if you want to start\n                        fresh or stop using this script.\n```\n\n## TESTING\nIt is recommended to set log level to `DEBUG2` this way the program should tell you what is happening and why.\nAlso use the dry run option `-n` at the beginning to avoid changes directly in NetBox.\n\n## Configuration\nThere are two ways to define configuration. Any combination of config file(s) and environment variables is possible.\n* config files (the [default config](https://github.com/bb-Ricardo/netbox-sync/blob/main/settings-example.ini) file name is set to `./settings.ini`.)\n* environment variables\n\nThe config from the environment variables will have precedence over the config file definitions.\n\n### Config files\nFollowing config file types are supported:\n* ini\n* yaml\n\nThere is also more than one config file permitted. Example (config file names are also just examples):\n```bash\n/opt/netbox-sync/netbox-sync.py -c common.ini all-sources.yaml additional-config.yaml\n```\n\nAll files are parsed in order of the definition and options will overwrite the same options if defined in a\nprevious config file.\n\nTo get config file examples which include descriptions and all default values, the `-g` can be used:\n```bash\n# this will create an ini example\n/opt/netbox-sync/netbox-sync.py -g -c settings-example.ini\n\n# and this will create an example config file in yaml format\n/opt/netbox-sync/netbox-sync.py -g -c settings-example.yaml \n```\n\n### Environment variables\nEach setting which can be defined in a config file can also be defined using an environment variable.\n\nThe prefix for all environment variables to be used in netbox-sync is: `NBS`\n\nFor configuration in the `common` and `netbox` section a variable is defined like this\n```\n\u003cPREFIX\u003e_\u003cSECTION_NAME\u003e_\u003cCONFIG_OPTION_KEY\u003e=value\n```\n\nFollowing example represents the same configuration:\n```yaml\n# yaml config example\ncommon:\n  log_level: DEBUG2\nnetbox:\n  host_fqdn: netbox-host.example.com\n  prune_enabled: true\n```\n```bash\n# this variable definition is equal to the yaml config sample above\nNBS_COMMON_LOG_LEVEL=\"DEBUG2\"\nNBS_netbox_host_fqdn=\"netbox-host.example.com\"\nNBS_NETBOX_PRUNE_ENABLED=\"true\"\n```\n\nThis way it is possible to expose for example the `NBS_NETBOX_API_KEY` only via an env variable.\n\nThe config definitions for `sources` need to be defined using an index. Following conditions apply:\n* a single source needs to use the same index\n* the index can be number or a name (but contain any special characters to support env var parsing)\n* the source needs to be named with `_NAME` variable\n\nExample of defining a source with config and environment variables.\n```ini\n; example for a source\n[source/example-vcenter]\nenabled = True\ntype = vmware\nhost_fqdn = vcenter.example.com\nusername = vcenter-readonly\n```\n```bash\n# define the password on command line\n# here we use '1' as index\nNBS_SOURCE_1_NAME=\"example-vcenter\"\nNBS_SOURCE_1_PASSWORD=\"super-secret-and-not-saved-to-the-config-file\"\nNBS_SOURCE_1_custom_dns_servers=\"10.0.23.23, 10.0.42.42\"\n```\n\nEven to just define one source variable like `NBS_SOURCE_1_PASSWORD` the `NBS_SOURCE_1_NAME` needs to be defined as\nto associate to the according source definition.\n\n## Cron job\nIn Order to sync all items regularly you can add a cron job like this one\n```\n # NetBox Sync\n 23 */2 * * *  /opt/netbox-sync/.venv/bin/python3 /opt/netbox-sync/netbox-sync.py \u003e/dev/null 2\u003e\u00261\n```\n\n## Docker\n\nRun the application in a docker container. You can build it yourself or use the ones from docker hub.\n\nAvailable here: [bbricardo/netbox-sync](https://hub.docker.com/r/bbricardo/netbox-sync)\n\n* The application working directory is ```/app```\n* Required to mount your ```settings.ini```\n\nTo build it by yourself just run:\n```shell\ndocker build -t bbricardo/netbox-sync:latest .\n```\n\nTo start the container just use:\n```shell\ndocker run --rm -it -v $(pwd)/settings.ini:/app/settings.ini bbricardo/netbox-sync:latest\n```\n\n## Kubernetes\n\nRun the containerized application in a kubernetes cluster\n\n* Create a config map with the default settings\n* Create a secret witch only contains the credentials needed\n* Adjust the provided [cronjob resource](https://github.com/bb-Ricardo/netbox-sync/blob/main/k8s-netbox-sync-cronjob.yaml) to your needs\n* Deploy the manifest to your k8s cluster and check the job is running\n\nconfig example saved as `settings.yaml`\n```yaml\nnetbox:\n  host_fqdn: netbox.example.com\n\nsource:\n  my-vcenter-example:\n    type: vmware\n    host_fqdn: vcenter.example.com\n    permitted_subnets: 172.16.0.0/12, 10.0.0.0/8, 192.168.0.0/16, fd00::/8\n    cluster_site_relation: Cluster_NYC = New York, Cluster_FFM.* = Frankfurt, Datacenter_TOKIO/.* = Tokio\n```\n\nsecrets example saved as `secrets.yaml`\n```yaml\nnetbox:\n  api_token: XYZXYZXYZXYZXYZXYZXYZXYZ\nsource:\n  my-vcenter-example:\n    username: vcenter-readonly\n    password: super-secret\n```\n\nCreate resource in your k8s cluster\n ```shell\nkubectl create configmap netbox-sync-config --from-file=settings.yaml\nkubectl create secret generic netbox-sync-secrets --from-file=secrets.yaml\nkubectl apply -f k8s-netbox-sync-cronjob.yaml\n ```\n\n# How it works\n**READ CAREFULLY**\n\n## Basic structure\nThe program operates mainly like this\n1. parsing and validating config\n2. instantiating all sources and setting up connection to NetBox\n3. read current data from NetBox\n4. read data from all sources and add/update objects in memory\n5. Update data in NetBox based on data from sources\n6. Prune old objects\n\n## NetBox connection\nRequest all current NetBox objects. Use caching whenever possible.\nObjects must provide \"last_updated\" attribute to support caching for this object type.\n\nActually perform the request and retry x times if request times out.\nProgram will exit if all retries failed!\n\n## Supported sources\nCheck out the documentations for the different sources\n* [vmware](https://github.com/bb-Ricardo/netbox-sync/blob/main/docs/source_vmware.md)\n* [check_redfish](https://github.com/bb-Ricardo/netbox-sync/blob/main/docs/source_check_redfish.md)\n\nIf you have multiple vCenter instances or check_redfish folders just add another source with the same type\nin the **same** file.\n\nExample:\n```ini\n[source/vcenter-BLN]\n\nenabled = True\nhost_fqdn = vcenter1.berlin.example.com\n\n[source/vcenter-NYC]\n\nenabled = True\nhost_fqdn = vcenter2.new-york.example.com\n\n[source/redfish-hardware]\n\ntype = check_redfish\ninventory_file_path = /opt/redfish_inventory\n```\n\nIf different sources overwrite the same attribute for ex. a host then the order of the sources should be considered.\nThe last source in order from top to bottom will prevail.\n\n## Pruning\nPrune objects in NetBox if they are no longer present in any source.\nFirst they will be marked as Orphaned and after X (config option) days they will be\ndeleted from NetBox.\n\nObjects subjected to pruning:\n* devices\n* VMs\n* device interfaces\n* VM interfaces\n* IP addresses\n\nAll other objects created (i.e.: VLANs, cluster, manufacturers) will keep the\nsource tag but will not be deleted. Theses are \"shared\" objects might be used\nby different NetBox objects\n\n# License\n\u003eYou can check out the full license [here](https://github.com/bb-Ricardo/netbox-sync/blob/main/LICENSE.txt)\n\nThis project is licensed under the terms of the **MIT** license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbb-Ricardo%2Fnetbox-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbb-Ricardo%2Fnetbox-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbb-Ricardo%2Fnetbox-sync/lists"}