{"id":24639415,"url":"https://github.com/netboxlabs/diode-sdk-python","last_synced_at":"2026-02-16T19:15:16.254Z","repository":{"id":243923676,"uuid":"806643429","full_name":"netboxlabs/diode-sdk-python","owner":"netboxlabs","description":"Diode SDK Python","archived":false,"fork":false,"pushed_at":"2025-04-17T19:32:44.000Z","size":305,"stargazers_count":21,"open_issues_count":8,"forks_count":2,"subscribers_count":5,"default_branch":"develop","last_synced_at":"2025-04-17T23:46:42.861Z","etag":null,"topics":["netbox","network-automation","sdk","sdk-python"],"latest_commit_sha":null,"homepage":"https://netboxlabs.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/netboxlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-27T15:39:59.000Z","updated_at":"2025-04-12T00:33:32.000Z","dependencies_parsed_at":"2024-08-29T18:16:47.214Z","dependency_job_id":"49bbf5ca-3383-4cf9-b8e8-f9a5b0a1628c","html_url":"https://github.com/netboxlabs/diode-sdk-python","commit_stats":null,"previous_names":["netboxlabs/diode-sdk-python"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netboxlabs%2Fdiode-sdk-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netboxlabs%2Fdiode-sdk-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netboxlabs%2Fdiode-sdk-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netboxlabs%2Fdiode-sdk-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/netboxlabs","download_url":"https://codeload.github.com/netboxlabs/diode-sdk-python/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253181922,"owners_count":21867091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["netbox","network-automation","sdk","sdk-python"],"created_at":"2025-01-25T11:11:29.628Z","updated_at":"2026-02-16T19:15:16.240Z","avatar_url":"https://github.com/netboxlabs.png","language":"Python","readme":"# Diode SDK Python\n\nDiode SDK Python is a Python library for interacting with the Diode ingestion service utilizing gRPC.\n\nDiode is a new [NetBox](https://netboxlabs.com/oss/netbox/) ingestion service that greatly simplifies and enhances the\nprocess to add and update network data\nin NetBox, ensuring your network source of truth is always accurate and can be trusted to power your network automation\npipelines. \n\nMore information about Diode can be found\nat [https://netboxlabs.com/blog/introducing-diode-streamlining-data-ingestion-in-netbox/](https://netboxlabs.com/blog/introducing-diode-streamlining-data-ingestion-in-netbox/).\n\n## Prerequisites\n- Python 3.10 or later installed\n\n## Installation\n\n```bash\npip install netboxlabs-diode-sdk\n```\n\n## Usage\n\n### Environment variables\n\n* `DIODE_SDK_LOG_LEVEL` - Log level for the SDK (default: `INFO`)\n* `DIODE_SENTRY_DSN` - Optional Sentry DSN for error reporting\n* `DIODE_CLIENT_ID` - Client ID for OAuth2 authentication\n* `DIODE_CLIENT_SECRET` - Client Secret for OAuth2 authentication\n* `DIODE_CERT_FILE` - Path to custom certificate file for TLS connections\n* `DIODE_SKIP_TLS_VERIFY` - Skip TLS verification (default: `false`)\n* `DIODE_DRY_RUN_OUTPUT_DIR` - Directory where `DiodeDryRunClient` will write JSON files\n\n### Example\n\n* `target` should be the address of the Diode service.\n  * Insecure connections: `grpc://localhost:8080/diode` or `http://localhost:8080/diode`\n  * Secure connections: `grpcs://example.com` or `https://example.com`\n\n```python\nfrom netboxlabs.diode.sdk import DiodeClient\nfrom netboxlabs.diode.sdk.ingester import (\n    Device,\n    Entity,\n)\n\n\ndef main():\n    with DiodeClient(\n        target=\"grpc://localhost:8080/diode\",\n        app_name=\"my-test-app\",\n        app_version=\"0.0.1\",\n    ) as client:\n        entities = []\n\n        \"\"\"\n        Ingest device with device type, platform, manufacturer, site, role, and tags.\n        \"\"\"\n\n        device = Device(\n            name=\"Device A\",\n            device_type=\"Device Type A\",\n            platform=\"Platform A\",\n            manufacturer=\"Manufacturer A\",\n            site=\"Site ABC\",\n            role=\"Role ABC\",\n            serial=\"123456\",\n            asset_tag=\"123456\",\n            status=\"active\",\n            tags=[\"tag 1\", \"tag 2\"],\n        )\n\n        entities.append(Entity(device=device))\n\n        response = client.ingest(entities=entities)\n        if response.errors:\n            print(f\"Errors: {response.errors}\")\n\n\nif __name__ == \"__main__\":\n    main()\n\n```\n\n### Using Metadata\n\nEntities support attaching custom metadata as key-value pairs. Metadata can be used to store additional context, tracking information, or custom attributes that don't fit into the standard NetBox fields.\n\n```python\nfrom netboxlabs.diode.sdk import DiodeClient, Entity\nfrom netboxlabs.diode.sdk.ingester import Device, Site, IPAddress\n\nwith DiodeClient(\n    target=\"grpc://localhost:8080/diode\",\n    app_name=\"my-app\",\n    app_version=\"1.0.0\",\n) as client:\n    # Create a device with metadata\n    # Note: Both the device and its nested site can have its own metadata\n    device = Device(\n        name=\"Device A\",\n        device_type=\"Device Type A\",\n        site=Site(\n            name=\"Site ABC\",\n            metadata={\n                \"site_region\": \"us-west\",\n                \"site_cost_center\": \"CC-001\",\n            },\n        ),\n        role=\"Role ABC\",\n        metadata={\n            \"source\": \"network_discovery\",\n            \"discovered_at\": \"2024-01-15T10:30:00Z\",\n            \"import_batch\": \"batch-123\",\n            \"priority\": 1,\n            \"verified\": True,\n        },\n    )\n\n    # Create an IP address with metadata\n    ip_address = IPAddress(\n        address=\"192.168.1.10/24\",\n        status=\"active\",\n        metadata={\n            \"last_scan\": \"2024-01-15T12:00:00Z\",\n            \"scan_id\": \"scan-456\",\n            \"response_time\": 23.5,\n            \"reachable\": True,\n            \"owner_team\": \"network-ops\",\n        },\n    )\n\n    # Create a site with metadata\n    site = Site(\n        name=\"Data Center 1\",\n        status=\"active\",\n        metadata={\n            \"region\": \"us-west\",\n            \"cost_center\": \"CC-001\",\n            \"capacity\": 500,\n            \"is_primary\": True,\n            \"contact_email\": \"dc1-ops@example.com\",\n        },\n    )\n\n    entities = [Entity(device=device), Entity(ip_address=ip_address), Entity(site=site)]\n    response = client.ingest(entities=entities)\n    if response.errors:\n        print(f\"Errors: {response.errors}\")\n```\n\n#### Adding request-level metadata\n\nIn addition to entity-level metadata, you can attach metadata to the entire ingestion request using the `metadata` keyword argument. This is useful for tracking information about the ingestion batch itself, such as the data source, batch ID, or processing context.\n\n```python\nfrom netboxlabs.diode.sdk import DiodeClient, Entity\nfrom netboxlabs.diode.sdk.ingester import Device, Site\n\nwith DiodeClient(\n    target=\"grpc://localhost:8080/diode\",\n    app_name=\"my-app\",\n    app_version=\"1.0.0\",\n) as client:\n    # Create device A\n    device_a = Device(\n        name=\"Device A\",\n        site=Site(name=\"Site ABC\"),\n    )\n\n    # Create device B\n    device_b = Device(\n        name=\"Device B\",\n        site=Site(name=\"Site XYZ\"),\n    )\n\n    entities = [Entity(device=device_a), Entity(device=device_b)]\n\n    # Add request-level metadata to track the ingestion batch\n    response = client.ingest(\n        entities=entities,\n        metadata={\n            \"batch_id\": \"import-2024-01-15\",\n            \"source_system\": \"network_scanner\",\n            \"import_type\": \"automated\",\n            \"record_count\": len(entities),\n            \"validated\": True,\n        },\n    )\n    if response.errors:\n        print(f\"Errors: {response.errors}\")\n```\n\nRequest-level metadata is included in the `IngestRequest` and can be useful for:\n- Tracking data sources and ingestion pipelines\n- Correlating entities within a batch\n- Debugging and auditing data imports\n- Adding contextual information for downstream processing\n\n### TLS verification and certificates\n\nTLS verification is controlled by the target URL scheme:\n- **Secure schemes** (`grpcs://`, `https://`): TLS verification enabled\n- **Insecure schemes** (`grpc://`, `http://`): TLS verification disabled\n\n```python\n# TLS verification enabled (uses system certificates)\nclient = DiodeClient(target=\"grpcs://example.com\", ...)\n\n# TLS verification disabled\nclient = DiodeClient(target=\"grpc://example.com\", ...)\n```\n\n### Proxy support\n\nThe SDK automatically detects and uses HTTP/HTTPS proxies configured via standard environment variables:\n\n```bash\n# For insecure connections\nexport HTTP_PROXY=http://proxy.example.com:8080\n\n# For secure connections\nexport HTTPS_PROXY=http://proxy.example.com:8080\n# Falls back to HTTP_PROXY if HTTPS_PROXY is not set\n\n# Bypass proxy for specific hosts\nexport NO_PROXY=localhost,127.0.0.1,.example.com\n```\n\n**Important notes for proxy usage:**\n\n1. **Proxy with SKIP_TLS_VERIFY**: When using HTTP(S) proxies, the SDK **always uses secure channels** because proxies require TLS for the CONNECT tunnel. Setting `DIODE_SKIP_TLS_VERIFY=true` with a proxy will log a warning and use a secure channel anyway.\n\n2. **MITM proxies (like mitmproxy)**: To use an intercepting proxy, you must provide the proxy's CA certificate:\n   ```bash\n   export HTTPS_PROXY=http://127.0.0.1:8080\n   export DIODE_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem\n   ```\n\n3. **Non-intercepting proxies**: Regular forwarding proxies work without additional configuration if the target server has a valid certificate trusted by system CAs.\n\nExample with proxy:\n```python\nimport os\n\n# Configure proxy\nos.environ[\"HTTPS_PROXY\"] = \"http://proxy.example.com:8080\"\n\nclient = DiodeClient(\n    target=\"grpcs://diode.example.com:443\",\n    app_name=\"my-app\",\n    app_version=\"1.0.0\",\n)\n```\n\n#### Using custom certificates\n\n```python\n# Via constructor parameter\nclient = DiodeClient(target=\"grpcs://example.com\", cert_file=\"/path/to/cert.pem\", ...)\n\n# Or via environment variable\nexport DIODE_CERT_FILE=/path/to/cert.pem\n```\n\n#### Disabling TLS verification\n\n```bash\nexport DIODE_SKIP_TLS_VERIFY=true\n```\n\n#### For legacy certificates (CN-only, no SANs)\n\n```python\nclient = DiodeClient(\n    target=\"grpcs://example.com\",\n    app_name=\"my-app\",\n    app_version=\"1.0.0\",\n    cert_file=\"/path/to/cert.pem\",\n    skip_tls_verify=True,\n)\n```\n\n### Message chunking\n\nWhen ingesting large numbers of entities, you may need to split them into smaller chunks to avoid exceeding the gRPC message size limit for a single `ingest()` call. The SDK provides chunking utilities that automatically split entity lists into appropriately sized chunks.\n\n#### How it works\n\nThe SDK uses a **greedy bin-packing algorithm** that:\n1. Accumulates entities until adding the next entity would exceed the size limit\n2. Starts a new chunk when the limit would be exceeded\n3. Ensures each chunk stays safely under the configured limit (default: 3 MB)\n\n#### Basic usage\n\n```python\nfrom netboxlabs.diode.sdk import DiodeClient, create_message_chunks\nfrom netboxlabs.diode.sdk.ingester import Device, Entity\n\nwith DiodeClient(\n    target=\"grpc://localhost:8080/diode\",\n    app_name=\"my-app\",\n    app_version=\"1.0.0\",\n) as client:\n    # Create a large list of entities\n    entities = []\n    for i in range(10000):\n        device = Device(\n            name=f\"Device {i}\",\n            device_type=\"Device Type A\",\n            site=\"Site ABC\",\n            role=\"Role ABC\",\n        )\n        entities.append(Entity(device=device))\n\n    # Split into chunks (default 3 MB per chunk), then ingest each chunk separately.\n    for chunk in create_message_chunks(entities):\n        client.ingest(entities=chunk)\n```\n\n#### Custom chunk size\n\nYou can customize the chunk size if needed:\n\n```python\nfrom netboxlabs.diode.sdk import create_message_chunks\n\n# Use a larger chunk size (3.5 MB)\nchunks = create_message_chunks(entities, max_chunk_size_mb=3.5)\n\n# Use a smaller chunk size for conservative chunking (2 MB)\nchunks = create_message_chunks(entities, max_chunk_size_mb=2.0)\n```\n\n#### Estimating message size\n\nYou can estimate the serialized size of entities before chunking:\n\n```python\nfrom netboxlabs.diode.sdk import estimate_message_size\n\nsize_bytes = estimate_message_size(entities)\nsize_mb = size_bytes / (1024 * 1024)\nprint(f\"Total size: {size_mb:.2f} MB\")\n\n# Decide whether chunking is needed\nif size_mb \u003e 3.0:\n    for chunk in create_message_chunks(entities):\n        client.ingest(entities=chunk)\nelse:\n    # Small enough to send in one request\n    client.ingest(entities=entities)\n```\n\n\n### Dry run mode\n\n`DiodeDryRunClient` generates ingestion requests without contacting a Diode server. Requests are printed to stdout by default, or written to JSON files when `output_dir` (or the `DIODE_DRY_RUN_OUTPUT_DIR` environment variable) is specified. The `app_name` parameter serves as the filename prefix; if not provided, `dryrun` is used as the default prefix. The file name is suffixed with a nanosecond-precision timestamp, resulting in the format `\u003capp_name\u003e_\u003ctimestamp_ns\u003e.json`.\n\n```python\nfrom netboxlabs.diode.sdk import DiodeDryRunClient\n\nwith DiodeDryRunClient(app_name=\"my_app\", output_dir=\"/tmp\") as client:\n    client.ingest([\n        Entity(device=\"Device A\"),\n    ])\n```\n\nThe produced file can later be ingested by a real Diode instance using\n`load_dryrun_entities` with a standard `DiodeClient` or via the bundled\n`diode-replay-dryrun` helper:\n\n```python\nfrom netboxlabs.diode.sdk import DiodeClient, load_dryrun_entities\n\nwith DiodeClient(\n    target=\"grpc://localhost:8080/diode\",\n    app_name=\"my-test-app\",\n    app_version=\"0.0.1\",\n) as client:\n    entities = list(load_dryrun_entities(\"my_app_92722156890707.json\"))\n    client.ingest(entities=entities)\n```\n\nAlternatively, the same file can be ingested using the `diode-replay-dryrun`\ncommand shipped with the SDK:\n\n```bash\ndiode-replay-dryrun \\\n  --target grpc://localhost:8080/diode \\\n  --app-name my-test-app \\\n  --app-version 0.0.1 \\\n  my_app_92722156890707.json\n```\n\n#### Adding request-level metadata to dry run output\n\nYou can include request-level metadata in the dry run output using the `metadata` keyword argument. This metadata will be included in the JSON output file as part of the `IngestRequest`:\n\n```python\nfrom netboxlabs.diode.sdk import DiodeDryRunClient, Entity\nfrom netboxlabs.diode.sdk.ingester import Device\n\nwith DiodeDryRunClient(app_name=\"my_app\", output_dir=\"/tmp\") as client:\n    # Add request-level metadata\n    client.ingest(\n        [Entity(device=Device(name=\"Device A\"))],\n        metadata={\n            \"batch_id\": \"import-2024-01\",\n            \"source\": \"csv_import\",\n            \"validated\": True,\n            \"record_count\": 150,\n        }\n    )\n```\n\nThe resulting JSON file will include the metadata in the `IngestRequest`, making it visible when reviewing the dry run output.\n\n### CLI to replay dry-run files\n\nA small helper command is included to ingest JSON files created by the\n`DiodeDryRunClient` and send them to a running Diode service.\n\nInstall the helper using `pip`:\n\n```bash\npip install netboxlabs-diode-sdk\n```\n\nRun it by providing one or more JSON files and connection details. The command supports replaying multiple dry-run files in a single request:\n\n```bash\ndiode-replay-dryrun \\\n  --file /tmp/my_app_92722156890707.json \\\n  --file /tmp/other.json \\\n  --target grpc://localhost:8080/diode \\\n  --app-name my-test-app \\\n  --app-version 0.0.1 \\\n  --client-id YOUR_CLIENT_ID \\\n  --client-secret YOUR_CLIENT_SECRET\n```\n\nThe `--file`, `--target`, `--app-name`, and `--app-version` arguments are required. You may\nrepeat `--file` to specify multiple files. OAuth2\ncredentials can be supplied using `--client-id` and `--client-secret` or the\n`DIODE_CLIENT_ID` and `DIODE_CLIENT_SECRET` environment variables.\n\n### OTLP client\n\n`DiodeOTLPClient` converts ingestion entities into OpenTelemetry log records and exports them to an OTLP endpoint (gRPC). This is useful when a collector ingests log data and forwards it to Diode.\n\n```python\nfrom netboxlabs.diode.sdk import Entity, DiodeOTLPClient\n\nwith DiodeOTLPClient(\n    target=\"grpc://localhost:4317\",\n    app_name=\"my-producer\",\n    app_version=\"0.0.1\",\n) as client:\n    client.ingest([Entity(site=\"Site1\")])\n```\n\nEach entity is serialised to JSON and sent as a log record with producer metadata so downstream collectors can enrich and forward the payload. The client raises `OTLPClientError` when the export fails. TLS behaviour honours the existing `DIODE_SKIP_TLS_VERIFY` and `DIODE_CERT_FILE` environment variables.\n\n#### Adding request-level metadata as OTLP resource attributes\n\nYou can add request-level metadata to OTLP exports using the `metadata` keyword argument. This metadata is automatically mapped to OTLP resource attributes with a `diode.metadata.` prefix:\n\n```python\nfrom netboxlabs.diode.sdk import DiodeOTLPClient, Entity\nfrom netboxlabs.diode.sdk.ingester import Site\n\nwith DiodeOTLPClient(\n    target=\"grpc://localhost:4317\",\n    app_name=\"otlp-producer\",\n    app_version=\"1.0.0\",\n) as client:\n    # Add request-level metadata\n    client.ingest(\n        [Entity(site=Site(name=\"Site 1\"))],\n        metadata={\n            \"environment\": \"production\",\n            \"deployment\": \"us-west-2\",\n            \"version\": \"1.2.3\",\n            \"priority\": 5,\n        },\n    )\n```\n\nThe resulting OTLP log records will include resource attributes like:\n- `diode.metadata.environment=\"production\"`\n- `diode.metadata.deployment=\"us-west-2\"`\n- `diode.metadata.version=\"1.2.3\"`\n- `diode.metadata.priority=5` (as integer)\n\nThese attributes are added alongside standard OTLP resource attributes (`service.name`, `service.version`, `diode.stream`, etc.), allowing downstream collectors and observability platforms to filter, route, and enrich the data based on this metadata.\n\n## Supported entities (object types)\n\n* ASN\n* ASN Range\n* Aggregate\n* Circuit\n* Circuit Group\n* Circuit Group Assignment\n* Circuit Termination\n* Circuit Type\n* Cluster\n* Cluster Group\n* Cluster Type\n* Console Port\n* Console Server Port\n* Contact\n* Contact Assignment\n* Contact Group\n* Contact Role\n* Device\n* Device Bay\n* Device Role\n* Device Type\n* FHRP Group\n* FHRP Group Assignment\n* Front Port\n* IKE Policy\n* IKE Proposal\n* IP Address\n* IP Range\n* IP Sec Policy\n* IP Sec Profile\n* IP Sec Proposal\n* Interface\n* Inventory Item\n* Inventory Item Role\n* L2VPN\n* L2VPN Termination\n* Location\n* MAC Address\n* Manufacturer\n* Module\n* Module Bay\n* Module Type\n* Platform\n* Power Feed\n* Power Outlet\n* Power Panel\n* Power Port\n* Prefix\n* Provider\n* Provider Account\n* Provider Network\n* RIR\n* Rack\n* Rack Role\n* Rack Type\n* Rear Port\n* Region\n* Role\n* Route Target\n* Service\n* Site\n* Site Group\n* Tag\n* Tenant\n* Tenant Group\n* Tunnel\n* Tunnel Group\n* Tunnel Termination\n* VLAN\n* VLAN Group\n* VLAN Translation Policy\n* VLAN Translation Rule\n* VM Interface\n* VRF\n* Virtual Chassis\n* Virtual Circuit\n* Virtual Circuit Termination\n* Virtual Circuit Type\n* Virtual Device Context\n* Virtual Disk\n* Virtual Machine\n* Wireless Lan\n* Wireless Lan Group\n* Wireless Link\n\n## Development notes\n\nCode in `netboxlabs/diode/sdk/diode/*` is generated from Protocol Buffers definitions (will be published and referenced here soon).\n\n#### Linting\n\n```shell\nruff netboxlabs/\nblack netboxlabs/\n```\n\n#### Testing\n\n```shell\nPYTHONPATH=$(pwd) pytest\n```\n\n## License\n\nDistributed under the Apache 2.0 License. See [LICENSE.txt](./LICENSE.txt) for more information.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetboxlabs%2Fdiode-sdk-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetboxlabs%2Fdiode-sdk-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetboxlabs%2Fdiode-sdk-python/lists"}