{"id":21030570,"url":"https://github.com/workfloworchestrator/example-orchestrator","last_synced_at":"2025-07-03T15:04:59.586Z","repository":{"id":211255905,"uuid":"711220308","full_name":"workfloworchestrator/example-orchestrator","owner":"workfloworchestrator","description":"Example Workflow Orchestrator implementation","archived":false,"fork":false,"pushed_at":"2025-04-03T15:07:15.000Z","size":1040,"stargazers_count":16,"open_issues_count":5,"forks_count":5,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-05-15T11:36:25.458Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/workfloworchestrator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-28T15:15:51.000Z","updated_at":"2025-04-09T10:27:23.000Z","dependencies_parsed_at":"2023-12-07T12:30:02.671Z","dependency_job_id":"b7056243-a90d-4c22-bae3-dd3b565fe032","html_url":"https://github.com/workfloworchestrator/example-orchestrator","commit_stats":null,"previous_names":["workfloworchestrator/example-orchestrator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/workfloworchestrator/example-orchestrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/workfloworchestrator%2Fexample-orchestrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/workfloworchestrator%2Fexample-orchestrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/workfloworchestrator%2Fexample-orchestrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/workfloworchestrator%2Fexample-orchestrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/workfloworchestrator","download_url":"https://codeload.github.com/workfloworchestrator/example-orchestrator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/workfloworchestrator%2Fexample-orchestrator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263347966,"owners_count":23452866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T12:19:16.123Z","updated_at":"2025-07-03T15:04:59.567Z","avatar_url":"https://github.com/workfloworchestrator.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Example Workflow Orchestrator\n\nExample workflow orchestrator implementation based on the\n[orchestrator-core](https://workfloworchestrator.org/orchestrator-core/) framework.\n\n- [Example Workflow Orchestrator](#example-workflow-orchestrator)\n  - [Quickstart](#quickstart)\n    - [Start application](#start-application)\n    - [Using the example orchestrator](#using-the-example-orchestrator)\n  - [Summary](#summary)\n  - [Introduction](#introduction)\n  - [Example orchestrator](#example-orchestrator)\n    - [Folder layout](#folder-layout)\n      - [migrations/versions/schema](#migrationsversionsschema)\n      - [products/product\\_types](#productsproduct_types)\n      - [products/product\\_blocks](#productsproduct_blocks)\n      - [products/services](#productsservices)\n      - [services](#services)\n      - [templates](#templates)\n      - [translations](#translations)\n      - [utils](#utils)\n      - [workflows](#workflows)\n      - [shared](#shared)\n    - [Main application](#main-application)\n    - [Implemented products](#implemented-products)\n    - [Product Hiearchy Diagram](#product-hiearchy-diagram)\n    - [How to use](#how-to-use)\n      - [Node](#node)\n      - [CoreLink](#corelink)\n      - [Port](#port)\n      - [L2vpn](#l2vpn)\n  - [Products](#products)\n    - [Product description in Python](#product-description-in-python)\n      - [Example of \"Runtime typecasting/safety\"](#example-of-runtime-typecastingsafety)\n      - [Product Structure](#product-structure)\n      - [Terminology](#terminology)\n    - [Product types](#product-types)\n      - [Domain Model a.k.a Product Type Definition](#domain-model-aka-product-type-definition)\n      - [Fixed Input](#fixed-input)\n      - [Wiring it up in the Orchestrator](#wiring-it-up-in-the-orchestrator)\n    - [Product blocks](#product-blocks)\n      - [Resource Type lifecycle. When to use `None`](#resource-type-lifecycle-when-to-use-none)\n      - [Example](#example)\n      - [Product Block customisation](#product-block-customisation)\n  - [Workflows - Basics](#workflows---basics)\n    - [Workflow Architecture - Passing information from one step to the next](#workflow-architecture---passing-information-from-one-step-to-the-next)\n      - [Example](#example-1)\n    - [Forms](#forms)\n      - [Form _Magic_](#form-magic)\n  - [Workflow examples](#workflow-examples)\n    - [Create workflow](#create-workflow)\n      - [Input Form](#input-form)\n      - [Extra Validation between dependant fields](#extra-validation-between-dependant-fields)\n    - [Modify workflow](#modify-workflow)\n    - [Terminate workflow](#terminate-workflow)\n    - [Validate workflows](#validate-workflows)\n  - [Services](#services-1)\n    - [Subscription descriptions](#subscription-descriptions)\n    - [Netbox](#netbox)\n      - [Payload](#payload)\n      - [Create](#create)\n      - [Update](#update)\n      - [Get](#get)\n      - [Delete](#delete)\n      - [Product block to Netbox object mapping](#product-block-to-netbox-object-mapping)\n    - [Federation](#federation)\n      - [Requirements](#requirements)\n      - [Example queries](#example-queries)\n  - [Configuration](#configuration)\n  - [Glossary](#glossary)\n\n## Quickstart\n\n### Start application\n\nMake sure you have docker installed and run:\n\n```\ndocker compose up\n```\n\nThis will start the `orchestrator`, `orchestrator-ui`, `netbox`, `federation`, `postgres` and `redis`.\n\nTo include LSO, run the following command instead:\n\n```\nCOMPOSE_PROFILES=lso docker compose up\n```\n\nThis will build the Docker image for LSO locally, and make the orchestrator use the included Ansible playbooks.\n\nTo access the new v2 `orchestrator-ui`, point your browser to:\n\n```\nhttp://localhost:3000/\n```\n\n\nTo access `netbox` (admin/admin), point your browser to:\n\n```\nhttp://localhost:8000/\n```\n\n\nTo access `federation`, point your browser to:\n\n```\nhttp://localhost:4000\n```\n\n### Using the example orchestrator\n\nUse the following steps to see the example orchestrator in action:\n\n1. bootstrap Netbox\n    1. from the `Tasks` page click `New Task`\n    2. select `Netbox Bootstrap` and click `Start task`\n    3. select `Expand all` on the following page to see the step details\n2. create a network node (need at least two to create a core link)\n    1. in the left-above corner, click on `New subscription`\n    2. select either the `Node Cisco` or `Node Nokia`\n    3. fill in the needed fields, click `Start workflow` and view the summary form\n    4. click `Start workflow` again to start the workflow, or click `Previous` to modify fields\n3. add interfaces to a node (needed by the other products)\n    1. on the `Subscriptions` page, click on the subscription description of the node to show the details\n    2. select `Update node interfaces` from the `Actions` pulldown\n4. create a core link\n    1. in the left-above corner, click on `New subscription`\n    2. select either the `core link 10G` or `core link 100G`\n    3. fill in the forms and finally click on `Start workflow` to start the workflow\n5. create a customer port (need at least two **tagged** ports to create a l2vpn)\n    1. use `New subscription` for either a `port 10G` or a `port 100G`\n    3. fill in the forms and click on `Start workflow` to start the workflow\n6. create a l2vpn\n    1. use `New subscription` for a `l2vpn`, fill in the forms, and `Start workflow`\n\nWhile running the different workflows, have a look at the following\nnetbox pages to see the orchestrator interact with netbox:\n\n- Devices\n    - Devices\n    - Interfaces\n- Connections\n    - Cables\n    - Interface Connections\n- IPAM\n    - IP Addresses\n    - Prefixes\n    - VLANs\n- Overlay\n    - L2VPNs\n    - Terminations\n\n## Summary\n\nMore and more NREN’s start automating and orchestrating their\noperational network procedures and flows of information, making use of\nthe open-source Workflow Orchestrator. When a NREN creates an\norchestrator based on the WFO framework, custom integration code needs\nto be written that is business specific. To accommodate NREN’s to help\neach other while writing this code, and to facilitate collaboration for\nthe further development of the framework and to achieve a set of\nstandardized products and workflows, a set of best common practices\n(BCP) is being set forth in this document.\n\nTo start with, a standard folder layout is described to organize the\ncustom integration code base, this helps in quickly finding similar code\nin different implementations. To help illustrate the BCP, an example\norchestrator has been implemented for a virtual NREN. The defined\nproducts model a network node, a core link between nodes, a customer\nport, and a customer L2VPN service between those ports. For all products\nthe complete set of create, modify, terminate, and validate workflows\nare implemented. Products and product blocks are described in Domain\nModels that are designed to help the developer manage complex data\nmodels and interact with the objects in a user-friendly way. The how and\nwhy of this all is described in great detail.\n\nFinally, the use of services is introduced. A service is collections of\nhelper functions that deliver a service to other parts of the code base.\nThe common programming pattern of function overloading is used for the\nimplementation of a service. A service can be as simple as the\ngeneration of a description based on the product block domain model, or\na complete interface to query, create, update or delete objects in an\nOSS or BSS. In the example orchestrator, a service is implemented to\ninterface with Netbox that is being used as IMS and IPAM. The mapping\nbetween the product blocks and the objects in Netbox are described, and\nthe interface is fully implemented for the supported products.\n\nThe example orchestrator is fully functional and showcases how the WFO\ncan be integrated with Netbox.\n\n## Introduction\n\nTo capture the Best Common Practices for implementing a network\norchestrator using the Workflow Orchestrator (WFO) software framework,\nan example orchestrator is implemented that can be found at\n\u003chttps://github.com/workfloworchestrator/example-orchestrator\u003e. This\ndocument can be seen as a reading guide for the code base that has been\nwritten, and will highlight many of these practices and the reasoning\nbehind them. A basic understanding of the inner workings of the Workflow\nOrchestrator is assumed up to a level as discussed in Milestone M7.3\nCommon NREN Network Service Product Models[^1] and is explained in the\nworkshops that can be found on the Workflow Orchestrator website[^2].\nBasic knowledge on designing and operating computer networks and an\naccompanying product portfolio and procedures is also assumed.\n\nThe products and workflows implemented in the example orchestrator are\nbased on a simple fictional NREN that has the following characteristics:\n\n- The network consists of Provider and Provider Edge network nodes\n- The network nodes are connected to each other through core links\n- On top of this substrate a set of services like Internet Access, L3VPN and L2VPN are offered\n- The Operations Support Systems (OSS) used are:\n    - An IP Administration Management (IPAM) tool\n    - A network Inventory Management System (IMS)\n    - A Network Resource Manager (NRM) to provision the network\n- There is no Business Support System (BSS) yet\n\nThis NREN decided on a phased introduction of automation in their\norganisation, only automating some of the procedures and flows of\ninformation while leaving others unautomated for the moment:\n\n- Automated administration and provisioning of:\n    - Network nodes including loopback IP addresses\n    - Core links in between network nodes including point-to-point IP addresses\n    - Customer ports\n    - Customer L2VPN’s\n- Not automated administration and provisioning of:\n    - Role, make and model of the network nodes\n    - Sites where network nodes are installed\n    - Customer services like Internet Access, L3VPN, …\n    - Internet peering\n\nNetBox[^3] is used as IMS and IPAM, and serves as the source of truth\nfor the complete IP address administration and physical and logical\nnetwork infrastructure. It has a REST based API that makes it easy to\nintegrate with the Workflow Orchestrator.\n\nThe GraphQL APIs of WFO and NetBox support Federation[^8] through the GraphQL framework Strawberry[^9].\n\n## Example orchestrator\n\nTo automate the administration and provisioning of the nodes, core\nlinks, customer ports and L2VPN’s of the virtual NREN, an orchestrator\nis implemented making use of the WFO framework.\n\n### Folder layout\n\nCreating an orchestrator based on the WFO framework needs custom\nintegration code that is business specific. This code can be organised\nlike described below. A standard folder layout does not only make it\neasier to navigate different orchestrator implementations, but also\nhelps with keeping the code organised while the number of products and\nassociated workflows increases. The following layout is recommended and\nis, for example, also being used in the WFO workshops and by many of the\nWFO users.\n\n```\n├── migrations\n│   └── versions\n│   └── schema\n├── products\n│   ├── product_blocks\n│   ├── product_types\n│   └── services\n│   └── \u003cservice\u003e\n├── services\n│ └── \u003cservice\u003e\n├── templates\n├── translations\n├── utils\n└── workflows\n├── \u003cproduct\u003e\n└── tasks\n```\n\n#### migrations/versions/schema\n\nThis is the default location used by Alembic to store migration files.\nAlembic is a lightweight database migration tool that is part of\nSQLAlchemy[^4], and uses multiple HEAD’s to allow both the\norchestrator-core package and the implementation using this package to\nmaintain its own list of migrations. Usually there is at least a\nmigration file for each new product plus associated workflows that is\nadded to the implementation.\n\n#### products/product_types\n\nEach product has its own file, named after the product, that describes\nthe product domain model in all its lifecycle stages. For example, the\nfilename for a L2VPN product would be `l2vpn.py`.\n\n#### products/product_blocks\n\nProducts can use one or more product blocks, and product blocks can be\nshared by different products. Every product block that is defined, has a\nfile with the same name as the product block, to store the domain models\nin all its lifecycle stages. For example, the core port product block\nused by the core link product has a file called `core_port.py` in this\nfolder.\n\n#### products/services\n\nCollection of helper functions that deliver a service to product related\ncode. For example, the generation of descriptions, or payload for\nOSS/BSS API’s, for different product and product blocks. For example,\nthe folder `products/services/netbox/` contains the NetBox API payload\nservice.\n\n#### services\n\nSimilar to the product services but with code base wide helper\nfunctions. For example, the folder `services/netbox/` contains the\nservice that interfaces with the NetBox API.\n\n#### templates\n\nList of product configuration templates, with a template per product.\nBased on a template, currently an experimental feature, the WFO can\ngenerate skeleton code for: the product and product block domain models,\nall four types of workflows including input forms, registration of the\nproduct and workflows, and the corresponding database migration.\n\n#### translations\n\nThe translations for the WFO GUI for input form fields, subscriptions,\nsubscription instances, and workflows.\n\n#### utils\n\nA collection of helper functions that are not directly related to the\ncode base but are, for example, used to setup a deployment environment\nor generate documentation.\n\n#### workflows\n\nEvery product has a folder here, named after the product. Each folder\ncontains the collection of workflows for that product. Every workflow\nhas its own file, and the filename is prefixed with the type of\nworkflow. For example, the folder `workflows/port/` contains the\nworkflows for the Port product, and the file\n`workflows/port/create_port.py` contains the Port subscription create\nworkflow.\n\n#### shared\n\nThroughout the code base, shared folders are used that contain helper\nfunctions for that module and below. As good coding practice, it is best\nto define the helper functions as locally as possible.\n\n### Main application\n\nThe `main.py` can be as simple as shown below, and can be deployed by a\nASGI server like Uvicorn[^5].\n\n```python\nfrom orchestrator import OrchestratorCore\nfrom orchestrator.cli.main import app as core_cli\nfrom orchestrator.settings import AppSettings\n\nimport products\nimport workflows\n\napp = OrchestratorCore(base_settings=AppSettings())\napp.register_graphql()\n\nif __name__ == \"__main__\":\n    core_cli()\n```\n\nAll other orchestrator code is referenced by importing the `products`\nand `workflows` modules. The application is started with:\n\n```shell\nuvicorn --host localhost --port 8080 main:app\n```\n\nTo use the orchestrator command line interface use:\n\n```shell\npython main.py --help\n```\n\n### Implemented products\n\nIn the `product.product_types` module the following products are\ndefined:\n\n- Node\n- CoreLink\n- Port\n- L2vpn\n\nAnd in the `product.product_blocks` module the following product blocks\nare defined:\n\n- NodeBlock\n- CoreLinkBlock\n- CorePortBlock\n- PortBlock\n- SAPBlock\n- VirtualCircuitBlock\n\nUsually, the top-level product block if a product is named after the\nproduct, but this is not true for the top-level product block of the\nL2VPN product. The more generic name `VirtualCIrcuitBlock` allows the\nreuse of this product block by other services like Internet Access and\nL3VPN.\n\nThe Service Access Point (SAP) product block `SAPBlock` is used to\nencapsulate transport specific service endpoint information, in our case\nEthernet 802.1Q is used and the SAP holds the VLAN used on the indicated\nport.\n\nWhen this example orchestrator is deployed, it can create a growing\ngraph of product blocks as is shown below.\n\n### Product Hiearchy Diagram\n\u003ccenter\u003e\u003cimg src=\".pictures/subscriptions.png\" alt=\"Product block graph\" width=75% height=75%\u003e\u003c/center\u003e\n\n### How to use\n\nHuman workflows regarding the delivery of products to customers are\noften comprehensive. To limit the scope of this example orchestrator,\nbut still show the BCP while automating procedures, only inventory\nmanagement and provisioning are modeled. The implemented products and\nworkflows are designed with particular procedures in mind. For example,\nit is assumed that the following is administered in IMS outside of the\norchestrator:\n\n- Sites\n- Device roles\n- Device Manufactures\n- Device Types\n- IPv4 and IPv6 prefix for node loopback addresses\n- IPv4 and IPv6 prefix for core link addressing\n\nThe task Netbox Bootstrap takes care of initializing Netbox with a\ndefault set of this information. For convenience, a task Netbox Wipe is\nadded as well, that will remove all object from Netbox again, including\nthe ones that are created by the different workflows. Tasks can be found\nin the orchestrator UI in the `New tasks` pulldown on the `tasks` page.\n\n#### Node\n\nThe Node create workflow will read all configured, sites and device\nroles, manufactures types, and allows the user to choose appropriate\nvalues using dropdowns. The only thing that needs to be entered by hand\nis a unique name for the node and an optional description. This is\nenough to create a node subscription and administer the node in the IMS.\n\nNetwork interfaces are installed in the nodes by field engineers. The\nNode product has a “Update node interfaces” workflow that will discover\nall interfaces on a physical node and will add or remove interfaces from\nthe IMS as needed. For this implementation, this workflow will always\nreturn a preconfigured list of 10 and 100 Gbit/s network interfaces. In\nreal world implementation this could have been fetched from the network\ndevice with SNMP, NETCONF, gNMI, or something similar. Only basic\ninformation on the interfaces is added, which make them available to be\nused by the create workflows of the core link and customer port\nproducts.\n\nThere are variants of the node product that allow the creation of nodes\nfor different manufactures, and only the matching device types will be\nshown in the dropdown.\n\n#### CoreLink\n\nTo build a core link, at least two node subscription should already\nexist, this is to satisfy the constraint that the A and B side of the\ncore link need to be different. On each node, there should be at least\none port available that matches the requested core link speed.\n\n#### Port\n\nTo create a customer port, at least one node should exist with at least\none free interface of the requested port speed. The type of port can be\nuntagged, tagged, or link member, but note that currently only one\nnetwork service product is implemented, and that product only supports\ntagged ports.\n\n#### L2vpn\n\nTo create a L2VPN services for a customer, at least two customer ports\nshould exist, and every port can only be used once in the same L2VPN.\nThis product is only supported on tagged interfaces, and VLAN retagging\nis not supported.\n\n## Products\n\nThe Orchestrator uses the concept of a Product to describe what can be built to the end user. When\na user runs a workflow to create a Product, this results in a unique instance of that product called a Subscription.\nA Subscription is always tied to a certain lifecycle state (eg. Initial, Provisioning, Active, Terminated, etc) and\nis unique per customer. In other words a Subscription contains all the information needed to uniquely identify a certain\nresource owned by a user/customer that conforms to a certain definition, namely a Product.\n\n### Product description in Python\nProducts are described in Python classes called Domain Models. These classes are designed to help the\ndeveloper manage complex subscription models and interact with the\nobjects in a developer-friendly way. Domain models use Pydantic[^6] with some\nadditional functionality to dynamically cast variables from the\ndatabase, where they are stored as a string, to their correct type in\nPython at runtime. Pydantic uses Python type hints to validate that the\ncorrect type is assigned. The use of typing, when used together with\ntype checkers, already helps to make the code more robust, furthermore the use of Pydantic makes it possible to check\nvariables at runtime which greatly improves reliability.\n\n#### Example of \"Runtime typecasting/safety\"\nIn the example below we attempt to access a resource that has been stored in an instance of a product\n(subscription instance). It shows how it can be done directly through the ORM and it shows the added value of Domain\nModels on top of the ORM.\n\n**Serialisation direct from the database**\n\n```python\n\u003e\u003e\u003e some_subscription_instance_value = SubscriptionInstanceValueTable.get(\"ID\")\n\u003e\u003e\u003e instance_value_from_db = some_subscription_instance_value.value\n\u003e\u003e\u003e instance_value_from_db\n\"False\"\n\u003e\u003e\u003e if instance_value_from_db is True:\n...    print(\"True\")\n... else:\n...    print(\"False\")\n\"True\"\n```\n\n**Serialisation using domain models**\n```python\n\u003e\u003e\u003e class ProductBlock(ProductBlockModel):\n...     instance_from_db: bool\n...\n\u003e\u003e\u003e some_subscription_instance_value = SubscriptionInstanceValueTable.get(\"ID\")\n\u003e\u003e\u003e instance_value_from_db = some_subscription_instance_value.value\n\u003e\u003e\u003e type(instance_value_from_db)\n\u003cclass str\u003e\n\u003e\u003e\u003e\n\u003e\u003e\u003e subscription_model = SubscriptionModel.from_subscription(\"ID\")\n\u003e\u003e\u003e type(subscription_model.product_block.instance_from_db)\n\u003cclass bool\u003e\n\u003e\u003e\u003e\n\u003e\u003e\u003e subscription_model.product_block.instance_from_db\nFalse\n\u003e\u003e\u003e\n\u003e\u003e\u003e if subscription_model.product_block.instance_from_db is True:\n...    print(\"True\")\n... else:\n...    print(\"False\")\n\"False\"\n```\nAs you can see in the example above, interacting with the data stored in the database rows, helps with some of the heavy\nlifting, and makes sure the database remains generic and it's schema remains stable.\n\n#### Product Structure\nA Product definition has two parts in its structure. The Higher order product type that contains information describing\nthe product in a more general sense, and multiple layers of product blocks that logically describe the set of resources\nthat make up the product definition. The product type describes the fixed inputs and the top-level product blocks.\nThe fixed inputs are used to differentiate between variants of the same product, for example the speed of a network\nport. There is always at least one top level product block that contains\nthe resource types to administer the customer facing input. Beside\nresource types, the product blocks usually contain links to other\nproduct blocks as well. If a fixed input needs a custom type, then it is\ndefined here together with fixed input definition.\n\n#### Terminology\n * **Product:** A definition of what can be instantiated through a Subscription.\n * **Product Type:** The higher order definition of a Product. Many different Products can exist within a Product Type.\n * **Fixed Input:** Product attributes that discriminate the different Products that adhere to the same Product Type definition.\n * **Product Block:** A (logical) construct that contain references to other Product Blocks or Resource Types. It gives\n   structure to the product definition and defines what resources are related to other resources\n * **Resource Types:** Customer facing attributes that are the result of choices made by the user whilst filling an\n   input form. This can be a value the user chose, or an identifier towards a different system.\n\n### Product types\n\nThe product types in the code are upper camel cased. Per default, the product type is declared for the _inactive_,\n_provisioning_ and _active_ lifecycle states, and the product type name is\nsuffixed with the state if the lifecycle is not active. Usually, the\nlifecycle state starts with inactive, and then transitions through\nprovisioning to active, and finally to terminated. During its life, the\nsubscription, an instantiation of a product for a particular customer,\ncan transition from active to provisioning and back again many times,\nbefore it ends up terminated. The terminated state does not have its own\ntype definition, but will default to initial unless otherwise defined.\n\n#### Domain Model a.k.a Product Type Definition\n```python\nclass PortInactive(SubscriptionModel, is_base=True):\n    speed: PortSpeed\n    port: PortBlockInactive\n\nclass PortProvisioning(PortInactive, lifecycle=[SubscriptionLifecycle.PROVISIONING]):\n    speed: PortSpeed\n    port: PortBlockProvisioning\n\nclass Port(PortProvisioning, lifecycle=[SubscriptionLifecycle.ACTIVE]):\n    speed: PortSpeed\n    port: PortBlock\n```\n\nAs can be seen in the above example, the inactive product type\ndefinition is subclassed from SubScriptionModel, and the following\ndefinitions are subclassed from the previous one. This product has one\nfixed input called speed and one port product block (see below about\nnaming). Notice that the port product block matches the lifecycle of the\nproduct, for example, the PortInactive product has a PortBlockInactive\nproduct block, but it is totally fine to use product blocks from\ndifferent lifecycle states if that suits your use case.\n\n#### Fixed Input\nBecause a port is only available in a limited number of speeds, a\nseparate type is declared with the allowed values, see below.\n\n```python\nfrom enum import IntEnum\n\nclass PortSpeed(IntEnum):\n    _1000 = 1000\n    _10000 = 10000\n    _40000 = 40000\n    _100000 = 100000\n    _400000 = 400000\n```\n\nThis type is not only used to ensure that the speed fixed input can only\ntake these values, but is also used in user input forms to limit the\nchoices, and in the database migration to register the speed variant of\nthis product.\n\n#### Wiring it up in the Orchestrator\n\u003cdetails\u003e\n\u003csummary\u003eThis section contains advanced information about how to configure the Orchestrator. It is also possible to use\na more user friendly tool available \u003ca href=\"https://workfloworchestrator.\norg/orchestrator-core/reference-docs/cli/#generate\"\u003ehere\u003c/a\u003e.\nThis tool uses a configuration file to generate the boilerplate, migrations and configuration necessary to make use of\nthe product straight away.\n\u003c/summary\u003e\n\nProducts need to be registered in two places. All product variants have\nto be added to the `SUBSCRIPTION_MODEL_REGISTRY`, in\n`products/__init__.py`, as shown below.\n\n```python\nfrom orchestrator.domain import SUBSCRIPTION_MODEL_REGISTRY\nfrom products.product_types.core_link import CoreLink\n\nSUBSCRIPTION_MODEL_REGISTRY.update(\n    {\n        \"core link 10G\": CoreLink,\n        \"core link 100G\": CoreLink,\n    }\n)\n```\nAnd all variants also have to entered into the database using a\nmigration. The migration uses the create helper function from\n`orchestrator.migrations.helpers` that takes the following dictionary as\nan argument, see below. Notice that the name of the product and the\nproduct type need to match with the subscription model registry.\n\n```python\nfrom orchestrator.migrations.helpers import create\n\nnew_products = {\n    \"products\": {\n        \"core link 10G\": {\n            \"product_id\": uuid4(),\n            \"product_type\": \"CoreLink\",\n            \"description\": \"Core link\",\n            \"tag\": \"CORE_LINK\",\n            \"status\": \"active\",\n            \"product_blocks\": [\n                \"CoreLink\",\n                \"CorePort\",\n            ],\n            \"fixed_inputs\": {\n                \"speed\": CoreLinkSpeed._10000.value,\n            },\n        },\n}\n\ndef upgrade() -\u003e None:\n    conn = op.get_bind()\n    create(conn, new_products)\n```\n\u003c/details\u003e\n\n### Product blocks\n\nLike product types, the product blocks are declared for the _inactive_,\n_provisioning_ and _active_ lifecycle states. The name of the product block\nis suffixed with the word Block, to clearly distinguish them from the\nproduct types, and again suffixed by the state if the lifecycle is not\nactive.\n\nEvery time a subscription is transitioned from one lifecycle to another,\nan automatic check is performed to ensure that resource types that are\nnot optional are in fact present on that instantiation of the product\nblock. This safeguards for incomplete administration for that lifecycle\nstate.\n\n#### Resource Type lifecycle. When to use `None`\nThe resource types on an inactive product block are usually all\noptional to allow the creation of an empty product block instance. All\nresource types that are used to hold the user input for the subscription\nis stored using resource types that are not optional anymore in the\nprovisioning lifecycle state. All resource types used to store\ninformation that is generated while provisioning the subscription is\nstored using resource types that are optional while provisioning but are\nnot optional anymore for the active lifecycle state. Resource types that\nare still optional in the active state are used to store non-mandatory\ninformation.\n\n#### Example\n\n```python\nclass NodeBlockInactive(ProductBlockModel, product_block_name=\"Node\"):\n    type_id: int | None = None\n    node_name: str | None = None\n    ims_id: int | None = None\n    nrm_id: int | None = None\n    node_description: str | None = None\n\nclass NodeBlockProvisioning(NodeBlockInactive, lifecycle=[SubscriptionLifecycle.PROVISIONING]):\n    type_id: int\n    node_name: str\n    ims_id: int | None = None\n    nrm_id: int | None = None\n    node_description: str | None = None\n\nclass NodeBlock(NodeBlockProvisioning, lifecycle=[SubscriptionLifecycle.ACTIVE]):\n    type_id: int\n    node_name: str\n    ims_id: int\n    nrm_id: int\n    node_description: str | None = None\n```\n\nIn the simplified node product block shown above, the type and the name\nof the node are supplied by the user and stored on the\n`NodeBlockInactive`. Then, the subscription transitions to Provisioning\nand a check is performed to ensure that both pieces of information are\npresent on the product block. During the provisioning phase the node is\nadministered in IMS and the handle to that information is stored on the\n`NodeBlockProvsioning`. Next, the node is provisioned in the NRM and the\nhandle is also stored. If both of these two actions were successful, the\nsubscription is transitioned to Active and it is checked that the type\nand node name, and the IMS and NRM ID, are present on the product block.\nThe description of the node remains optional, even in the active state.\nThese checks ensure that information that is necessary for a particular\nstate is present so that the actions that are performed in that state do\nnot fail.\n\n#### Product Block customisation\nSometimes there are resource types that depend on information stored on\nother product blocks, even on linked product blocks that do not belong\nto the same subscription. This kind of types need to be calculated at\nrun time so that they include the most recent information. Consider the\nfollowing example of a, stripped down version, of a port and node\nproduct block, and a title for the port block that is generated\ndynamically.\n\n```python\nclass NodeBlock(NodeBlockProvisioning, lifecycle=[SubscriptionLifecycle.ACTIVE]):\n    node_name: str\n\nclass PortBlock(PortBlockProvisioning, lifecycle=[SubscriptionLifecycle.ACTIVE]):\n    port_name: str\n    node: NodeBlock\n\n    @serializable_property\n    def title(self) -\u003e str:\n        return f\"{self.port_name} on {self.node.node_name}\"\n\nclass Port(PortProvisioning, lifecycle=[SubscriptionLifecycle.ACTIVE]):\n    port: PortBlock\n```\n\nA `@serializable_property` has been added that will dynamically render\nthe title of the port product block. Even after a modify workflow was\nrun to change the node name on the node subscription, the title of the\nport block will always be up to date. The title can be referenced as any\nother resource type using subscription.port.title. This is not a random\nexample, the title of a product block is used by the orchestrator GUI\nwhile displaying detailed subscription information.\n\n## Workflows - Basics\n\nWorkflows are used to orchestrate the lifecycle of a Product Subscription and process the user or systems intent and\napply that to the service. As mentioned above a Subscription is created, then modified `N` number of times, after\nwhich it is terminated. During it's life a Subscription may also be validated on a regular basis to check whether\nthere is any drift between the state captured in the Orchestrator and actual state on the system. This workflow is\nslightly different compared to the workflows that process intent and apply that to a system, as it does not modify\nthe system.\n\nFour types of workflows are defined, three lifecycle related ones to\ncreate, modify and terminate subscriptions, and a fourth one to validate\nsubscriptions against the OSS and BSS. The decorators\n`@create_workflow`, `@modify_workflow`, `@terminate_workflow`, and\n`@validate_workflow` are used to define the different types of workflow,\nand the `@step` decorator is used to define workflow steps that can be\nused in any type of workflow.\n\n### Workflow Architecture - Passing information from one step to the next\n\nInformation between workflow steps is passed using `State`, which is\nnothing more than a collection of key/value pairs, in Python represented\nby a `Dict`, with string keys and arbitrary values. Between steps the\n`State` is serialized to JSON and stored in the database. The step\ndecorator is used to turn a function into a workflow step, all arguments\nto the step function will automatically be initialised with the value\nfrom the matching key in the `State`. In turn the step function will\nreturn a `Dict` of new and/or modified key/value pairs that will be\nmerged into the `State` to be consumed by the next step. The\nserialization and deserialization between JSON and the indicated Python\ntypes is done automatically. That is why it is important to correctly\ntype the step function parameters.\n\n#### Example\nGiven this function, when a user correctly makes use of the step decorator it is very easy to extract variables and\nmake a calculation. It creates readable code, that is easy to understand and reason about. Furthermore the variables\nbecome available in the step in their correct type according to the domain model. Logic errors due wrong type\ninterpretation are much less prone to happen.\n\n**Bad use of the step decorator**\n```python\n@step(\"A Bad example of using input params\")\ndef my_ugly_step(state: State) -\u003e State:\n    variable_1 = int(state[\"variable_1\"])\n    variable_2 = str(state[\"varialble_2\"])\n    subscription = SubscriptionModel.from_subscription_id(state[\"subscription_id\"])\n\n    if variable_1 \u003e 42:\n        subscription.product_block_model.variable_1 = -1\n        subscription.product_block_model.variable_2 = \"Infinity\"\n    else:\n        subscription.product_block_model.variable_1 = variable_1\n        subscription.product_block_model.variable_2 = variable_2\n\n    state[\"subscription\"] = subscription\n    return state\n```\nIn the above example you see we do a simple calculation based on `variable_1`. When computing with even more\nvariables, you van imagine how unreadable the function will be. Now consider the next example.\n\n**Good use of the step decorator**\n```python\n@step(\"Good use of the input params functionality\")\ndef my_beautiful_step(variable_1: int, variable_2: str, subscription: SubscriptionModel) -\u003e State:\n    if variable_1 \u003e 42:\n        subscription.product_block_model.variable_1 = -1\n        subscription.product_block_model.variable_2 = \"Infinity\"\n    else:\n        subscription.product_block_model.variable_1 = variable_1\n        subscription.product_block_model.variable_2 = variable_2\n\n    return state | {\"subscriotion\": subscription}\n```\n\nAs you can see the Orchestrator the orchestrator helps you a lot to condense the logic in your function. The `@step`\ndecorator does the following:\n\n* Loads the previous steps state from the database.\n* Inspects the step functions signature\n* Finds the arguments in the state and injects them as function arguments to the step function\n* It casts them to the correct type by using the type hints of the step function.\n* Finally it updates the state of the workflow and persists all model changes to the database upon reaching the\n  `return` of the step function.\n\n### Forms\n\nThe input form is where a user can enter the details for a subscription\non a certain product at the start of the workflow, or can enter\nadditional information during the workflow. The input forms are\ndynamically generated in the backend and use Pydantic to define the type\nof the input fields. This also allows for the definition of input\nvalidations. Input forms are (optionally) used by all types of workflows\nto gather and validate user input. It is possible to have more than one\ninput form, with the ability to navigate back and forth between the\nforms, until the last input form is submitted, and the first (or next)\nstep of the workflow is started. This allows for on-the-fly generation\nof input forms, where the content of the following form(s) depend on the\ninput of the previous form(s). For example, when creating a core link\nbetween two nodes, a first input form could ask to choose two nodes from\na list of active nodes, and the second form will present two lists with\nports on these two nodes to choose from.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eBest Practices for writing workflows\u003c/b\u003e\u003c/summary\u003e\nWhile developing a new product, the workflows can be written in any\norder. For those that use a test-driven development style probably will\nstart with the validate workflow. But in general people start with the\ncreate workflow as it helps to discuss the product model (the\ninformation involved) and the workflows (the procedures involved) with\nthe stakeholders to get the requirements clear. Once the minimal viable\ncreate workflow is implemented, the validate workflow can be written to\nensure that all information is administered correctly in all touched OSS\nand BSS and is not changed again by hand because human workflows were\nnot correctly adapted yet. Then after the terminate workflow is written,\nthe complete lifecycle of the product can be tested. Even when the\nmodify is not implemented, a change to a subscription can be carried out\nby terminating the subscription and creating it again with the modified\ninput. Finally, the modify workflow is implemented to allow changes to a\nsubscription with minimal or no impact to the customer.\n\u003c/details\u003e\n\n#### Form _Magic_\nAs mentioned before, forms are dynamically created from the backend. This means, **little to no** frontend coding is\nneeded to make complex wizard like input forms available to the user. When selecting an action in the UI. The first\nthing the frontend does is make an api call to load a form from the backend. The resulting `JSONschema` is parsed\nand the correct widgets are loaded in the frontend. Upon submit this is posted to the backend that does all\nvalidation and signals to the user if there are any errors. The following forms are supported:\n\n* Multiselect\n* Drop-down\n* Text field (restricted)\n* Number (float and dec)\n* Radio\n\n## Workflow examples\nWhat follows are a few examples of how workflows implement the best common practices implemented by SURF. It\nexplains in detail what a typical workflow could look like for provision in network element. These examples can be\nexamined in greater detail by exploring the `.workflows.node` directory.\n\n### Create workflow\n\nA create workflow needs an initial input form generator and defines the\nsteps to create a subscription on a product. The `@create_workflow`\ndecorator adds some additional steps to the workflow that are always\npart of a create workflow. The steps of a create workflow in general\nfollow the same pattern, as described below using the create node\nworkflow as an example.\n\n```python\n@create_workflow(\"Create node\", initial_input_form=initial_input_form_generator)\ndef create_node() -\u003e StepList:\n    return (\n        begin\n        \u003e\u003e construct_node_model\n        \u003e\u003e store_process_subscription(Target.CREATE)\n        \u003e\u003e create_node_in_ims\n        \u003e\u003e reserve_loopback_addresses\n        \u003e\u003e provision_node_in_nrm\n    )\n```\n\n1. Collect input from user (`initial_input_form`)\n2. Instantiate subscription (`construct_node_model`):\n    1. Create inactive subscription model\n    2. assign user input to subscription\n    3. transition to subscription to provisioning\n3. Register create process for this subscription (`store_process_subscription`)\n4. Interact with OSS and/or BSS, in this example\n    1. Administer subscription in IMS (`create_node_in ims`)\n    2. Reserve IP addresses in IPAM (`reserve_loopback_addresses`)\n    3. Provision subscription in the network (`provision_node_in_nrm`)\n5. Transition subscription to active and ‘in sync’ (`@create_workflow`)\n\nAs long as every step remains as idempotent as possible, the work can be\ndivided over fewer or more steps as desired.\n\n#### Input Form\nThe input form is created by subclassing the `FormPage` and add the\ninput fields together with the type and indication if they are optional\nor not. Additional form settings can be changed via the Config class,\nlike for example the title of the form page.\n\n```python\nclass CreateNodeForm(FormPage):\n    model_config = ConfigDict(title=product_name)\n\n    role_id: NodeRoleChoice\n    node_name: str\n    node_description: str | None = None\n```\n\nBy default, Pydantic validates the input against the specified type and\nwill signal incorrect input and/or missing but required input fields.\nType annotations can be used to describe additional constraints, for\nexample a check on the validity of the entered VLAN ID can be specified\nas shown below, the type `Vlan` can then be used instead of `int`.\n\n```python\nVlan = Annotated[int, Ge(2), Le(4094), doc(\"Allowed VLAN ID range.\")]\n```\n\nThe node role is defined as type Choice and will be rendered as a\ndropdown that is filled with a mapping between the role IDs and names as\ndefined in Netbox.\n\n```python\ndef node_role_selector() -\u003e Choice:\n    roles = {str(role.id): role.name for role in netbox.get_device_roles()}\n    return Choice(\"RolesEnum\", zip(roles.keys(), roles.items()))\n\nNodeRoleChoice: TypeAlias = cast(type[Choice], node_role_selector())\n```\n\nWhen more than one item needs to be selected, a `choice_list` can be\nused to specify the constraints, for example to select two ports for a\npoint-to-point service:\n\n```python\ndef ports_selector(number_of_ports: int) -\u003e type[list[Choice]]:\n    subscriptions = subscriptions_by_product_type(\"Port\", [SubscriptionLifecycle.ACTIVE])\n    ports = {str(subscription.subscription_id): subscription.description for subscription in subscriptions)}\n    return choice_list(\n        Choice(\"PortsEnum\", zip(ports.keys(), ports.items())),\n        min_items=number_of_ports,\n        max_items=number_of_ports,\n        unique_items=True,\n    )\n\nPortsChoiceList: TypeAlias = cast(type[Choice], ports_selector(2))\n```\n\n#### Extra Validation between dependant fields\nValidations between multiple fields is also possible by making use of\nthe Pydantic `@model_validator` decorator that gives access to all\nfields. To check if the A and B side of a point-to-point service are not\non the same network node one could use:\n\n```python\n@model_validator(mode=\"after\")\ndef separate_nodes(self) -\u003e \"SelectNodes\":\n    if self.node_subscription_id_b == self.node_subscription_id_a:\n        raise ValueError(\"node B cannot be the same as node A\")\n    return self\n```\n\nFor more information on validation, see the [Pydantic\nValidators](https://docs.pydantic.dev/latest/concepts/validators/)\ndocumentation\n\nFinally, a summary form is shown with the user supplied values. When a\nvalue appears to be incorrect, the user can go back to the previous form\nto correct the mistake, otherwise, when the form is submitted, the\nworkflow is kicked off.\n\n```python\nsummary_fields = [\"role_id\", \"node_name\", \"node_description\"]\nyield from create_summary_form(user_input_dict, product_name, summary_fields)\n```\n\n### Modify workflow\n\nA modify workflow also follows a general pattern, like described below.\nThe `@modify_workflow` decorator adds some additional steps to the\nworkflow that are always needed.\n\n```pyrhon\n@modify_workflow(\"Modify node\", initial_input_form=initial_input_form_generator)\ndef modify_node() -\u003e StepList:\n    return (\n        begin\n        \u003e\u003e set_status(SubscriptionLifecycle.PROVISIONING)\n        \u003e\u003e update_subscription\n        \u003e\u003e update_node_in_ims\n        \u003e\u003e update_node_in_nrm\n        \u003e\u003e set_status(SubscriptionLifecycle.ACTIVE)\n    )\n```\n\n1. Collect input from user (`initial_input_form`)\n2. Necessary subscription administration (`@modify_workflow`):\n    1. Register modify process for this subscription\n    2. Set subscription ‘out of sync’ to prevent the start of other processes\n3. Transition subscription to Provisioning (`set_status`)\n4. Update subscription with the user input\n5. Interact with OSS and/or BSS, in this example\n    1. Update subscription in IMS (`update_node_in ims`)\n    2. Update subscription in NRM (`update_node_in nrm`)\n6. Transition subscription to active (`set_status`)\n7. Set subscription ‘in sync’ (`@modify_workflow`)\n\nLike a create workflow, the modify workflow also uses an initial input\nform but this time to only collect the values from the user that need to\nbe changed. Usually, only a subset of the values may be changed. To\nassist the user, additional values can be shown in the input form using\n`ReadOnlyField`. In the example below, the name of the node is shown but\ncannot be changed, the node status can be changed and the dropdown is\nset to the current node status, and the node description is still\noptional.\n\n```python\nclass ModifyNodeForm(FormPage):\n    node_name: ReadOnlyField(port.node.node_name)\n    node_status: NodeStatusChoice = node.node_status\n    node_description: str | None = node.node_description\n```\n\nAfter a summary form has been shown that lists the current and the new\nvalues, the modify workflow is started.\n\n```python\nsummary_fields = [\"node_status\", \"node_name\", \"node_description\"]\nyield from modify_summary_form(user_input_dict, subscription.node, summary_fields)\n```\n\n### Terminate workflow\n\nAt the end of the subscription lifecycle, the terminate workflow updates\nall OSS and BSS accordingly, and the `@terminate_workflow` decorator\ntakes care of most of the necessary subscription administration.\n\n```python\n@terminate_workflow(\"Terminate node\",\ninitial_input_form=initial_input_form_generator)\ndef terminate_node() -\u003e StepList:\n    return (\n        begin\n        \u003e\u003e load_initial_state\n        \u003e\u003e delete_node_from_ims\n        \u003e\u003e deprovision_node_in_nrm\n    )\n```\n\n1. Show subscription details and ask user to confirm termination (`initial_input_form`)\n2. Necessary subscription administration (`@terminate_workflow`):\n    1. Register terminate process for this subscription\n    2. Set subscription ‘out of sync’ to prevent the start of other processes\n3. Get subscription and add information for following steps to the State (`load_initial_state`)\n4. Interact with OSS and/or BSS, in this example\n    1. Delete node in IMS (`delete_node_in ims`)\n    2. Deprovision node in NRM (`deprovision_node_in_nrm`)\n5. Necessary subscription administration (`@terminate_workflow`)\n    1. Transition subscription to terminated\n    2. Set subscription ‘in sync’\n\nThe initial input form for the terminate workflow is very simple, it\nonly has to show the details of the subscription:\n\n```python\nclass TerminateForm(FormPage):\n    subscription_id: DisplaySubscription = subscription_id\n```\n\n### Validate workflows\n\nAnd finally, the validate workflow, used to check if the information in\nall OSS and BSS is still the same with the information in the\nsubscription. One way to do this is to reconstruct the payload sent to\nthe external system using information queried from that system, and\ncompare this with the payload that would have been sent by generating a\npayload based on the current state of the subscription. The\n`@validate_workflow` decorator takes care of necessary subscription\nadministration. There is no initial input form for this type of\nworkflow.\n\n```python\n@validate_workflow(\"Validate l2vpn\")\ndef validate_l2vpn() -\u003e StepList:\n    return (\n        begin\n        \u003e\u003e validate_l2vpn_in_ims\n        \u003e\u003e validate_l2vpn_terminations_in_ims\n        \u003e\u003e validate_vlans_on_ports_in_ims\n   )\n```\n\n1. Necessary subscription administration (`@validate_workflow`):\n    1. Register validate process for this subscription\n    2. Set subscription ‘out of sync’, even when subscription is already out of sync\n2. One or more steps to validate the subscription against all OSS and BSS:\n    1. Validate subscription against IMS:\n        1. `validate_l2vpn_in_ims`\n        2. `validate_l2vpn_terminations_in_ims`\n        3. `validate_vlans_on_ports_in_ims`\n3. Set subscription ‘in sync’ again (`@validate_workflow`)\n\nWhen one of the validation steps fail, the subscription will stay ‘out\nof sync’, prohibiting other workflows to be started for this\nsubscription. The failed validation step can be retried as many times as\nneeded until it succeeds, which finally will set the subscription ‘in\nsync’ and allow other workflows to be started again. This safeguards\nworkflows to be started for subscription with mismatching information in\nOSS and BSS which would make these workflows likely to fail.\n\nIt is better to limit the number of validations done in each step. This\nwill make it easier to see in a glance what discrepancy was found and\nwill make a retry of the failed step much faster. A commonly used\nstrategy is to use separate steps for each OSS and BSS, and separate\nsteps per external system for each payload that was sent. This can be\ndone by comparing a payload created for a product block in the\norchestrator with a payload that is generated by querying the external\nsystem.\n\nNot only validations per subscription can be done, is also possible to\nvalidate other requirements. For example, to make sure that there are no\nL2VPNs administered in IMS that do not have a matching subscription in\nthe orchestrator, a task (a workflow with `Target.SYSTEM`) can be\nwritten that will retrieve a list of all L2VPNs from IMS and compare it\nagainst a list of all L2VPN subscription from the orchestrator.\n\n## Services\n\nServices are collections of helper functions that deliver a service to\nother parts of the code base. The common programming pattern of function\noverloading is used for the implementation of the service. Function\noverloading allows the use of multiple functions with the same name that\nwill execute the right function based on the type of the parameters.\nPython does not allow function overloading, but similar functionality\ncan be achieved through the use of the single dispatch feature that is\npart of the standard Python library.\n\nFirst, an interface is defined and decorated with `@singledispatch`.\nThen different nameless functions can be registered that implement that\ninterface but for different parameters. Note that only the first\nparameter will be taken into account to decide which one of the\nfunctions need to be execute.\n\nA helper function called `single_dispatch_base()` is used to keep track\nof all registered functions and the type of their first argument.  This\nallows for more informative error messages when the single dispatch\nfunction is called with an unsupported parameter.\n\n### Subscription descriptions\n\nAn example of a service is the generation of descriptions for\nsubscriptions or product block instances, the description is generated\nbased on the type of subscription or product block instance. Organising\nit this way, there is one place where every description is being\ngenerated, and changes to the way a description is generated will\nautomatically appear in all places where that description is being used.\n\nThe description single dispatch allows a first argument of type product\nmodel, product block model, or subscription model, and will call the\nmatching function.\n\n```python\n@singledispatch\ndef description(model: Union[ProductModel, ProductBlockModel, SubscriptionModel]) -\u003e str:\n    return single_dispatch_base(description, model)\n```\n\nThen, implementations of the description function can be registered,\nlike the generation of a description for a Node product, starting from\nthe provisioning lifecycle state, that will show the name of the node\nfollowed by the status of the node in parenthesis.\n\n```python\n@description.register\ndef _(product: NodeProvisioning) -\u003e str:\n    return f\"node {product.node.node_name} ({product.node.node_status})\"\n```\n\n### Netbox\n\nThe Netbox service is an interplay between several single dispatch\nfunctions, one to generate the payload for a specific product block, and\ntwo others that create or modify an object in Netbox based on the type\nof payload. The Pynetbox[^7] Python API client library is used to\ninterface with Netbox.\n\n#### Payload\n\nThe `build_payload()` single dispatch allows a first argument of type\nproduct block model, and a subscription model parameter that is used\nwhen related information is needed from other parts of the subscription.\nThe specified return type is the base class that is used for all Netbox\npayload definitions.\n\n```python\n@singledispatch\ndef build_payload(model: ProductBlockModel, subscription: SubscriptionModel, **kwargs: Any) -\u003e netbox.NetboxPayload:\n    return single_dispatch_base(build_payload, model)\n```\n\nWhen the payload is generated from a product block, the correct mapping\nis made between the types used in the orchestrator and the types used in\nthe OSS or BSS. For example, the Port product block maps on the\nInterface type in Netbox, as can be seen below.\n\n```python\n@build_payload.register\ndef _(model: PortBlockProvisioning, subscription: SubscriptionModel) -\u003e netbox.InterfacePayload:\n    return build_port_payload(model, subscription)\n\ndef build_port_payload(model: PortBlockProvisioning, subscription: SubscriptionModel) -\u003e netbox.InterfacePayload:\n    return netbox.InterfacePayload(\n        device=model.node.ims_id,\n        name=model.port_name,\n        type=model.port_type,\n        tagged_vlans=model.vlan_ims_ids,\n        mode=\"tagged\" if model.port_mode == PortMode.TAGGED else \"\",\n        description=model.port_description,\n        enabled=model.enabled,\n        speed=subscription.speed * 1000,\n    )\n```\n\nThe values from the product block are copied to the appropriate place in\nthe Interface payload. The interface payload field names match the ones\nthat are expected by Netbox. The speed of the interface is taken from\nthe fixed input speed with the same name on the subscription, the\nmultiplication by 1000 is to convert between Mbit/s and Kbit/s.\n\n#### Create\n\nTo create an object in Netbox based on the type of Netbox payload, the\nsingle dispatch `create()` is used:\n\n```python\n@singledispatch\ndef create(payload: NetboxPayload, **kwargs: Any) -\u003e int:\n    return single_dispatch_base(create, payload)\n```\n\nWhen registering the payload type, a keyword argument is used to inject\nthe matching endpoint on the Netbox API that is used to create the\ndesired object. In the example below can be seen that interface payload\nis to be used against the `api.dcim.interfaces` endpoint.\n\n```python\n@create.register\ndef _(payload: InterfacePayload, **kwargs: Any) -\u003e int:\n    return _create_object(payload, endpoint=api.dcim.interfaces)\n```\n\nFinally, the payload is used to generate a dictionary as expected by\nthat Netbox API endpoint. Notice that the names of the fields of the\nNetbox payload have to match the names of the fields that are expected\nby the Netbox API.\n\n```python\ndef _create_object(payload: NetboxPayload, endpoint: Endpoint) -\u003e int:\n    object = endpoint.create(payload.dict())\n    return object.id\n```\n\nThe ID of the object that is created in Netbox is returned so that it\ncan be registered in the subscription for later reference, e.q. when the\nobject needs to be modified or deleted.\n\n#### Update\n\nThe single dispatch `update()` is defined in a similar way, the only\ndifference is that an additional argument is used to specify the ID of\nthe object in Netbox that needs to be updated.\n\n```python\n@update.register\ndef _(payload: InterfacePayload, id: int, **kwargs: Any) -\u003e bool:\n    return _update_object(payload, id, endpoint=api.dcim.interfaces)\n```\n\nThe ID is used to fetch the object from the Netbox API, update the\nobject with the dictionary created from the supplied payload, and send\nthe update to Netbox.\n\n```python\ndef _update_object(payload: NetboxPayload, id: int, endpoint: Endpoint) -\u003e bool:\n    object = endpoint.get(id)\n    object.update(payload.dict())\n    return object.save()\n```\n\n#### Get\n\nThe Netbox service defines other helpers as well. For example, to get an\nsingle object, or a list of objects, of a specific type from Netbox.\n\n```python\ndef get_interfaces(**kwargs) -\u003e List:\n    return api.dcim.interfaces.filter(**kwargs)\n\ndef get_interface(**kwargs):\n    return api.dcim.interfaces.get(**kwargs)\n```\n\nBoth types of helpers accept keyword arguments that can be used to\nspecify the object(s) that are wanted. For example `get_inteface(id=3)`\nwill fetch the single interface object with ID equal to 3 from Netbox.\nAnd `get_interfaces(speed=1000000)` will get a list of all interface\nobjects from Netbox that have a speed of 1Gbit/s.\n\n#### Delete\n\nAnother set of helpers is defined to delete objects from Netbox. For\nexample, to delete an Interface object from Netbox, see below.\n\n```python\ndef delete_interface(**kwargs) -\u003e None:\n    delete_from_netbox(api.dcim.interfaces, **kwargs)\n```\n\nThe keyword arguments allow for different ways to select the object to\nbe deleted, as long as the supplied arguments result in a single object.\n\n```python\ndef delete_from_netbox(endpoint, **kwargs) -\u003e None:\n    object = endpoint.get(**kwargs)\n    object.delete()\n```\n\n#### Product block to Netbox object mapping\n\nThe modeling used in the orchestrator does not necessarily have to\nmatch exactly with the modeling in your OSS or BSS. In many cases,\ndifferent names are used, or a one-to-many or many-to-one relation needs\nto be created. To make a future transition to a different external\nsystem as easy as possible, any needed mappings, or translations between\nthe models, are isolated in the workflow step(s) that deal with those\nexternal systems as much as possible.\n\nThe diagram below shows the product blocks and relations as used in a\ncore link between two nodes, and how they map to the objects as\nadministered in Netbox. The product blocks are in orange and the Netbox\nobjects are in green.\n\n\u003ccenter\u003e\u003cimg src=\".pictures/netbox_node_core_link.png\" alt=\"Node and core link type mapping\" width=45% height=45%\u003e\u003c/center\u003e\n\nAnd the following diagram shows the mapping and relation between product\nblocks and Netbox objects for a L2VPN on customer ports between two\nnodes.\n\n\u003ccenter\u003e\u003cimg src=\".pictures/netbox_node_port_l2vpn.png\" alt=\"Node, port and L2VPN type mapping\" width=40% height=40%\u003e\u003c/center\u003e\n\n### Federation\n\nWFO and NetBox both use the GraphQL framework Strawberry[^9] which supports Apollo Federation[^8]. This allows to expose both GraphQL backends as a single *supergraph*. WFO can be integrated with any other GraphQL backend that supports[^10] federation and of which you can modify the code. In case of NetBox we don't have direct control over the source code, so we patched it for purposes of demonstration.\n\n#### Requirements\n\nThe following is required to facilitate GraphQL federation on top of WFO and other GraphQL backend(s):\n\n* WFO must be configured with `FEDERATION_ENABLED=True`\n  * [`docker/orchestrator/orchestrator.env`](docker/orchestrator/orchestrator.env)\n* The other backend must also enable federation\n  * NetBox: [`docker/netbox/Dockerfile`](docker/netbox/Dockerfile)\n* In both backends set a federation key on the GraphQL types to join\n  * WFO: [`graphql_federation.py`](graphql_federation.py)\n  * NetBox: [`docker/netbox/patch_federation.py`](docker/netbox/patch_federation.py)\n* Define the supergraph config with both backends\n  * [`docker/federation/supergraph-config.yaml`](docker/federation/supergraph-config.yaml)\n* Compile the supergraph schema with rover[^12]\n  * `rover-compose` startup service in [`docker-compose.yml`](docker-compose.yml)\n* Run Apollo Router to serve the supergraph\n  * `federation` service in [`docker-compose.yml`](docker-compose.yml)\n\nFor more information on federating new GraphQL types, or the existing WFO GraphQL types, please refer to our reference documentation[^11].\n\n#### Example queries\n\nThe following queries assume a running docker-compose environment with 2 configured Nodes. We'll demonstrate how 2 separate GraphQL queries can now be performed in 1 federated query.\n\n**NetBox**: NetBox device details can be queried from the NetBox GraphQL endpoint at http://localhost:8000/graphql/ (be sure to authenticate first with admin/admin)\n\n```graphql\nquery GetNetboxDevices {\n  device_list {\n    id\n    name\n    device_type {\n      manufacturer {\n        name\n      }\n    }\n    site {\n      name\n    }\n  }\n}\n```\n\n\u003cimg src=\".pictures/graphql_netbox.png\" alt=\"netbox query\" width=\"75%\" height=\"auto\"\u003e\n\n**WFO**: Node subscriptions can be queried from the WFO GraphQL endpoint at http://localhost:8080/api/graphql\n\n```graphql\nquery GetSubscriptions {\n  subscriptions(filterBy:\n    \t{field: \"product\", value: \"Node\"}\n  ) {\n    page {\n      ... on NodeSubscription {\n        subscriptionId\n        description\n        node {\n          imsId\n          nodeName\n        }\n      }\n    }\n  }\n}\n```\n\n\u003cimg src=\".pictures/graphql_wfo.png\" alt=\"wfo query\" width=\"75%\" height=\"auto\"\u003e\n\n**Federation**: Node subscriptions enriched with NetBox device details can be queried from the Federation endpoint at http://localhost:4000\n\n```graphql\nquery GetEnrichedSubscriptions {\n  subscriptions(filterBy:\n    {field: \"product\", value: \"Node\"}\n  ) {\n    page {\n      ... on NodeSubscription {\n        subscriptionId\n        description\n        node {\n          imsId\n          nodeName\n          netboxDevice {\n            name\n            device_type {\n              manufacturer {\n                name\n              }\n            }\n            site {\n              name\n            }\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n\u003cimg src=\".pictures/graphql_federation.png\" alt=\"federated query\" width=\"75%\" height=\"auto\"\u003e\n\n## Configuration\n\nEnvironment variables and orchestrator-core can be overridden for development purposes. Please refer to [this documentation](./docker/overrides/configuration.md) for more details.\n\n## Glossary\n\n\u003cdl\u003e\n\u003cdt\u003e API  \u003c/dt\u003e\u003cdd\u003e  Application Programming Interface \u003c/dd\u003e\n\u003cdt\u003e ASGI \u003c/dt\u003e\u003cdd\u003e  Asynchronous Server Gateway Interface \u003c/dd\u003e\n\u003cdt\u003e BCP \u003c/dt\u003e\u003cdd\u003e Best Common Practice \u003c/dd\u003e\n\u003cdt\u003e BSS \u003c/dt\u003e\u003cdd\u003e Business Support System \u003c/dd\u003e\n\u003cdt\u003e gNMI \u003c/dt\u003e\u003cdd\u003e gRPC Network Management Interface \u003c/dd\u003e\n\u003cdt\u003e gRPC \u003c/dt\u003e\u003cdd\u003e generic Remote Procedure Call \u003c/dd\u003e\n\u003cdt\u003e GUI \u003c/dt\u003e\u003cdd\u003e Graphical User Interface \u003c/dd\u003e\n\u003cdt\u003e IMS \u003c/dt\u003e\u003cdd\u003e Inventory Management System \u003c/dd\u003e\n\u003cdt\u003e IPAM \u003c/dt\u003e\u003cdd\u003e IP Address Management \u003c/dd\u003e\n\u003cdt\u003e L2VPN \u003c/dt\u003e\u003cdd\u003e Layer 2 Virtual Private Network \u003c/dd\u003e\n\u003cdt\u003e L3VPN \u003c/dt\u003e\u003cdd\u003e Layer 3 Virtual Private Network \u003c/dd\u003e\n\u003cdt\u003e NETCONF \u003c/dt\u003e\u003cdd\u003e NETwork CONFiguration protocol \u003c/dd\u003e\n\u003cdt\u003e NREN \u003c/dt\u003e\u003cdd\u003e National Research and Education Network \u003c/dd\u003e\n\u003cdt\u003e OSS \u003c/dt\u003e\u003cdd\u003e Operation Support System \u003c/dd\u003e\n\u003cdt\u003e REST \u003c/dt\u003e\u003cdd\u003e REpresentational state transfer \u003c/dd\u003e\n\u003cdt\u003e SAP \u003c/dt\u003e\u003cdd\u003e Service Access Point \u003c/dd\u003e\n\u003cdt\u003e SNMP \u003c/dt\u003e\u003cdd\u003e Simple Network Management Protocol \u003c/dd\u003e\u003cdt\u003e\n\u003cdt\u003e WFO \u003c/dt\u003e\u003cdd\u003e WorkFlow Orchestrator \u003c/dd\u003e\n\u003c/dl\u003e\n\n[^1]: M7.3 Common NREN Network Service Product Models -\nhttps://resources.geant.org/wp-content/uploads/2023/06/M7.3_Common-NREN-Network-Service-Product-Models.pdf\n\n[^2]: Workflow Orchestrator website -\nhttps://workfloworchestrator.org/orchestrator-core/\n\n[^3]: Netbox is a tool for data center infrastructure management and IP\naddress management - https://netbox.dev\n\n[^4]: The Python SQL Toolkit and Object Relational Mapper -\nhttps://www.sqlalchemy.org\n\n[^5]: ASGI server Uvicorn - https://www.uvicorn.org\n\n[^6]: Pydantic is a data validation library for Python -\nhttps://pydantic.dev/\n\n[^7]: Pynetbox Python API - https://github.com/netbox-community/pynetbox\n\n[^8]: Apollo Federation - https://www.apollographql.com/docs/federation/\n\n[^9]: Strawberry Federation - https://strawberry.rocks/docs/federation/introduction\n\n[^10]: Apollo Federation support - https://www.apollographql.com/docs/federation/building-supergraphs/supported-subgraphs\n\n[^11]: WFO GraphQL Documentation - https://workfloworchestrator.org/orchestrator-core/reference-docs/graphql/\n\n[^12]: Apollo Rover - https://www.apollographql.com/docs/rover/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworkfloworchestrator%2Fexample-orchestrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fworkfloworchestrator%2Fexample-orchestrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworkfloworchestrator%2Fexample-orchestrator/lists"}