{"id":20588988,"url":"https://github.com/ciscodevnet/sdwan-cor-labinfra","last_synced_at":"2025-04-14T22:04:37.853Z","repository":{"id":142020902,"uuid":"407616064","full_name":"CiscoDevNet/sdwan-cor-labinfra","owner":"CiscoDevNet","description":"Set of Terraform scripts to spin up virtual lab infra for Cisco Cloud onRamp (CoR) for Multicloud","archived":false,"fork":false,"pushed_at":"2023-10-25T17:59:01.000Z","size":911,"stargazers_count":13,"open_issues_count":0,"forks_count":8,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-14T22:04:20.871Z","etag":null,"topics":["aws","cisco","cloud","sd-wan","sdwan"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CiscoDevNet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-17T16:51:38.000Z","updated_at":"2024-09-23T14:24:07.000Z","dependencies_parsed_at":"2023-07-07T07:45:40.314Z","dependency_job_id":"eba5b420-1364-4918-9afc-ce0da37a2945","html_url":"https://github.com/CiscoDevNet/sdwan-cor-labinfra","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CiscoDevNet%2Fsdwan-cor-labinfra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CiscoDevNet%2Fsdwan-cor-labinfra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CiscoDevNet%2Fsdwan-cor-labinfra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CiscoDevNet%2Fsdwan-cor-labinfra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CiscoDevNet","download_url":"https://codeload.github.com/CiscoDevNet/sdwan-cor-labinfra/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248968737,"owners_count":21191159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cisco","cloud","sd-wan","sdwan"],"created_at":"2024-11-16T07:27:25.043Z","updated_at":"2025-04-14T22:04:37.828Z","avatar_url":"https://github.com/CiscoDevNet.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cisco SD-WAN Cloud onRamp for Multi-Cloud lab\n\n## TL;DR Summary\n\nThis project creates virtual infrastructure with Terraform on public cloud, which will be used for Cisco SD-WAN Cloud onRamp for Multi-cloud lab.\nSome modules can be used separate from each other. For example Centralized Firewall Design or Multi-Region Fabric use cases can be deployed totally separated.\n\nIf you are new to Cisco Cloud onRamp, please visit [Cisco Online Documentation](https://www.cisco.com/c/en/us/solutions/enterprise-networks/sd-wan/cloud-onramp.html) \nand do the [Sandbox lab](http://cs.co/CoR-Trial) first. The Sandbox offers a possibility to learn the basics and configuration workflow by following step-by-step\nconfiguration guide. You do not need any cloud accounts or prerequisites, the whole setup will be brought up for you automatically and you have full access for 2 hours free of charge.\n\nIf you want to create your own setup and test specific use cases, this project helps you and automates creation of two virtual branches and two cloud-based apps.\nYou can use this infrastructure to test site-to-site and site-to-cloud uses cases, which Cisco Cloud onRamp offers. The following diagram shows topology, which will be created.\n![Topology](img1-topology.png)\n\nAs cloud- and branch-based apps we use a simple web server running on different ports on Linux VM. By creating an SD-WAN policy, you can route traffic via public internet or CSP backbone. \nAn example for such traffic steering policy will be given in the SD-WAN part based on TCP ports 8001-8004.\n\nBoth virtual branches will be implemented with Cisco CSR 1000v virtual routers. Branch 1 has also a WAN emulator, which can introduce configurable loss, latency and delay on public internet path.\nThis is implemented using Linux Traffic Control (tc) capabilities.\n\nAmazon Web Services (AWS) is used as Cloud Service Provider (CSP) to host both virtual branches. This can easily be changed to Azure or GCP by changing Terraform Providers and adjusting the code.\nCSP choice for branch hosting is not relevant for Cloud onRamp functionality and tests. \n\n____\n\nCentralized Firewall Inspection with SD-WAN is a very common design topic. It is implemented in the chapter 05. This section can be used standalone if you are interested only in this topic.\nThe implemented solution allows scalable north-south, east-west traffic inspection using Cisco FTDv virtual firewalls and AWS Gateway Load Balancer (GWLB) and shown below.\n![Topology](img3-fw-and-sdwan.png)\n\n____\n\nCisco SD-WAN Multi-Region Fabric (MRF) provides the ability to segment the SD-WAN overlay into multiple regional networks that operate distinctly from one another, and a central core-region network for managing inter-regional traffic.\nMany customers have at least two Cloud Service Providers (CSP) and would like to use CSP backbone for site-to-site communication between different regions.\nIn a flat (non MRF) design without any segmentation a control policy for traffic steering between different sites can be complex and not easy to create. It does not scale well with the number of sites. MRFsolves this problem by splitting the network into different regions.\nSecond problem beside complexity is the ask to isolate specific subregions within one particular region. For example, in emergency case all branches within Northern California must be isolated from the rest of the network with one click of a button.\n![Topology](img4-mrf-multicloud.png)\n\n***\n\n**Table of Content**\n- [TL;DR Summary](#tl-dr-summary)\n- [Getting Started](#getting-started)\n  * [Prerequisites](#prerequisites)\n  * [Software versions used during creation of this lab](#software-versions-used-during-creation-of-this-lab)\n  * [SD-WAN Controllers](#sd-wan-controllers)\n  * [Creating virtual branch infrastructure](#creating-virtual-branch-infrastructure)\n  * [Configuring WAN Emulator](#configuring-wan-emulator)\n  * [Post-deployment fine tuning](#post-deployment-fine-tuning)\n  * [Creating Branch2](#creating-branch2)\n  * [Creating cloud-based Apps](#creating-cloud-based-apps)\n  * [Shared Services VPC with FTDv Firewall](#shared-services-vpc-with-ftdv-firewall)\n  * [Multi-Region Fabric and Multicloud](#multi-region-fabric-and-multicloud)  \n- [SD-WAN Cloud onRamp Configuration](#sd-wan-cloud-onramp-configuration)\n- [Authors](#authors)\n\n***\n\n## Getting Started\n\n### Prerequisites\n\nBefore you begin, please make sure, that the following mandatory criteria are met:\n- [x] a computer with installed Terraform environment including CSP provider and configuration.\n- [x] you have an account and can use at least one of the following CSP: AWS, Azure or GCP\n- [x] you have an existing SD-WAN network with vManage, vBond and vSmart (SD-WAN Controllers) running the following software versions:\n  \u003e| CSP   | Controller |\n  \u003e| :----: | :----:     |\n  \u003e| AWS   |  20.9      |\n  \u003e| Azure |  20.9      |  \n  \u003e| GCP   |  20.9      |  \n- [x] 4 SD-WAN licenses for Cisco Catalyst 8000v (c8kv) virtual routers acting as cloud gateways. In each CSP region, Cloud onRamp Automation will spin up two c8kv instances.\n- [x] 2 SD-WAN licenses for Cisco CSR1000v virtual routers acting as branch routers.\n\n\nFor example if you want to test Cloud onRamp with Azure vWAN, you will need at least 20.4 software running on all SD-WAN controllers.\nYou can still have virtual branches hosted on a different CSP, but if you want to test Cloud onRamp automation workflow for Azure, you need at least 20.4 controller software.\\\nIf you do not have SD-WAN setup at all, you will need to create one. Please work with your local Cisco contacts and ask for Proof of Concept (PoC) setup.\n\nPlease make sure, that your CSP account has enough ressources to spin up needed infrastructure. In AWS case, please double check the following:\n- you can create enough basic elements like VPCs, Subnets, IGWs, TGW, etc.\n- you have enough elastic IPs: calculate with two Elastic IPs per VM (one for out-of-band management and one for internet access) to be conservative\n- you have ssh key registered in appropriate regions\n\n\n### Software versions used during creation of this lab\n\n- Terraform v1.3.7, AWS Provider v4.49.0, Google Provider v4.47.0\n- SD-WAN Controllers: 20.10.0\n- CSR1000v on virtual branches: 17.03.02prd9\n- C8kv as cloud gateways: 17.09.01, 17.10 for MRF with Subregions Use Case\n- Linux for apps and WAN-Emulator: Amazon Linux 2 AMI, kernel version 4.14.219-164.354.amzn2.x86_64\n\n\n### SD-WAN Controllers\nBefore you can start any tests with Cisco Cloud onRamp for Multicloud, you will need to have running SD-WAN controllers: vManage, vBond and vSmart.\nCisco supports two different operations options:\n- on-prem: controllers running in a Data Center for example on ESXi\n- cloud-based: Cisco or MSP cloudops team will create a set of controllers for you.\n\nThis lab repo provides Terraform scripts, which will spin up a basic controller environment \nonly if you will have private controllers AMIs shared with you by Cisco.\nPlease note, that SD-WAN controller images are currently not available on marketplace.\nPlease work with your Cisco Account team if your project requires private AMIs. \nThis can be done for testing only special cases and currently cannot be used for production.\nThe main reason is performance and stability considerations - Cisco performs intensive tests with ESXi and for Cloudops-operated controllers,\ndefines strict requirements for Hypervisors and underlying hardware. \n\nThe included scripts will create new basic controller installation. \nOnce vManage, vSmart and vBond are up and running, you will need to provide basic configuration: org-id, system IPs, etc.\nAlso if own PKI infrastructure is used, you will need to install own certificates.\nPlease note, that the current version of scripts is not using cloud-init files like others sections. \nPlease refer to the branch scripts if you need an example for cloud-init Day 0 configuration, which is used during the first VM boot.\n\n\n### Creating virtual branch infrastructure\n\nThis section describes creation of a virtual branch consisting of one SD-WAN virtual router (CSR1000v) and Linux App.\nBranch 1 also has virtual WAN emulator, which can introduce configurable loss and latency to the public internet connection.\n\n**Step 1**: ssh pem file\u003cbr\u003e \nMake sure, that your have ssh pem file in the same folder with terraform files for the virtual branch.\nIn the example below you see \"aws-key-20-3-setup.pem\" file, which will be used for ssh connectivity to the virtual routers.\nPlease note, that you need to copy your pem file to the Terraform folder or provide a path to it. Otherwise, you will be not able to create VMs.\n\n\n```\naws-key-20-3-setup.pem           \u003c-- pem ssh key for the appropriate AWS region\nbranch1.tf                       \u003c-- Main Terraform script\ncloud-init-branch1-r1.user_data  \u003c-- Day-0 SD-WAN CSR1000v configuration generated by vManage\nprovider.tf                      \u003c-- Provider file with AWS region definition\nvars.tf                          \u003c-- File with all variables, which are used in the main Terraform script\n```\n\n**Step 2**: Day-0 config for SD-WAN CSR1000v\u003cbr\u003e\nIn Cisco vManage under configuration - devices, select free CSR100v and generate cloud-init file.\nPlease refer to [the following Cisco Online Documentation](https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/hardware-and-software-installation.html#iosxe-generic-bootstrap) for details about cloud-init file.\n\nIn the example for cloud-init file shown below we use the following NIC allocation:\n- 1st NIC (GigabitEthernet1) for out-of-band management aka VPN512 or Mgmt-intf vrf\n- 2nd NIC (GigabitEthernet2) for SD-WAN over public-internet aka VPN0\n- 3rd NIC for \"LAN\" connectivity to the Branch App (Linux VM) aka service vpn or vrv 10.\n\nYou can create the router configuration using vManage Feature and Device Templates, CLI template or simply generate \nthe cloud-init file without vManage-generated config and add the lines shown below to the cloud-init file.\nIn this case the virtual router will come up, join the SD-WAN fabric and remain in CLI mode.\nPlease refer to [this Cisco Online Documentation](https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/system-interface/vedge-20-x/systems-interfaces-book/configure-devices.html)\nfor more details on Device Templates.\n\n```\nContent-Type: multipart/mixed; boundary=\"==BOUNDARY==\"\nMIME-Version: 1.0\n\n--==BOUNDARY==\nContent-Type: text/cloud-config; charset=\"us-ascii\"\n\n#cloud-config\nvinitparam:\n - uuid : CSR-BBD516D8-XXXX-XXXX-XXXX-4FA9CB62259D\n - org : Demo-npitaev\n - vbond : 44.XXX.XXX.68\n - otp : xxxx50b93df741dbb3719489e98axxxx\n\n--==BOUNDARY==\nContent-Type: text/cloud-boothook; charset=\"us-ascii\"\n\n#cloud-boothook\n\nhostname Branch1-R1\n!\nsystem\n system-ip             10.111.1.11\n site-id               111\n organization-name     Demo-npitaev\n vbond 44.XXX.XXX.68\n!\n!\nvrf definition 10\n rd 1:10\n address-family ipv4\n  route-target export 64550:1\n  route-target import 64550:1\n  exit-address-family\n !\n address-family ipv6\n  exit-address-family\n !\n!\nvrf definition Mgmt-intf\n description Management\n rd          1:512\n address-family ipv4\n  route-target export 1:512\n  route-target import 1:512\n  exit-address-family\n !\n address-family ipv6\n  exit-address-family\n !\n!\ninterface GigabitEthernet1\n no shutdown\n vrf forwarding Mgmt-intf\n ip address dhcp client-id GigabitEthernet1\n ip dhcp client default-router distance 1\n ip mtu    1500\n mtu           1500\n negotiation auto\nexit\n!\ninterface GigabitEthernet2\n no shut\n ip address dhcp client-id GigabitEthernet2\n ip dhcp client default-router distance 1\n ip mtu    1500\n mtu           1500\n negotiation auto\n!\n!\ninterface GigabitEthernet3\n no shut\n!\n!\ninterface Tunnel2\n no shutdown\n ip unnumbered GigabitEthernet2\n no ip redirects\n ipv6 unnumbered GigabitEthernet2\n no ipv6 redirects\n tunnel source GigabitEthernet2\n tunnel mode sdwan\nexit\n!\n!\nsdwan\n interface GigabitEthernet2\n  tunnel-interface\n   encapsulation ipsec weight 1\n   no border\n   color default\n   no last-resort-circuit\n   no low-bandwidth-link\n   no vbond-as-stun-server\n   vmanage-connection-preference 5\n   port-hop\n   carrier                       default\n   nat-refresh-interval          5\n   hello-interval                1000\n   hello-tolerance               12\n   allow-service all\n   no allow-service bgp\n   allow-service dhcp\n   allow-service dns\n   allow-service icmp\n   allow-service sshd\n   allow-service netconf\n   allow-service ntp\n   no allow-service ospf\n   no allow-service stun\n   allow-service https\n   no allow-service snmp\n  exit\n exit\n appqoe\n  no tcpopt enable\n !\n omp      \n  no shutdown\n  send-path-limit  4\n  ecmp-limit       4\n  graceful-restart\n  no as-dot-notation\n  timers\n   holdtime               60\n   advertisement-interval 1\n   graceful-restart-timer 43200\n   eor-timer              300\n  exit\n  address-family ipv4\n   advertise bgp\n   advertise connected\n   advertise static\n  !\n  address-family ipv6\n   advertise bgp\n   advertise connected\n   advertise static\n  !\n !\n!\n!\n--==BOUNDARY==\n```\n\n\n**Step 3**: configure Terraform variables in the vars.tf file\u003cbr\u003e\nPlease review and change **ALL** variables defined in vars.tf.\nThe default AWS region fro Branch1 is us-west-2. If you want to change it, please make sure to change the AMIs for CSR1000v and Linux.\nYou can follow [this HowTo](https://leaherb.com/how-to-find-an-aws-marketplace-ami-image-id/) if you need to find out the appropriate AMI ID from AWS Marketplace for CSR1000v and AMI Linux. \n\nAlso make sure to use the right ssh pem file, which is available in the appropriate region.\n\n\n**Step 4**: test your Terraform files\u003cbr\u003e\nOnce you configured all variables in the vars.tf file and double checked the region in the provider.tf file, please initiate and test all before deployment.\n```\n~/terraform-cisco-sdwan-cor-lab/01-Branch1# terraform init\n\nInitializing the backend...\n\nInitializing provider plugins...\n- Finding latest version of hashicorp/aws...\n- Installing hashicorp/aws v3.34.0...\n- Installed hashicorp/aws v3.34.0 (signed by HashiCorp)\n\nTerraform has created a lock file .terraform.lock.hcl to record the provider\nselections it made above. Include this file in your version control repository\nso that Terraform can guarantee to make the same selections by default when\nyou run \"terraform init\" in the future.\n\nTerraform has been successfully initialized!\n\nYou may now begin working with Terraform. Try running \"terraform plan\" to see\nany changes that are required for your infrastructure. All Terraform commands\nshould now work.\n\nIf you ever set or change modules or backend configuration for Terraform,\nrerun this command to reinitialize your working directory. If you forget, other\ncommands will detect it and remind you to do so if necessary.\n~/terraform-cisco-sdwan-cor-lab/01-Branch1# \n~/terraform-cisco-sdwan-cor-lab/01-Branch1# \n~/terraform-cisco-sdwan-cor-lab/01-Branch1# \n~/terraform-cisco-sdwan-cor-lab/01-Branch1# \n~/terraform-cisco-sdwan-cor-lab/01-Branch1# terraform plan\nprovider.aws.region\n  The region where AWS operations will take place. Examples\n  are us-east-1, us-west-2, etc.\n\n  Enter a value: us-west-2      \u003c-- enter your region\n  \n  ... output omitted for brevity ...\n  \nPlan: 32 to add, 0 to change, 0 to destroy.   \u003c-- check the output, scroll up and validate ressources to be created\n\n~/terraform-cisco-sdwan-cor-lab/01-Branch1#  \n```\n\n**Step 5**: deploy your Terraform files\u003cbr\u003e\nOnce you verified the output from \"terraform plan\", you can now deploy the virtual branch by executing \"terraform apply\"\n```\n~/Downloads/terraform-cisco-sdwan-cor-lab/01-Branch1# terraform apply\nprovider.aws.region\n  The region where AWS operations will take place. Examples\n  are us-east-1, us-west-2, etc.\n\n  Enter a value: us-west-2   \u003c-- enter your region\n  \n    ... output omitted for brevity ...\n\nPlan: 32 to add, 0 to change, 0 to destroy.\n\nDo you want to perform these actions?\n  Terraform will perform the actions described above.\n  Only 'yes' will be accepted to approve.\n\n  Enter a value: yes   \u003c-- confirm by typing yes\n  \n    ... output omitted for brevity ...\n```\n\nAfter few minutes your virtual branch will be ready. In case of any problems, for example if you will run into some AWS limits like not enough elastic IPs, \nyou can increase the AWS limits and repeat \"terraform apply\". Terraform will add needed ressources and not touch already existing items, which do not need modifications.\n\n\n\n### Configuring WAN Emulator\n\nThe WAN Emulator Linux VM will be able to imair public internet link and introduce loss and latency using Network Address Translation (NAT) and Linux Traffic Control (TC) capabilities.\nThis section describes NAT, Firewall and TC cionfiguration.\n\nWAN Emulator has 3 NICs:\n1. eth0 for out-of-band management\n2. eth1 pointing to CSR1000v\n3. eth2 pointing to public internet.\n\nPreparation:\n- make sure, that the WAN Emulator VM was successfully deployed and can be accessed via ssh.\n- AWS source and destination check should be disabled on all NICs. Please double check it in AWS EC2 console.\n- Check the AWS route tables and make sure, that the routing is correct.\n\nConfiguring Firewall and NAT:\n1. Install firewall: *sudo yum install firewalld*\n2. Start firewall: *sudo systemctl start firewalld*\n3. Check and enable IP forwarding:\\\n    *sudo sysctl net.ipv4.ip_forward*\\\n    *sudo sysctl -w net.ipv4.ip_forward=1*\\\n    *sudo sysctl net.ipv4.ip_forward*\n4. Configure NAT and define internal and external zones:\\\n    *sudo firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o eth2 -j MASQUERADE -s 10.201.2.0/24*\\\n    *sudo firewall-cmd --zone=external --add-masquerade --permanent*\\\n    *sudo firewall-cmd --zone=external --add-interface=eth2 --permanent*\\\n    *sudo firewall-cmd --zone=internal --add-interface=eth1 --permanent*\\\n    *sudo firewall-cmd --complete-reload*\\\n    *sudo firewall-cmd --list-all-zones*\n5. Reboot: sudo reboot\n\nPlease check additional documentation on NAT and Firewall described in [this article](https://linuxtechlab.com/turning-centosrhel-6-7-machine-router/).\n\nCheck FW status:\n```\n[ec2-user@linux-tc-for-gcp ~]$ sudo systemctl status firewalld\n firewalld.service - firewalld - dynamic firewall daemon\n   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)\n   Active: active (running) since Thu 2020-03-19 17:58:32 UTC; 7min ago\n     Docs: man:firewalld(1)\n Main PID: 2020 (firewalld)\n   CGroup: /system.slice/firewalld.service\n            2020 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid\nMar 19 17:58:32 linux-tc-for-gcp systemd[1]: Starting firewalld - dynamic firewall daemon...\nMar 19 17:58:32 linux-tc-for-gcp systemd[1]: Started firewalld - dynamic firewall daemon.\n[ec2-user@linux-tc-for-gcp ~]$ \n```\n\nCheck IP forwarding:\n```\n[ec2-user@ip-10-13-1-10 ~]$ sudo sysctl net.ipv4.ip_forward\nnet.ipv4.ip_forward = 1\n[ec2-user@ip-10-13-1-10 ~]$\n```\n\nYou may want to adjust route table on WAN Emulator and make sure, that is routes packets via needed NIC.\nCheck the route table with *ip ro* and change it as needed. The following example installs a route for the out-of-band management\nvia 1st NIC (eth0( and then leaves only one default route via NIC3 (eth2).\n![WANEM](img2-wanem-branch1-topology.png) \n\n```\nsudo route add -net 128.107.0.0/16 gw 10.111.1.1                           \u003c-- Allowed CIDR for the out of band management interface\nsudo route del default gw 10.111.1.1\nsudo route del default gw 10.111.2.1\nsudo route del default gw 10.111.4.1\nsudo route add default gw 10.111.4.1                                       \u003c-- Public internet access only via eth2\n```\n\nThe following example shows the full interface configuration and route table for the working WAN Emulator in Branch1\n```\n[ec2-user@ip-10-111-1-10 ~]$ ifconfig\neth0: flags=4163\u003cUP,BROADCAST,RUNNING,MULTICAST\u003e  mtu 9001                  \u003c-- Out of band management interface\n        inet 10.111.1.10  netmask 255.255.255.0  broadcast 10.111.1.255\n        ...\neth1: flags=4163\u003cUP,BROADCAST,RUNNING,MULTICAST\u003e  mtu 9001\n        inet 10.111.2.10  netmask 255.255.255.0  broadcast 10.111.2.255     \u003c-- NIC pointing to CSR1000v\n        ...\neth2: flags=4163\u003cUP,BROADCAST,RUNNING,MULTICAST\u003e  mtu 9001                  \u003c-- NIC pointing to public internet via IGW\n        inet 10.111.4.10  netmask 255.255.255.0  broadcast 10.111.4.255\n        ...\n[ec2-user@ip-10-111-1-10 ~]$ \n[ec2-user@ip-10-111-1-10 ~]$ ip ro\ndefault via 10.111.4.1 dev eth2 \n10.111.1.0/24 dev eth0 proto kernel scope link src 10.111.1.10 \n10.111.2.0/24 dev eth1 proto kernel scope link src 10.111.2.10 \n10.111.4.0/24 dev eth2 proto kernel scope link src 10.111.4.10 \n128.107.0.0/16 via 10.111.1.1 dev eth0 \n169.254.169.254 dev eth0 \n[ec2-user@ip-10-111-1-10 ~]$ \n```\n\nOnce you have verified NAT and reachability, you can configure delay and/or loss using Linux Traffic Control capabilities.\n[This article](https://www.cyberciti.biz/faq/linux-traffic-shaping-using-tc-to-control-http-traffic/) describes basic functionality and provides additional details.\nYou can install tc using the following commands *sudo yum install tc* and *sudo yum install iproute-tc*.\nThen you can enable delay of 200 ms like shown in this example:\n```\n[ec2-user@linux-tc-for-gcp ~]$ sudo tc qdisc add dev eth1 root netem delay 200ms\n[ec2-user@linux-tc-for-gcp ~]$\n```\nCheck tc and delays on the appropriate interface: \n```\n[ec2-user@linux-tc-for-gcp ~]$ sudo tc -s qdisc ls dev eth1\nqdisc netem 8001: root refcnt 3 limit 1000 delay 200.0ms\n Sent 1240 bytes 18 pkt (dropped 0, overlimits 0 requeues 0) \n backlog 0b 0p requeues 0\n[ec2-user@linux-tc-for-gcp ~]$ \n```\n\nYou can also match on a specific IP and introduce delays only for it as described in [this article](https://serverfault.com/questions/389290/using-tc-to-delay-packets-to-only-a-single-ip-address).\nThe following example introduces latency of 400ms for all packets sent to 54.112.112.125 (public IP of the branch2):\n```\nsudo tc qdisc add dev eth2 root handle 1: prio\nsudo tc filter add dev eth2 protocol all parent 1: prio 1 u32 match ip dst 54.253.240.125/32 flowid 1:1\nsudo tc filter add dev eth2 protocol all parent 1: prio 2 u32 match ip dst 0.0.0.0/0 flowid 1:2\nsudo tc qdisc add dev eth2 parent 1:1 handle 10: netem delay 400ms\nsudo tc qdisc add dev eth2 parent 1:2 handle 20: sfq\n```\n\n\n\n### Post-deployment fine tuning \n\nIn order to enable the direct communication between two branches over public internet, you need to find out public IP addresses and adjust security groups, which were\ncreated during the initial deployment. You can locate the public IP address in AWS EC2 Management Console and then adjust the following security group lines in the branch1.tf\nand branch2.tf files. The example below shows just one side.\n\n```\n    //Allow all from Branch1 Public IP WANEM - IP address must be changed after initial deployment\n    ingress {\n        from_port = 0\n        to_port = 0\n        protocol = \"-1\"  \n        cidr_blocks = [\"54.111.111.122/32\"]      \u003c-- change to your public IP\n    } \n\n    //Allow all from Branch1 Public IP Linux Host with TE Mgmt - IP address must be changed after initial deployment\n    ingress {\n        from_port = 0\n        to_port = 0\n        protocol = \"-1\"  \n        cidr_blocks = [\"52.111.111.14/32\"]       \u003c-- change to your public IP\n    } \n        \n```\n\n\n\n### Creating Branch2\n\nThe folder *02-Branch2* will be used for Branch2 creation. It has the same structure as Branch1 folder.\nFollow the same steps as for Branch1. Branch2 does not need WAN Emulator, which simplifies the process.\nPlease make sure, that you are using the right pem ssh file for this right region and will not run into cloud limits\ndescribed above in the branch1 section.\n\n\n\n### Creating cloud-based Apps\n\nTerraform scripts in the folders *03-CSP-Region1-Cloud-App* and *04-CSP-Region2-Cloud-App* create cloud apps in a configured AWS region.\nA simple nginx webserver running on port 8003 and 8004 will be used to simulate critical traffic.\nThe following post-install CLIs will install and configure the web server.\n```\nsudo amazon-linux-extras install nginx1\nsudo nano /etc/nginx/nginx.conf       \u003c-- adjust TCP port to 8003 / 8004\nsudo service nginx start\nsudo chkconfig nginx on\nss -tlpn | grep :8003                 \u003c-- verify that the web server is running on the right port, in this case 8003\n```\n\nYou can install any traffic generation / monitoring tool. We use [ThousandEyes](https://www.thousandeyes.com/) agent on Branch1 for monitoring and page load tests.\nAssuming you have access to ThousandEyes Dashboard, you can copy the token and install the agent with the following 3 lines.\n```\ncurl -Os https://downloads.thousandeyes.com/agent/install_thousandeyes.sh\nchmod +x install_thousandeyes.sh\nsudo ./install_thousandeyes.sh -b XXX-Token-XXX\n```\n\n____\n\n### Shared Services VPC with FTDv Firewall\nThe first two scripts will create two cloud Apps with Web Server running in two different Availability Zones (AZ).\nThe third script will create Shared Services VPC with two Cisco FTDv virtual firewall in two different AZs and AWS GWLB load-balancing traffic across two AZs.\nThe last script creates SD-WAN VPC with two Catalyst 8000v virtual SD-WAN routers running in two different AZs.\n\nPlease note, that Terraform currently (Nov. 2021) does NOT support TGW Connect (GRE) attachments - see details [here](https://github.com/hashicorp/terraform-provider-aws/pull/20780)\nYou can connect SD-WAN VPC manually as Connect Attachment to TGW, use [VPN Attachment](https://github.com/terraform-aws-modules/terraform-aws-vpn-gateway/tree/v2.11.0/examples/complete-vpn-connection-transit-gateway) instead or use other tools.\n\nSimplified Packet From Host VPC to SD-WAN: Host VPC -\u003e AWS TGW -\u003e GWLB -\u003e FTDv -\u003e TGW -\u003e SD-WAN\nReturning traffic: SD-WAN -\u003e AWS TGW -\u003e GWLB -\u003e FTDv -\u003e TGW -\u003e Host VPC\nPlease see detailed steps for the packet flow in [this AWS blog](https://aws.amazon.com/blogs/networking-and-content-delivery/centralized-inspection-architecture-with-aws-gateway-load-balancer-and-aws-transit-gateway/).\n\nGENEVE protocol is used for load balancing between GWLB and FTDv. FTDv software version 7.1 or later supports GENEVE protocol.\n\nAppliance mode is required for symmetric routing, it will be enabled for the Shared Services VPC attachment to AWS TGW.\n\n____\n\n### Multi-Region Fabric and Multicloud\nThe MRF Multicloud solution described below has the fiollowing benefits:\n   * Simplifies network design by introducing regions like US West or Europe.\n   * Improves scalability\n   * Improves reliability by using two different CSPs\n   * Simplifies control policies for traffic steering\n   * Subregions provide another granularity level for control and isolation\n\nSummary of the deployment steps \n1. Use Terraform scripts to deploy the topology shown above.\n2. Make sure, that your Controllers and SD-WAN routers are running 20.10 / 17.10 Software\n3. Configure MRF - refer to [this CCO Config Guide](https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/hierarchical-sdwan/hierarchical-sdwan-guide/h-sd-wan-basics.html). In a nutshell, you will need to configure the following:\n  * Correct Region and Subregion on the Edge routers\n  * Regional vSmart serving both regions\n  * Enable MRF on vManage and create regions 1 and 2 in the Network Hierarchy Manager.\n\nUse Case 1: Redundancy / Load Balancing\nThe following requirements will be met:\n* Subregion 1 (SJ) uses by default San Francisco Border Router\n* Subregion 2 (San Diego) uses Los Angeles Border Router\n* In case of a single backbone failure -\u003e auto failover\n\nBorder Router for San Francisco will be configured with Region 1 and Subregion 2 as shown below:\n```\nsystem\n system-ip             101.1.1.1\n site-id               101\n region 1\n  subregion 1\n !\n role                  border-router\n organization-name     mrf-multicloud-demo\n vbond 44.227.177.103\n!\n```\nBorder Router in Los Angeles will be placed in the Subregion 2:\n```\nsystem\n system-ip             103.1.1.1\n site-id               103\n region 1\n  subregion 2\n !\n role                  border-router\n organization-name     mrf-multicloud-demo\n vbond 44.227.177.103\n!\n```\n\nIn normal case an edge router in San Jose (Subregion 1) will have the following entry for an IP address of the Region 2:\n```\nReg1-Sub1-ER1#sh ip ro vrf 10\n...\nm        10.211.1.11 [251/0] via 101.1.1.1, 06:58:01, Sdwan-system-intf\n...\nReg1-Sub1-ER1#\n```\n\nIn case of San Francisco Border Router outage, the route will be replaced with the next hop of the Los Angeles BR:\n```\nReg1-Sub1-ER1#sh ip ro vrf 10\n...\nm        10.211.1.11 [251/0] via 103.1.1.1, 06:58:01, Sdwan-system-intf\n...\nReg1-Sub1-ER1#\n```\nUse Case 2: Isolate a subregion with a simple control policy\nA centralized control policy will be used for on-demand isolation of one Subregion from the rest of the network:\n```\npolicy\n control-policy block-reg1-sub1\n  sequence 1\n   match route\n    region-enhanced region 1\n    region-enhanced subregion 1\n   !\n   action reject\n   !\n  !\n  sequence 2\n   match tloc\n    region-enhanced region 1\n    region-enhanced subregion 1\n   !\n   action reject\n   !\n  !\n  default-action accept\n !\n!\napply-policy\n region 1\n  role border-router\n  control-policy block-reg1-sub1 out\n !\n!\n```\n\n____\n\n## SD-WAN Cloud onRamp Configuration\n\nOnce you have successfully brought up two virtual branches and cloud-based app infrastructure from the previous section, you can proceed with Cisco SD-WAN Cloud onRamp configuration.\nAll configuration steps for site-to-site and site-to-cloud use cases will be done in Cisco vManage.\nPlease refer to [this Cisco Online Documentation article](https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/cloudonramp/ios-xe-17/cloud-onramp-book-xe/cloud-onramp-multi-cloud.html) for details.\n\nIn a nutshell, the workflow has the following steps:\n- setup cloud account\n- discover cloud infrastructure\n- create cloud gateways\n- connect cloud infrastructure with SD-WAN networks.\n\nAfter successful creation of two cloud gateways in appropriate regions, you should create an SD-WAN control policy, \nwhich will route critical traffic via CSP backbone and send best effort traffic via public internet.\nIn our example we use ports 8001-8004 to simulate critical traffic. Such policy can be created in vManage using graphical user interface.\nThe following control policy is a CLI preview/export from vManage. It matches traffic sent to TCP port 8001 and sets next hop to the closest CSP entry location. \nIt has 4 different blocks because it follows the traffic flow: Branch1 -\u003e CSP Region 1 -\u003e CSP Region 2 -\u003e Branch2. And back.\n```\nviptela-policy:policy\n data-policy _vpn10-list_USwest-Branch\n  vpn-list vpn10-list\n    sequence 1\n     match\n      source-ip 0.0.0.0/0\n      destination-port 8001               \u003c-- matching the critical traffic\n     !\n     action accept\n      set\n       vpn 10\n       tloc  41.41.41.1 color public-internet encap ipsec   \u003c-- setting next hop\n      !\n     !\n    !\n  default-action accept\n !\n data-policy _vpn10-list_Megaport-GW-Sydney\n  vpn-list vpn10-list\n    sequence 1\n     match\n      source-ip 0.0.0.0/0\n      source-port 8001\n     !\n     action accept\n      set\n       vpn 10\n       tloc  41.41.41.1 color private2 encap ipsec\n      !\n     !\n    !\n  default-action accept\n !\n data-policy _vpn10-list_Sydney-Branch\n  vpn-list vpn10-list\n    sequence 1\n     match\n      source-ip 0.0.0.0/0\n      source-port 8001\n     !\n     action accept\n      set\n       vpn 10\n       tloc  42.42.42.1 color public-internet encap ipsec\n      !\n     !\n    !\n  default-action accept\n !\n data-policy _vpn10-list_Megaport-GW-USwest\n  vpn-list vpn10-list\n    sequence 1\n     match\n      source-ip 0.0.0.0/0\n      destination-port 8001\n     !\n     action accept\n      set\n       vpn 10\n       tloc  42.42.42.1 color private2 encap ipsec\n      !\n     !\n    !\n  default-action accept\n !\n lists\n  site-list megaport-sydney-list\n   site-id 42 \n  !\n  site-list megaport-uswest-list\n   site-id 41 \n  !\n  site-list sydney-branch-list\n   site-id 112 \n  !\n  site-list uswest-branch-list\n   site-id 111 \n  !\n  vpn-list vpn10-list\n   vpn 10 \n  !\n !\n!\napply-policy\n site-list megaport-sydney-list\n  data-policy _vpn10-list_Megaport-GW-Sydney from-tunnel\n !\n site-list uswest-branch-list\n  data-policy _vpn10-list_USwest-Branch from-service\n !\n site-list sydney-branch-list\n  data-policy _vpn10-list_Sydney-Branch from-service\n !\n site-list megaport-uswest-list\n  data-policy _vpn10-list_Megaport-GW-USwest from-tunnel\n !\n!\n```\n\n\n## Authors\n\n* **Nikolai Pitaev** - *Initial work* - [Cisco](https://www.linkedin.com/in/npitaev/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fciscodevnet%2Fsdwan-cor-labinfra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fciscodevnet%2Fsdwan-cor-labinfra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fciscodevnet%2Fsdwan-cor-labinfra/lists"}