{"id":155,"date":"2019-01-30T13:52:29","date_gmt":"2019-01-30T13:52:29","guid":{"rendered":"http:\/\/www.tangosierratech.com\/blog\/wordpress\/?p=155"},"modified":"2019-01-30T13:52:29","modified_gmt":"2019-01-30T13:52:29","slug":"vagrant-ansible-cumulus-and-evpn-orchestrating-a-virtualized-network-on-your-laptop","status":"publish","type":"post","link":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/2019\/01\/30\/vagrant-ansible-cumulus-and-evpn-orchestrating-a-virtualized-network-on-your-laptop\/","title":{"rendered":"Vagrant, Ansible, Cumulus, and EVPN….Orchestrating a Virtualized Network on your Laptop."},"content":{"rendered":"

In this post we will talk about how to use Vagrant, Ansible, and Cumulus Linux to build a virtual layer 2 extension across an IP fabric. We will use the standard protocol MP-BGP and its EVPN extension to run the control plane, and VXLAN to provide the virtual data plane.<\/p>\n

Each technology deserves a brief introduction prior making things happen.<\/p>\n

Vagrant up!<\/h6>\n

Vagrant is an orchestration tool that uses “providers” to deploy virtual machines to a variety of platforms. Vagrant is probably most famous for its integration with Virtualbox. It’s quite common for cloud and application developers to use Vagrant to build their application stacks locally. Vagrant then makes it easier to then deploy them to production in cloud environments, in an automated fashion. We will use Vagrant to deploy the switches and servers in our topology.<\/p>\n

Ansible.<\/h6>\n

Ansible is another orchestration tool that helps you manage your infrastructure as code. It is agent-less and has probably the best tool box for network automation out of the various orchestration tools. In this post we will use ansible to provision, or configure, our devices.<\/p>\n

Cumulus Linux.<\/h6>\n

Cumulus Networks is a virtual networking company that has made available a machine image running their networking software. This is a linux image that will be run as the switches in our topology. In the real world, you can run this software on a physical switch and operate the topology in the same way, just replace Vagrant with your OPs guy installing and cabling them. They publish the CumulusVX image, along with several automated topologies, so that you can do exactly what we are doing in this post, automate your network with Cumulus.<\/p>\n

Last but not least….EVPN.<\/h6>\n

EVPN stands for Ethernet Virtual Private Network. Its another address-family under the MP-BGP standard. The technology is used as the control plane for transporting and updating the MAC address table to your switches across an IP fabric.<\/p>\n

Another technology that should be mentioned, that is not in the title, is VXLAN. We will use VXLAN in the dataplane for encapsulating and transporting our ethernet frames.<\/p>\n

Ok, here is the topology we are going to build:<\/p>\n

 <\/p>\n

\"\"<\/a><\/p>\n

On the bottom are our servers. Each server is connected to a single leaf switch in a mode 2<\/a> port channel. Many of the labs that Cumulus publishes dual attach the servers to two leaf switches and bonds them using their m-lag implementation<\/a>. This requires lacp, which unfortunately, I was not able to get working locally. To keep the project moving forward, I implemented a work around which was to change the physical connectivity, and configure all of the bonds as mode 2 (balance-xor)<\/a>.<\/p>\n

Cumulus provides some great tools for building these topologies, one of them is their topology converter<\/a>.<\/p>\n

Using their python script topology_converter.py, I took the following “.dot” file and converted it into a Vagrantfile. Vagrant will use this to build, and most importantly connect, all of the instances.<\/p>\n

graph dc1 {\r\n \"spine1\" [function=\"spine\" config=\".\/helper_scripts\/extra_switch_config.sh\"]\r\n \"spine2\" [function=\"spine\" config=\".\/helper_scripts\/extra_switch_config.sh\"]\r\n \"leaf1\" [function=\"leaf\" config=\".\/helper_scripts\/extra_switch_config.sh\"]\r\n \"leaf2\" [function=\"leaf\" config=\".\/helper_scripts\/extra_switch_config.sh\"]\r\n \"server1\" [function=\"host\" config=\".\/helper_scripts\/extra_server_config.sh\"]\r\n \"server2\" [function=\"host\" config=\".\/helper_scripts\/extra_server_config.sh\"]\r\n   \"spine1\":\"swp3\" -- \"leaf1\":\"swp3\"\r\n   \"spine1\":\"swp4\" -- \"leaf2\":\"swp4\"\r\n   \"spine2\":\"swp5\" -- \"leaf1\":\"swp5\"\r\n   \"spine2\":\"swp6\" -- \"leaf2\":\"swp6\"\r\n   \"leaf1\":\"swp40\" -- \"leaf2\":\"swp40\"\r\n   \"leaf1\":\"swp50\" -- \"leaf2\":\"swp50\"\r\n   \"server1\":\"eth1\" -- \"leaf1\":\"swp1\"\r\n   \"server1\":\"eth2\" -- \"leaf1\":\"swp2\"\r\n   \"server2\":\"eth1\" -- \"leaf2\":\"swp1\"\r\n   \"server2\":\"eth2\" -- \"leaf2\":\"swp2\"\r\n}<\/pre>\n
The Vagrant file that it creates is fairly long and took a number of modifications for what I wanted. Luckily, I’m going to share this with you… so clone this repo<\/a> and take a look at it yourself. The repo will be needed to complete the build anyway.<\/p>\n
The modifications that I made to the file involve commenting out some of the helper scripts, and using Vagrant’s ansible integration to run the playbooks.<\/p>\n
~$ git clone https:\/\/github.com\/tsimson\/cumulus_evpn.git\r\n~$\r\n~$ cd cumulus_evpn\/\r\n~\/cumulus_evpn$\r\n~\/cumulus_evpn$<\/pre>\n
Lets run a few commands:<\/p>\n
~\/cumulus_evpn$ vagrant status\r\nCurrent machine states:\r\n\r\nspine1                    not created (virtualbox)\r\nspine2                    not created (virtualbox)\r\nleaf1                     not created (virtualbox)\r\nleaf2                     not created (virtualbox)\r\nserver1                   not created (virtualbox)\r\nserver2                   not created (virtualbox)\r\n\r\nThis environment represents multiple VMs. The VMs are all listed\r\nabove with their current state. For more information about a specific\r\nVM, run `vagrant status NAME`.<\/pre>\n
This shows the instances we are about to create…so lets build and provision them!<\/p>\n
~\/cumulus_evpn$vagrant up\r\n\r\n... output ommitted ...\r\n\r\n~\/cumulus_evpn$vagrant status\r\nCurrent machine states:\r\n\r\nspine1                    running (virtualbox)\r\nspine2                    running (virtualbox)\r\nleaf1                     running (virtualbox)\r\nleaf2                     running (virtualbox)\r\nserver1                   running (virtualbox)\r\nserver2                   running (virtualbox)\r\n\r\nThis environment represents multiple VMs. The VMs are all listed\r\nabove with their current state. For more information about a specific\r\nVM, run `vagrant status NAME`.\r\n~\/cumulus_evpn$ ~\/cumulus_evpn$\r\n\r\n\r\n<\/pre>\n
Our VMs are up…cool!<\/p>\n
If you watched vagrant do its magic, you will see that it also ran the ansible playbooks in a consumption model. Each time a machine build is completed, the playbooks are run, the dynamic inventory is matched against the new machine, and ansible deploys the new configuration to the machine. The next machine that is built is provisioned in the same way, but the ansible does not provision any of the previous instances because it has already completed them.<\/p>\n
Lets jump one of our servers and verify its working as expected. Here we log into server1 (10.1.1.101) and ping server2 (10.1.1.102).<\/p>\n
~\/cumulus_evpn$ vagrant ssh server1\r\nWelcome to Ubuntu 16.04 LTS (GNU\/Linux 4.4.0-22-generic x86_64)\r\n\r\n * Documentation:  https:\/\/help.ubuntu.com\/\r\n\r\n271 packages can be updated.\r\n159 updates are security updates.\r\n\r\nNew release '18.04.1 LTS' available.\r\nRun 'do-release-upgrade' to upgrade to it.\r\n\r\n\r\nLast login: Tue Jan 29 06:24:03 2019 from 10.0.2.2\r\nvagrant@server1:~$\r\nvagrant@server1:~$\r\nvagrant@server1:~$ ping 10.1.1.102\r\nPING 10.1.1.102 (10.1.1.102) 56(84) bytes of data.\r\n64 bytes from 10.1.1.102: icmp_seq=1 ttl=64 time=1.55 ms\r\n64 bytes from 10.1.1.102: icmp_seq=2 ttl=64 time=1.88 ms\r\n^C\r\n--- 10.1.1.102 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min\/avg\/max\/mdev = 1.555\/1.719\/1.884\/0.169 ms\r\nvagrant@server1:~$<\/pre>\n
It works! If you’ve gotten this far congratulations! You have layer-2 connected a couple of hosts across a layer-3 IP fabric….all using EVPN as the control plane, and vxlan as the virtual dataplane!<\/p>\n
Now its time to let our networking geek-birds fly. Lets view the configurations, verify the control and dataplane functionality using command line, and dig even deeper with a review of network captures from the IP fabric.<\/p>\n
Here is the leaf control plane configuration. Notice how the neighbor statements refer to interfaces, and not actual neighbor IP addresses. This is because the neighbors are established using an IPv6 link local addressing. Using this strategy, you simply specify the interface and the peer addressing is derived from IPv6 ND\/RD.<\/p>\n
vrf RED\r\n vni 104001\r\n exit-vrf\r\n!\r\nvrf BLUE\r\n vni 104002\r\n exit-vrf\r\n!\r\nrouter bgp 65101\r\n bgp router-id 10.255.255.11\r\n neighbor swp4 interface remote-as external\r\n neighbor swp5 interface remote-as external\r\n !\r\n address-family ipv4 unicast\r\n  redistribute connected route-map LOOPBACK_ROUTES\r\n exit-address-family\r\n !\r\n address-family l2vpn evpn\r\n  neighbor swp4 activate\r\n  neighbor swp5 activate\r\n  advertise-all-vni\r\n exit-address-family\r\n!\r\nroute-map LOOPBACK_ROUTES permit 10\r\n  match interface lo<\/pre>\n
Here is the interface configuration:<\/p>\n
vagrant@leaf1:~$ cat \/etc\/network\/interfaces\r\nauto lo\r\niface lo inet loopback\r\n    address 10.255.255.11\/32\r\n\r\nauto vagrant\r\niface vagrant inet dhcp\r\n\r\nauto eth0\r\niface eth0 inet dhcp\r\n\r\nauto swp1\r\niface swp1\r\n\r\nauto swp2\r\niface swp2\r\n\r\n...\r\n\r\n######\r\n### Define VRF and L3VNI\r\n######\r\nauto RED\r\niface RED\r\n    vrf-table auto\r\n\r\nauto SERVER01\r\niface SERVER01\r\n    bond-slaves swp2 swp3\r\n    bond-mode balance-xor\r\n    bridge-access 10\r\n\r\n#####\r\n### Define bridge\r\n#####\r\nauto bridge\r\niface bridge\r\n    bridge-ports SERVER01 VXLAN10\r\n    bridge-vids 10\r\n    bridge-vlan-aware yes\r\n\r\n#####\r\n### Define VXLAN interfaces\r\n#####\r\nauto VXLAN10\r\niface VXLAN10\r\n    bridge-access 10\r\n    bridge-arp-nd-suppress on\r\n    bridge-learning off\r\n    mstpctl-bpduguard yes\r\n    mstpctl-portbpdufilter yes\r\n    vxlan-id 10010\r\n    vxlan-local-tunnelip 10.255.255.11\r\n\r\nauto vlan10\r\niface vlan10\r\n    address 10.1.1.2\/24\r\n    address-virtual 00:00:00:00:00:1a 10.1.1.1\/24\r\n    vlan-id 10\r\n    vlan-raw-device bridge\r\n    vrf RED\r\n<\/pre>\n
In the above snippet, we combine our vlan10, VXLAN10, and SERVER01 into one bridge domain….named “bridge”. Our Layer3 interface, vlan 10, is assigned to the RED vrf.<\/p>\n
So there’s the config, lets verify it at the command line.<\/p>\n
~\/cumulus_evpn$ vagrant ssh leaf1\r\n\r\nWelcome to Cumulus VX (TM)\r\n\r\nCumulus VX (TM) is a community supported virtual appliance designed for\r\nexperiencing, testing and prototyping Cumulus Networks' latest technology.\r\nFor any questions or technical support, visit our community site at:\r\nhttp:\/\/community.cumulusnetworks.com\r\n\r\nThe registered trademark Linux (R) is used pursuant to a sublicense from LMI,\r\nthe exclusive licensee of Linus Torvalds, owner of the mark on a world-wide\r\nbasis.\r\nvagrant@leaf1:~$ sudo net show bgp evpn route\r\nBGP table version is 20, local router ID is 10.255.255.11\r\nStatus codes: s suppressed, d damped, h history, * valid, > best, i - internal\r\nOrigin codes: i - IGP, e - EGP, ? - incomplete\r\nEVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]\r\nEVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]\r\nEVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]\r\n\r\n   Network          Next Hop            Metric LocPrf Weight Path\r\nRoute Distinguisher: 10.255.255.11:3\r\n*> [2]:[0]:[0]:[48]:[44:38:39:00:00:09]\r\n                    10.255.255.11                      32768 i\r\n*> [2]:[0]:[0]:[48]:[44:38:39:00:00:09]:[32]:[10.1.1.101]\r\n                    10.255.255.11                      32768 i\r\n*> [2]:[0]:[0]:[48]:[44:38:39:00:00:09]:[128]:[fe80::4638:39ff:fe00:9]\r\n                    10.255.255.11                      32768 i\r\n*> [2]:[0]:[0]:[48]:[46:38:39:00:00:07]\r\n                    10.255.255.11                      32768 i\r\n*> [2]:[0]:[0]:[48]:[46:38:39:00:00:09]\r\n                    10.255.255.11                      32768 i\r\n*> [3]:[0]:[32]:[10.255.255.11]\r\n                    10.255.255.11                      32768 i\r\nRoute Distinguisher: 10.255.255.12:3\r\n*> [2]:[0]:[0]:[48]:[44:38:39:00:00:0d]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*  [2]:[0]:[0]:[48]:[44:38:39:00:00:0d]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*> [2]:[0]:[0]:[48]:[44:38:39:00:00:0d]:[32]:[10.1.1.102]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*  [2]:[0]:[0]:[48]:[44:38:39:00:00:0d]:[32]:[10.1.1.102]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*> [2]:[0]:[0]:[48]:[44:38:39:00:00:0d]:[128]:[fe80::4638:39ff:fe00:d]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*  [2]:[0]:[0]:[48]:[44:38:39:00:00:0d]:[128]:[fe80::4638:39ff:fe00:d]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*  [2]:[0]:[0]:[48]:[46:38:39:00:00:0d]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*> [2]:[0]:[0]:[48]:[46:38:39:00:00:0d]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*  [2]:[0]:[0]:[48]:[46:38:39:00:00:11]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*> [2]:[0]:[0]:[48]:[46:38:39:00:00:11]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*  [3]:[0]:[32]:[10.255.255.12]\r\n                    10.255.255.12                          0 65201 65102 i\r\n*> [3]:[0]:[32]:[10.255.255.12]\r\n                    10.255.255.12                          0 65201 65102 i\r\n\r\nDisplayed 12 prefixes (18 paths)\r\nvagrant@leaf1:~$<\/pre>\n
Look closely, the bgp evpn family is talking about MAC addresses. Its associating each MAC with a loopback address as the next hop. VXLAN will use this information to establish the overlay in the dataplane. Based on this output, the control plane seems to be working.<\/p>\n
Next….we capture the dataplane and test fault tolerance. We are using ECMP across our uplinks so we will have to shut one path down to make sure our captures are plentiful.<\/p>\n
~\/cumulus_evpn$ vagrant destroy spine1 -f\r\n==> spine1: Forcing shutdown of VM...\r\n==> spine1: Destroying VM and associated drives...\r\ntsimson@GMTI-Desktop-Tims-MacBook-Pro:~\/cumulus_evpn$ vagrant ssh server1\r\nWelcome to Ubuntu 16.04 LTS (GNU\/Linux 4.4.0-22-generic x86_64)\r\n\r\n * Documentation:  https:\/\/help.ubuntu.com\/\r\n\r\n271 packages can be updated.\r\n159 updates are security updates.\r\n\r\nNew release '18.04.1 LTS' available.\r\nRun 'do-release-upgrade' to upgrade to it.\r\n\r\n\r\nLast login: Tue Jan 29 08:35:20 2019 from 10.0.2.2\r\nvagrant@server1:~$ ping 10.1.1.1\r\nPING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.\r\n64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.388 ms\r\n^C\r\n--- 10.1.1.1 ping statistics ---\r\n1 packets transmitted, 1 received, 0% packet loss, time 0ms\r\nrtt min\/avg\/max\/mdev = 0.388\/0.388\/0.388\/0.000 ms\r\nvagrant@server1:~$ ping 10.1.1.102\r\nPING 10.1.1.102 (10.1.1.102) 56(84) bytes of data.\r\n64 bytes from 10.1.1.102: icmp_seq=1 ttl=64 time=1.93 ms\r\n64 bytes from 10.1.1.102: icmp_seq=2 ttl=64 time=1.66 ms<\/pre>\n
We toasted spine1 and the network continues to roll…awesome.<\/p>\n
We can now see that we are single threaded through spine2…and can be sure we are capturing everything through a single interface.<\/p>\n
~\/cumulus_evpn$ vagrant ssh leaf1\r\n\r\nvagrant@leaf1:~$ sudo net show bgp sum\r\n\r\nshow bgp ipv4 unicast summary\r\n=============================\r\nBGP router identifier 10.255.255.11, local AS number 65101 vrf-id 0\r\nBGP table version 7\r\nRIB entries 3, using 456 bytes of memory\r\nPeers 2, using 39 KiB of memory\r\n\r\nNeighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up\/Down State\/PfxRcd\r\nspine1(swp4)    4      65201    2354    2363        0    0    0 00:15:48      Connect\r\nspine2(swp5)    4      65201    2648    2657        0    0    0 01:07:25            1\r\n\r\nTotal number of neighbors 2\r\n\r\n\r\nshow bgp ipv6 unicast summary\r\n=============================\r\n% No BGP neighbors found\r\n\r\n\r\nshow bgp l2vpn evpn summary\r\n===========================\r\nBGP router identifier 10.255.255.11, local AS number 65101 vrf-id 0\r\nBGP table version 0\r\nRIB entries 3, using 456 bytes of memory\r\nPeers 2, using 39 KiB of memory\r\n\r\nNeighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up\/Down State\/PfxRcd\r\nspine1(swp4)    4      65201    2354    2363        0    0    0 00:15:48      Connect\r\nspine2(swp5)    4      65201    2648    2657        0    0    0 01:07:25            6\r\n\r\nTotal number of neighbors 2\r\nvagrant@leaf1:~$<\/pre>\n
Time to take captures.<\/p>\n
vagrant@leaf1:~$ sudo tcpdump -npi swp5 -w BB5.pcap && ping 10.1.1.102\r\ntcpdump: listening on swp5, link-type EN10MB (Ethernet), capture size 262144 bytes\r\n\r\n...\r\n\r\nvagrant@leaf1:~$ scp BB5.pcap tsimson@10.0.2.2:<\/pre>\n
Here’s our capture. I’ve filtered everything but some ICMP traffic. Check out the VXLAN header, more specifically the VNI.\u00a0 As defined by RFC-7348<\/a>:<\/p>\n
“Each VXLAN segment is identified through a 24-bit segment ID, termed the “VXLAN Network Identifier (VNI)”. This allows up to 16 M VXLAN segments to coexist within the same administrative domain. The VNI identifies the scope of the inner MAC frame originated by the individual VM.”<\/em><\/strong><\/p>\n
\"\"<\/a><\/p>\n
Once again we made it to the end of another pretty cool post. The use cases for combining Vagrant, Ansible, and Cumulus Linux are vast. In future posts, I hope to build on this topology by establishing routing between networks, to external networks, and by implementing security within the fabric.<\/p>\n
I had an absolute blast building and sharing this environment. I… and hopefully we… learned a ton!<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
In this post we will talk about how to use Vagrant, Ansible, and Cumulus Linux to build a virtual layer 2 extension across an IP fabric. We will use the standard protocol MP-BGP and its EVPN extension to run the control plane, and VXLAN to provide the virtual data plane. Each technology deserves a brief … <\/p>\n
Continue reading “Vagrant, Ansible, Cumulus, and EVPN….Orchestrating a Virtualized Network on your Laptop.”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-155","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/posts\/155","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/comments?post=155"}],"version-history":[{"count":28,"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/posts\/155\/revisions"}],"predecessor-version":[{"id":188,"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/posts\/155\/revisions\/188"}],"wp:attachment":[{"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/media?parent=155"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/categories?post=155"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tangosierratech.com\/blog\/wordpress\/wp-json\/wp\/v2\/tags?post=155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}