Vagrant, Ansible, Cumulus, and EVPN….Orchestrating a Virtualized Network on your Laptop.

In this post we will talk about how to use Vagrant, Ansible, and Cumulus Linux to build a virtual layer 2 extension across an IP fabric. We will use the standard protocol MP-BGP and its EVPN extension to run the control plane, and VXLAN to provide the virtual data plane.

Each technology deserves a brief introduction prior making things happen.

Vagrant up!

Vagrant is an orchestration tool that uses “providers” to deploy virtual machines to a variety of platforms. Vagrant is probably most famous for its integration with Virtualbox. It’s quite common for cloud and application developers to use Vagrant to build their application stacks locally. Vagrant then makes it easier to then deploy them to production in cloud environments, in an automated fashion. We will use Vagrant to deploy the switches and servers in our topology.

Ansible.

Ansible is another orchestration tool that helps you manage your infrastructure as code. It is agent-less and has probably the best tool box for network automation out of the various orchestration tools. In this post we will use ansible to provision, or configure, our devices.

Cumulus Linux.

Cumulus Networks is a virtual networking company that has made available a machine image running their networking software. This is a linux image that will be run as the switches in our topology. In the real world, you can run this software on a physical switch and operate the topology in the same way, just replace Vagrant with your OPs guy installing and cabling them. They publish the CumulusVX image, along with several automated topologies, so that you can do exactly what we are doing in this post, automate your network with Cumulus.

Last but not least….EVPN.

EVPN stands for Ethernet Virtual Private Network. Its another address-family under the MP-BGP standard. The technology is used as the control plane for transporting and updating the MAC address table to your switches across an IP fabric.

Another technology that should be mentioned, that is not in the title, is VXLAN. We will use VXLAN in the dataplane for encapsulating and transporting our ethernet frames.

Ok, here is the topology we are going to build:

 

On the bottom are our servers. Each server is connected to a single leaf switch in a mode 2 port channel. Many of the labs that Cumulus publishes dual attach the servers to two leaf switches and bonds them using their m-lag implementation. This requires lacp, which unfortunately, I was not able to get working locally. To keep the project moving forward, I implemented a work around which was to change the physical connectivity, and configure all of the bonds as mode 2 (balance-xor).

Cumulus provides some great tools for building these topologies, one of them is their topology converter.

Using their python script topology_converter.py, I took the following “.dot” file and converted it into a Vagrantfile. Vagrant will use this to build, and most importantly connect, all of the instances.

The Vagrant file that it creates is fairly long and took a number of modifications for what I wanted. Luckily, I’m going to share this with you… so clone this repo and take a look at it yourself. The repo will be needed to complete the build anyway.

The modifications that I made to the file involve commenting out some of the helper scripts, and using Vagrant’s ansible integration to run the playbooks.

Lets run a few commands:

This shows the instances we are about to create…so lets build and provision them!

Our VMs are up…cool!

If you watched vagrant do its magic, you will see that it also ran the ansible playbooks in a consumption model. Each time a machine build is completed, the playbooks are run, the dynamic inventory is matched against the new machine, and ansible deploys the new configuration to the machine. The next machine that is built is provisioned in the same way, but the ansible does not provision any of the previous instances because it has already completed them.

Lets jump one of our servers and verify its working as expected. Here we log into server1 (10.1.1.101) and ping server2 (10.1.1.102).

It works! If you’ve gotten this far congratulations! You have layer-2 connected a couple of hosts across a layer-3 IP fabric….all using EVPN as the control plane, and vxlan as the virtual dataplane!

Now its time to let our networking geek-birds fly. Lets view the configurations, verify the control and dataplane functionality using command line, and dig even deeper with a review of network captures from the IP fabric.

Here is the leaf control plane configuration. Notice how the neighbor statements refer to interfaces, and not actual neighbor IP addresses. This is because the neighbors are established using an IPv6 link local addressing. Using this strategy, you simply specify the interface and the peer addressing is derived from IPv6 ND/RD.

Here is the interface configuration:

In the above snippet, we combine our vlan10, VXLAN10, and SERVER01 into one bridge domain….named “bridge”. Our Layer3 interface, vlan 10, is assigned to the RED vrf.

So there’s the config, lets verify it at the command line.

Look closely, the bgp evpn family is talking about MAC addresses. Its associating each MAC with a loopback address as the next hop. VXLAN will use this information to establish the overlay in the dataplane. Based on this output, the control plane seems to be working.

Next….we capture the dataplane and test fault tolerance. We are using ECMP across our uplinks so we will have to shut one path down to make sure our captures are plentiful.

We toasted spine1 and the network continues to roll…awesome.

We can now see that we are single threaded through spine2…and can be sure we are capturing everything through a single interface.

Time to take captures.

Here’s our capture. I’ve filtered everything but some ICMP traffic. Check out the VXLAN header, more specifically the VNI.  As defined by RFC-7348:

“Each VXLAN segment is identified through a 24-bit segment ID, termed the “VXLAN Network Identifier (VNI)”. This allows up to 16 M VXLAN segments to coexist within the same administrative domain. The VNI identifies the scope of the inner MAC frame originated by the individual VM.”

Once again we made it to the end of another pretty cool post. The use cases for combining Vagrant, Ansible, and Cumulus Linux are vast. In future posts, I hope to build on this topology by establishing routing between networks, to external networks, and by implementing security within the fabric.

I had an absolute blast building and sharing this environment. I… and hopefully we… learned a ton!