VXLAN Bridging with MLAG

VXLAN Bridging with MLAG

• Key takeaways:
• FH VTEP Encap/Decap
• Routing between MLAG peers
• MLAG peers share the same loopback/VTI address
• https://eos.arista.com/vxlan-with-mlag-configuration-guide/
• Provides remote L2 connectivity between racks or DC;
• Each MLAG domain(2 MLAG peers) has ONE logical VTEP
• Same virtual tunnel ip address (VTI)
• 因为两个MLAG Peer work as ONE physical switch
• MAC Sync:
• For encap/decap traffic, both local and remote MAC address need to be sync'ed between peers via peer-link
• remote = remote MAC associated with remote VTEP ip address. 

Configuration (same on both Mlag peers)

interface loopback1
 ip address 192.168.0.1/32
interface vxlan1
 vlan source-interface loopback 1
 vxlan udp-port 4789
 vxlan vlan 10 vni 10
 vxlan vlan 10 flood 192.168.0.2

MAC, ARP, Traffic例子

• serverA (macA) under MLAG domain 1 (Peer1A和1B)，比方说VLAN 10，sends ARP request
• ARP Req 会被Hash over 1 link of 2-port LAG. 
• Peer1A受到这个ARP req, 有4个Actions
• Act#1: peer1A floods this ARP Req所有本地VLAN 10的端口，因为是Bcast Pkt
• Act#2: peer1A floods it to peer1B，这是给peer1B上面的Singly端口
• peer1B只会flood singly ports，而不会flood dual-home ports
• Act#3: peer1A sync with 1B，peer1B知道 macA 是在Port-channel上面
• 这个Sync是另外的 MLAG signaling, 
• Act#4: peer1A ENCAP ARP in VXLAN and floods all VTEP
• FH GW 负责encap/decap vxlan traffic
• VXLAN pkt is ECMP'ed to spine then to remote Peer2A/B，
• Pear2A/B 和 1A/B一样，share一个VTI address，所以逻辑上是一个
• peer1A ECMP to one spine;
• This spine 有2个path to VTEP 192.168.0.2, 比方说ECMP to peer2A
• Peer2A首到ARP req, DECAP VXLAN pkt and learns MACa in from VTEP 192.168.0.1, 以下是标准的MLAG流程 和 Peer1A很类似
• Act#1: Peer2A flood ARP req all local ports
• Act#2: Peer2A flood it via peer-link for those singly ports on Peer2B
• Act#3: Peer2A sync‘s with Peer2B, MACa from VTEP 192.168.0.1
• peerRemoteDynamic
• ServerB unicasts ARP response to ServerA
• dstMAC = MAC.AAA; srcMAC = MAC.BBB
• Now both peer2A/2B know MAC.AAA is on VTEP 192.168.0.1, and ARP response is encap into VXLAN and routed to peer1B
• ENCAP ARP reply on FH device
• 如果peer1B 收到这个ARP reply, 
• Learns MAC.BBB from VTEP 192.168.0.2, remoteDynamic;
• sync with peer1A
• and pkt fwd down to port-ch 10

Useful CLIs:

• show mac address, Ports里面有Vx1
• show vxlan address-table, 有Mac/Vtep/Port

Switch over 例子，例如Peer1A lose all uplinks

• First Hop MLAG Peer/VTEP，负责encap/decap pkts, 这个是Principle
• 例如MLAG peer1A lost all uplinks，但是device is up running
• 还是Peer1A encap/decap pkts
• 需要Routing between peer via Peerlink
• Best Practise是建议routing on a dedicated VLAN而不是Peerlink VLAN 

Arista L3 Leaf/Spine Step by Step (2d) - VxLAN Bridging, interesting behavior

This post is for the paranoid, who really wants to know how it works under the hood. Based on previous setup, we see 2 interesting behaviors

1. Jericho MLAG peers also receive the arp reply

wa462.vtepB1#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.171267 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.171590 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

wa463.vtepB2#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.172821 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.173117 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

44:4c:a8:97:72:b7 is the MAC address of host2 belonging to vtep2. Why these 2 peers receive this unicast packet? The reason is that, when the VtepX1/X2 the first packet from remote Vtep, it will need some time to create a hw tunnel. During this gap, Vtep has to flood this ucast packet to all Vteps. 

When Vtep has no remote Vxlan MAC

ckp341.vtepX1#sh mac address-table dynamic interface vxlan 1
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
Total Mac Addresses for this criterion: 0

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

So Vtep has no remote vtep

ckp341.vtepX1#show vxlan vtep
Remote VTEPS for Vxlan1:
Total number of remote VTEPS:  0

Let's create a static MAC pointing to remote VTEP. After there is a remote Vtep. 

ckp341.vtepX1(config)#mac address-table static 0000.1111.2222 vlan 2000 interface vxlan 1 vtep 200.255.200.8

ckp341.vtepX1(config)#show vxlan vtep

Remote VTEPS for Vxlan1:

200.255.200.8

Total number of remote VTEPS:  1

Repeat the ping. Now the Mlag peer doesn't receive the unicast packet. 

wa462.vtepB1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

22:22:16.623983 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

2. ONLY 1 T2+ MLAG peer see receive the arp request, and no arp reply.

vtep1 has no arp req

snp262.vtepA1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

^C

0 packets captured

vtepA2 sees arp req

snp263.vtepA2(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

06:27:56.665383 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

^C

1 packet captured

This is because Trident family ASIC pipeline can't handle vxlan decap and routing in 1 pass. The arp req seen on vtepA2 is flooded by vtepA1. From the output of "show mac address mlag-peer", you can see vtepA2 learning this MAC from mlag-peer.

snp263.vtepA2(config)#sh mac address-table mlag-peer

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

2000    0000.1111.2222    STATIC      Vx1

2000    001c.73ff.405e    STATIC      Po2000

2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:25 ago

4094    001c.73ff.405e    STATIC      Po2000

Total Mac Addresses for this criterion: 4

Arista L3 Leaf/Spine Step by Step (2a) - Vxlan Bridging, configuration

Topology:

Configuration:


As long as you can reach the loopback 10 of each VTEPs, the Vxlan bridging configuration is quite straightforward:

wa465.vtep2#sh run int vxlan 1
interface Vxlan1
   vxlan source-interface Loopback10
   vxlan udp-port 4789
   vxlan vlan 2000 vni 2000
   vxlan flood vtep 200.255.200.7 200.255.200.8 200.255.200.9 200.255.200.12 200.255.200.34 200.255.200.56

Basically, you need to specify the following parameters:

• vxlan source interface. (loopback10 in this case)
• vxlan udp port (by default is 4789)
• vlan and vni mapping
• vxlan vtep flood list. 

Please note that you can include the local VTI in the flood list to keep vxlan interface configuration same across the whole domain. 

Ping tests

We can verify the setup by pinging the host address, and GW phy/virtual address. For example, from wa466.host2 (the host behind the bow-tie MLAG), 

ping 20.0.12.250
ping 20.0.12.251

ping 20.0.12.252

ping 20.0.12.253

!

ping 20.0.34.250

ping 20.0.34.251

ping 20.0.34.252

ping 20.0.34.253

!

ping 20.0.56.250

ping 20.0.56.251

ping 20.0.56.252

ping 20.0.56.253
!

ping 20.0.7.250

ping 20.0.7.253

!

ping 20.0.8.250

ping 20.0.8.253

!

ping 20.0.9.250

ping 20.0.9.253 <<<< failed

....

! all successful except 20.0.9.253

--- 20.0.9.250 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 12ms

rtt min/avg/max/mdev = 0.088/0.230/0.577/0.201 ms, ipg/ewma 3.049/0.424 ms

The ping to all host and GW are successful, except the one from wa466.host2 to Vtep3. But the ping from bn303.host3 to vtep3's SVI works. Why?

bn303.spine.host3(s1)(vrf:host3)#ping 20.0.9.253
PING 20.0.9.253 (20.0.9.253) 72(100) bytes of data.
80 bytes from 20.0.9.253: icmp_seq=1 ttl=64 time=0.214 ms
80 bytes from 20.0.9.253: icmp_seq=2 ttl=64 time=0.096 ms
80 bytes from 20.0.9.253: icmp_seq=3 ttl=64 time=0.100 ms
80 bytes from 20.0.9.253: icmp_seq=4 ttl=64 time=0.086 ms
80 bytes from 20.0.9.253: icmp_seq=5 ttl=64 time=0.087 ms

This is because recirculation, which is required for Vxlan routing on T2 platform. We will cover it in next blog. 

Configuration Files