Arista L3 Leaf/Spine Step by Step (2d) - VxLAN Bridging, interesting behavior

This post is for the paranoid, who really wants to know how it works under the hood. Based on previous setup, we see 2 interesting behaviors

1. Jericho MLAG peers also receive the arp reply

wa462.vtepB1#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.171267 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.171590 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

wa463.vtepB2#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.172821 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.173117 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

44:4c:a8:97:72:b7 is the MAC address of host2 belonging to vtep2. Why these 2 peers receive this unicast packet? The reason is that, when the VtepX1/X2 the first packet from remote Vtep, it will need some time to create a hw tunnel. During this gap, Vtep has to flood this ucast packet to all Vteps. 

When Vtep has no remote Vxlan MAC

ckp341.vtepX1#sh mac address-table dynamic interface vxlan 1
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
Total Mac Addresses for this criterion: 0

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

So Vtep has no remote vtep

ckp341.vtepX1#show vxlan vtep
Remote VTEPS for Vxlan1:
Total number of remote VTEPS:  0

Let's create a static MAC pointing to remote VTEP. After there is a remote Vtep. 

ckp341.vtepX1(config)#mac address-table static 0000.1111.2222 vlan 2000 interface vxlan 1 vtep 200.255.200.8

ckp341.vtepX1(config)#show vxlan vtep

Remote VTEPS for Vxlan1:

200.255.200.8

Total number of remote VTEPS:  1

Repeat the ping. Now the Mlag peer doesn't receive the unicast packet. 

wa462.vtepB1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

22:22:16.623983 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

2. ONLY 1 T2+ MLAG peer see receive the arp request, and no arp reply.

vtep1 has no arp req

snp262.vtepA1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

^C

0 packets captured

vtepA2 sees arp req

snp263.vtepA2(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

06:27:56.665383 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

^C

1 packet captured

This is because Trident family ASIC pipeline can't handle vxlan decap and routing in 1 pass. The arp req seen on vtepA2 is flooded by vtepA1. From the output of "show mac address mlag-peer", you can see vtepA2 learning this MAC from mlag-peer.

snp263.vtepA2(config)#sh mac address-table mlag-peer

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

2000    0000.1111.2222    STATIC      Vx1

2000    001c.73ff.405e    STATIC      Po2000

2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:25 ago

4094    001c.73ff.405e    STATIC      Po2000

Total Mac Addresses for this criterion: 4

Arista L3 Leaf/Spine Step by Step (2b) - Recirculation

Topology:

Recirculation

In the previous blog, we see the host1 and host2 can NOT ping vtep3's SVI - VLAN2000 within the same VLAN. But host3 can reach this SVI. Why this happens?

This is because the Trident II ASIC doesn't support routing with overlay, recirculation channel is needed to loop the VXLAN inter-VLAN traffic back to the pipeline for routing lookup on some Arista switches like 7050QX. 

Bridging or Routing?

Based on the dstMAC, the ASIC determines the incoming packets to go to bridging or routing. The ping/ICMP packets from host2 to vtep3 will proceed vxlan decapsulation. Because its dstMAC is to routerMAC of vtep3, it is routing. So T2 can't handle vxlan decap and routing in 1 pass, thus ping failed

But if host2 pings host3, after vxlan decap, the packets are bridged. 

Similar host3 pings vtep3, the packets don't go thru vxlan decap, so ping is good. 

Which platform needs recirculation?

Only all Trident-2 or TH based platforms have this limitation. From above topology, other vteps like Jericho, T2+ doesn't need this. 

How to tell the chip model? The best way is to ask the account engineer who serves your account. Another way is to run the following CLI (based on my own experiences, if you know a better one please comment here. thanks!)

7280QR-C36-F(config)#sh platform fap

.....

Jericho0   !!! clearly this is a Jericho-based

7050QX-32-F#show platform fap

% Invalid input !!! FAP = Sand/Petra/Arad/Jericho, not supported

7050QX-32-F#show platform trident sys !! well this is a Trident 

         Slice              Chip       ModId    GenId

----------------- ----------------- ----------- -----

   FixedSystem       Linecard0/0           1        1  

------------------------------------------------------

Front panel vs internal ports

On the T2 system, the circulation can be done by front panel port and internal ports, depending on the switch model. A T2 chip can support 32 x 40G ports, some platforms like 7050TX-72/96, 7050SX/72/96, 7050S-64 don't use all ports at front panel, while the remaining ports are called internal ports. Using internal ports is definitely better than front panel, because it doesn't impact your switch connectivity capacity. 

So the next question is, how to tell if this switch has internal ports:-) Use CLI - "show inventory". 

7050SX-64-F.10:36:59(config)#show inventory

System has 81 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         64

  Unconnected      16  !!! has 16 unconnected ports

7050QX-32-F(config)#show inventory

System has 105 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         104 !!! No unconnected ports

Configuration:

Step1: Expose all internal ports (if the system has Unconnected ports under "show inventory", actually we don't need this for vtep3)
mLeafB.cd631.Z(config)#service interface unconnected expose
mLeafB.cd631.Z(config)#switch scheduler oversubscribed

Step2: Configure Recirc-channel (if T2 system)
upp224.vtep3(config)#int recirc-Channel 1
upp224.vtep3(config-if-Re1)#switchport recirculation features vxlan

Step3: Assign physical (front panel or internal) ports to recirc-channel

upp224.vtep3(config-if-Re1)#int et34
upp224.vtep3(config-if-Et34)#traffic-loopback source system device mac
upp224.vtep3(config-if-Et34)#channel-group recirculation 1
upp224.vtep3(config-if-Et34)#


Step4: Verify

upp224.vtep3#sh int recirc-Channel 1
Recirc-Channel1 is up, line protocol is up (connected)
  Hardware is Port-Channel, address is 2899.3a8b.e6fa
  Ethernet MTU 9214 bytes , BW 10000000 kbit
  Full-duplex, 10Gb/s
  Active members in this channel: 1
  ... Ethernet34 , Full-duplex, 10Gb/s
  Fallback mode is: off

Step5: ping from remote hosts

wa466.host2(vrf:host2)#ping 20.0.9.253
PING 20.0.9.253 (20.0.9.253) 72(100) bytes of data.
80 bytes from 20.0.9.253: icmp_seq=1 ttl=64 time=0.218 ms
80 bytes from 20.0.9.253: icmp_seq=2 ttl=64 time=0.150 ms
80 bytes from 20.0.9.253: icmp_seq=3 ttl=64 time=0.109 ms
80 bytes from 20.0.9.253: icmp_seq=4 ttl=64 time=0.107 ms
80 bytes from 20.0.9.253: icmp_seq=5 ttl=64 time=0.106 ms

--- 20.0.9.253 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.138/0.218/0.043 ms, ipg/ewma 0.195/0.175 ms