Arista L3 Leaf/Spine Step by Step (2b) - Recirculation

Topology:

Recirculation

In the previous blog, we see the host1 and host2 can NOT ping vtep3's SVI - VLAN2000 within the same VLAN. But host3 can reach this SVI. Why this happens?

This is because the Trident II ASIC doesn't support routing with overlay, recirculation channel is needed to loop the VXLAN inter-VLAN traffic back to the pipeline for routing lookup on some Arista switches like 7050QX. 

Bridging or Routing?

Based on the dstMAC, the ASIC determines the incoming packets to go to bridging or routing. The ping/ICMP packets from host2 to vtep3 will proceed vxlan decapsulation. Because its dstMAC is to routerMAC of vtep3, it is routing. So T2 can't handle vxlan decap and routing in 1 pass, thus ping failed

But if host2 pings host3, after vxlan decap, the packets are bridged. 

Similar host3 pings vtep3, the packets don't go thru vxlan decap, so ping is good. 

Which platform needs recirculation?

Only all Trident-2 or TH based platforms have this limitation. From above topology, other vteps like Jericho, T2+ doesn't need this. 

How to tell the chip model? The best way is to ask the account engineer who serves your account. Another way is to run the following CLI (based on my own experiences, if you know a better one please comment here. thanks!)

7280QR-C36-F(config)#sh platform fap

.....

Jericho0   !!! clearly this is a Jericho-based

7050QX-32-F#show platform fap

% Invalid input !!! FAP = Sand/Petra/Arad/Jericho, not supported

7050QX-32-F#show platform trident sys !! well this is a Trident 

         Slice              Chip       ModId    GenId

----------------- ----------------- ----------- -----

   FixedSystem       Linecard0/0           1        1  

------------------------------------------------------

Front panel vs internal ports

On the T2 system, the circulation can be done by front panel port and internal ports, depending on the switch model. A T2 chip can support 32 x 40G ports, some platforms like 7050TX-72/96, 7050SX/72/96, 7050S-64 don't use all ports at front panel, while the remaining ports are called internal ports. Using internal ports is definitely better than front panel, because it doesn't impact your switch connectivity capacity. 

So the next question is, how to tell if this switch has internal ports:-) Use CLI - "show inventory". 

7050SX-64-F.10:36:59(config)#show inventory

System has 81 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         64

  Unconnected      16  !!! has 16 unconnected ports

7050QX-32-F(config)#show inventory

System has 105 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         104 !!! No unconnected ports

Configuration:

Step1: Expose all internal ports (if the system has Unconnected ports under "show inventory", actually we don't need this for vtep3)
mLeafB.cd631.Z(config)#service interface unconnected expose
mLeafB.cd631.Z(config)#switch scheduler oversubscribed

Step2: Configure Recirc-channel (if T2 system)
upp224.vtep3(config)#int recirc-Channel 1
upp224.vtep3(config-if-Re1)#switchport recirculation features vxlan

Step3: Assign physical (front panel or internal) ports to recirc-channel

upp224.vtep3(config-if-Re1)#int et34
upp224.vtep3(config-if-Et34)#traffic-loopback source system device mac
upp224.vtep3(config-if-Et34)#channel-group recirculation 1
upp224.vtep3(config-if-Et34)#


Step4: Verify

upp224.vtep3#sh int recirc-Channel 1
Recirc-Channel1 is up, line protocol is up (connected)
  Hardware is Port-Channel, address is 2899.3a8b.e6fa
  Ethernet MTU 9214 bytes , BW 10000000 kbit
  Full-duplex, 10Gb/s
  Active members in this channel: 1
  ... Ethernet34 , Full-duplex, 10Gb/s
  Fallback mode is: off

Step5: ping from remote hosts

wa466.host2(vrf:host2)#ping 20.0.9.253
PING 20.0.9.253 (20.0.9.253) 72(100) bytes of data.
80 bytes from 20.0.9.253: icmp_seq=1 ttl=64 time=0.218 ms
80 bytes from 20.0.9.253: icmp_seq=2 ttl=64 time=0.150 ms
80 bytes from 20.0.9.253: icmp_seq=3 ttl=64 time=0.109 ms
80 bytes from 20.0.9.253: icmp_seq=4 ttl=64 time=0.107 ms
80 bytes from 20.0.9.253: icmp_seq=5 ttl=64 time=0.106 ms

--- 20.0.9.253 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.138/0.218/0.043 ms, ipg/ewma 0.195/0.175 ms