Arista J/J+ FlexRoute with Internet Routing Table

https://www.arista.com/assets/data/pdf/Whitepapers/FlexRoute-WP.pdf

• CLI: 
• ip hardware fib optimize prefix-length internet
• show ip bgp summary
• show ip route summary
• show hardware capacity | grep Routing
• Routing HW resources are sliced to fit the route distribution of REAL internet routing table
• If loading a lab configuration like Ixia, you mean see the error message like - "%CAPACITY-1-UTILIZATION_HIGH. LEM table utilization is currently at 91%, crossed threshold 90%"
• For example, if too many /24, it will be moved to LEM
• And another important concept of prefix, route and FIB:
• 3 x internet feeds = 3 x 600K prefixes
• 3 prefixes of same dest = 1 route with 3 NH if ECMP
• 1 route = 1 FIB entry

IP fragmentation on Arad platform

• Fragmented/MTU violated packet is sw forwarded and shaped by CoPP or PDP;
• CoPP = 2~2.5 Mbps
• "no shape" under copp-system-mtu can make it 100Mbps
• PDP = 100 Mbps
• But not a good idea to do it. 

TH/Trident low LPM

On Arista DCS-7260CX-64-F platform, if you see hw resource - LPM/V4Routes running out, have to change the "platform trident forward partition". Agent will restart but doesn't a reboot. 

nv475.08:45:07(config)#sh hardware capacity utilization percent exceed 50
Forwarding Resources Usage

Table   Feature    Chip       Used   Used      Free    Committed    Best Case        High
                           Entries    (%)   Entries      Entries          Max   Watermark
                                                                      Entries
------- ---------- ------ --------- ------ ---------- ------------ ------------ ---------
LPM                           8190   100%         0            0         8190        8190
LPM     V4Routes              8190   100%         0            0         8190        8190

nv475.08:52:45(config)#sh ip route 151.131.34.0

VRF: default
======================================================
WARNING: Some of the routes are not programmed in
hardware, and they are marked with '*'.
======================================================

*B E    151.131.34.0/24 [200/0] via 100.1.0.44, Vlan10

nv476.11:05:47#sh platform trident forwarding-table partition

L2 Table Size: 72k

L3 Host Table Size: 72k

LPM Table Size: 16k, uRPF is disabled

nv475.09:27:35(config)#platform trident forwarding-table partition ?

  0  136k l2 entries, 8k l3 host, 16k lpm entries

  1  104k l2 entries, 40k l3 host, 16k lpm entries

  2  72k l2 entries, 72k l3 host, 16k lpm entries

  3  40k l2 entries, 104k l3 host, 16k lpm entries

  4  8k l2 entries, 8k l3 host, 90k lpm entries <<< this one!

But this change will impact MLAG because lower MAC capacity. 

Take away notes from RFC5549 video by AMX-IX

• https://www.youtube.com/watch?v=uJOtfiHDCMw
• AMS-IX running out /22 ipv4 NH address space
• Solution #1: more address space:
• ARP broadcast too much
• Solution #2: private address by RFC1918, 10/8, 172.16/12, 192.168/16
• Space used by ISP customer
• More specific in IGP
• Break traceroute
• Solution #5: RFC5549
• IPv4 NLRI over IPv6 NH
• 2 ways to implement it:
• v4 in v6 tunnel, add 40B head
• direct forwarding - just replace the MAC, no change;

MLAG Fast Convergence - MAC Redirection/Promption

https://eos.arista.com/eos-4-18-0f/mlag-unicast-convergence/

Problem Description:

Consider the following setup 

• mlagA and mlagB are 2 mlag peers with port-ch 2000;
• The host MAC - 0000:1111:2222 is learnt on MLAG 10 and A is the owner. 
• In another way, MAC 0000:1111:2222 is A's local MAC and B's remote
• A bit background:
• The MAC address and ARP information are all sync'ed during boot-up;
• After that, only MAC table is sync'ed, for example, A tells B that MAC a.b.c is from mlag po10, or singly interface, or remote vtep. 
• So in the MAC table, the MAC has at least 4 states:
• learnedDynamic (local mlag), 
• peerDynamic (remote mlag), 
• learnedRemoteDynamic (vxlan) 
• peerRemoteDynamic (remote vxlan)
• Get this information by command - show mac address mlag-peer

Now saying we have 

• link failure, which has 2 loss: down and up
• when B's po 10 is down, all MAC are re-programmed from po 10 to po 2000, so be MAC move is done one by one in old releases before 4.18.1F. (#1)
• when B's po1 10 is back up, the ACL on peerLink to block BUM traffic immediately to break L2 loop while the MAC move needs time to be completed. (#2)
• node failure, which introduces 3 loss, down, up and delay timeout
• node down, 100s msec loss depending on scale
• node up, 100s msec loss (#3)
• When the peerlink is up, why? At this time, peerB has no uplink or downlink up, which are all in reload-delay. 
• Remember the MAC sync mentioned above? A needs to sync up with B on the MAC, so on A these MAC learnt from B will be flushed!!
• A has to relearn these MAC, 50% all of sudden. Still ok for locally switched packets because the hw flooding kicks in. 
• But bad for Vxlan, which requires software flood for the head-end-replication. 
• reload-delay timeout, 100s msec, actually 2 times
• Need to have iBGP or IGP L3 routing between 2 peers.
• Since peerlink is up fast and much earlier than mlag/non-mlag interface. So when non-mlag or mlag interfaces are up, they can send the traffic to peer link before the optimal path converged. 

Feature and solution

So from 4.18F, a feature called MLAG fast MAC redirection is developed to address above issues. This feature has 2 aspects:

1. MAC redirect, for #1 and #2 loss

• With this feature, the the interface attribute of impacted MACs still point to MLAG po 10 in host table. So, no move at all. 
• Strata and Sand implementation are slightly different but same idea. 
• On Sand, it is to use a recirc channel on each Arad/Jericho chip to recycle the MLAG destined packets over to peer-link. 
• Requirements and limitation:
• Peerlink must be a LAG not Ethernet on Strata. 
• On Sand "platform sand lag hardware-only" must be enabled, I believe only hw LAG can share member port - the recirc channel. 
• MLAG ASU2 cannot co-exist. 

2. MAC address promotion, targeting #3. 

• When 1) peer reboot; 2) hitful restart of fwding plane, the remote MACs (learnt via peer switch) will be flushed, which cause
• 1) before the MAC is re-learnt, packets needs to flooded. Still ok in pure L2/L3 environment because done by hw;
• 2) software forward of Vxlan packets. That's a big issue because it results in drops by CoPP. 
• Why the MACs are flushed?
• when peerB is up, MAC are sync'ed from A to B. These remote MAC are flushed. No MAC, then flooding. 
• Solution: 
• when peerB is down, the ownership of MAC are transfered to peerA
• when peerB recovers, *ALL* MAC are sync'ed from A to B
• Details:
• when peerB is down, peerA enters failover state;
• peerDynamic, peerLearnedRemote to learnedDynamic, learnedRemoteDynamic
• NOT single-leg host

Misc:

• From AD1554:
• If all uplinks are L3 interfaces, then it is preferable to keep non-mlag reload-delay timer < mlag timer, so
• Upstream/L3 up first before downstream/L2;
• In this way, S-N traffic should be no loss. 
• Enabled "reload-delay mode lacp standby", need to have non-mlag timre >= mlag timer,
• MLAG interfaces with LACP are kept warm for LAG membership table, MAC table programming. 
• But upstream/L3 must be up after L2/downstream/mlag interfaces, otherwise S/N traffic are blackhole'd. 
• From AD3152:
• SandL3Unicast - managing NH and ensuring EEDB no change
• SandACL - programming DROP ACL on the peerLinkRecircPort avoid pkts from peerLink back to peerLink
• Assigning LagMemberID is interesting, this peerLinkRecircPort needs a member id. what about overflown?
• LAG member, C/D bit
• C = collecting, D = Distributing
• if static LAG, C/D=True, added to LAG
• if LACP enabled, only C=True, D=True, member can be added. 
• peerLinkRecircPort is always C/D=False/True
• LC removal event
• If all ports on this LC, 
• L3 will have some downtime since losing all ARP entries; 
• L2 should be fine once recirc port is programmed. 
• If at least 1 member on another LC, L2/L3 should be fine
• This is quite complicated! 
• 3/1, 3/36, 4/1 are local member of mlag Po 10
• all 3 ports down, all 3 members retained with C/D=False
• PeerLinkRecicFap is added from either 3/0, 3/2 or 4/0 
• If LC3 is pulled, peerLinkRecirFap 4/0 is added. 
• And a lot of combination of events, LAG config change, member port down...
• 4 events:
• LC removal
• LAG config change: unconfig and change config
• Member port down or cable unplug
• port-ch shutdown
• CLIs:
• show plat trident counter int e27/1
• Drops counts in Vlan boundary = Vlan ID missed. In another word, the VLAN id programming on this interface is not done yet. 

Reference:

• PeerOne Vxlan + MLAG
• AD3398, BG141435/96642
• AD3152
• AD1554

Arista EOS - %HARDWARE-3-DROP_COUNTER_ALERT, DchUnreachables

What does the following log message mean?

May 15 15:45:42 bn302 EventMgr: %HARDWARE-3-DROP_COUNTER_ALERT: Persistent Internal Drop 'DchUnreachables': 792271126642 detected on Fe3600-3/1

And there is high # of drop in the output of "sh hardware counter drop"

bn302.jhm.mlagB.profA0.w.15:47:58#show hardware counter drop
Summary:
Total Adverse (A) Drops: 20098028670058
Total Congestion (C) Drops: 0
Total Packet Processor (P) Drops: 72029
Type  Chip         CounterName                    :           Count : First Occurrence    : Last Occurrence
--------------------------------------------------------------------------------------------------------------
A     Fe3600-4/2   DchUnreachables-1              :    103285456899 : 2018-05-15 15:34:55 : 2018-05-15 15:35:07

Basically when the LCs are not ready and up, the Fabric modules don't know where to forward the packets, they will drop them and increase this counter. It is quite common during system booting up. 

How to use python/yaml

import yaml
import sys

with open(sys.argv[1], 'r') as stream:
   try:
      paramDict = yaml.load(stream)
   except yaml.YAMLError as err:
      print(err)
      sys.ext(0)

print paramDict['Devices']['Switch']['mlagA']['Mgmt']['Host']

============
Devices:
   Switch:
      mlagA:                                 # MLAG Dut1
         Mgmt:                               
            Host: bn303
            AccessMethod:    ssh             # ssh or capi
            SshUsername:     admin           # default admin
            ChassisType:     modular         # fixed or modular
            CapiProtocol:    https

~/py @arst1.sjc> python test.py test.yaml

bn303

Arista L3 Leaf/Spine Step by Step (2d) - VxLAN Bridging, interesting behavior

This post is for the paranoid, who really wants to know how it works under the hood. Based on previous setup, we see 2 interesting behaviors

1. Jericho MLAG peers also receive the arp reply

wa462.vtepB1#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.171267 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.171590 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

wa463.vtepB2#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.172821 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.173117 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

44:4c:a8:97:72:b7 is the MAC address of host2 belonging to vtep2. Why these 2 peers receive this unicast packet? The reason is that, when the VtepX1/X2 the first packet from remote Vtep, it will need some time to create a hw tunnel. During this gap, Vtep has to flood this ucast packet to all Vteps. 

When Vtep has no remote Vxlan MAC

ckp341.vtepX1#sh mac address-table dynamic interface vxlan 1
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
Total Mac Addresses for this criterion: 0

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

So Vtep has no remote vtep

ckp341.vtepX1#show vxlan vtep
Remote VTEPS for Vxlan1:
Total number of remote VTEPS:  0

Let's create a static MAC pointing to remote VTEP. After there is a remote Vtep. 

ckp341.vtepX1(config)#mac address-table static 0000.1111.2222 vlan 2000 interface vxlan 1 vtep 200.255.200.8

ckp341.vtepX1(config)#show vxlan vtep

Remote VTEPS for Vxlan1:

200.255.200.8

Total number of remote VTEPS:  1

Repeat the ping. Now the Mlag peer doesn't receive the unicast packet. 

wa462.vtepB1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

22:22:16.623983 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

2. ONLY 1 T2+ MLAG peer see receive the arp request, and no arp reply.

vtep1 has no arp req

snp262.vtepA1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

^C

0 packets captured

vtepA2 sees arp req

snp263.vtepA2(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

06:27:56.665383 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

^C

1 packet captured

This is because Trident family ASIC pipeline can't handle vxlan decap and routing in 1 pass. The arp req seen on vtepA2 is flooded by vtepA1. From the output of "show mac address mlag-peer", you can see vtepA2 learning this MAC from mlag-peer.

snp263.vtepA2(config)#sh mac address-table mlag-peer

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

2000    0000.1111.2222    STATIC      Vx1

2000    001c.73ff.405e    STATIC      Po2000

2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:25 ago

4094    001c.73ff.405e    STATIC      Po2000

Total Mac Addresses for this criterion: 4

Arista L3 Leaf/Spine Step by Step (2c) - VxLAN Bridging, packet walkthru

Step 1: Ping hostX from host2. 

Before that we clear arp so force the host2 to send out broadcast arp req. Also run "bash tcpdump -nvvi vlan2000 arp" to see the arp packet behavior


Step2: host2 sends the arp to vtep2. 



Step 3: Vtep flood ARP req.

Besides flooding out the local vlan ports, the vtep will also do hardware replication to send 1 copy to each remote vtep, which is called HER (head-end-replication)

Please note that, only ONE mlag peer receive this arp request based on the hashing. Because 2 mlag peers share the same loopback, they appear as 1 destination with 2-way ecmp. 



Step 4: Remote Vtep learn and flood

When the remote VTEP receives this arp req, they will do:
1) flood all local ports to reach the destination; 
2) send it to mlag peer via peer link if applied; 
3) learn the MAC address of host2, like

wa462.vtepB1#sh mac address-table dynamic | grep 72b7
2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:42 ago

wa463.vtepB2#sh mac address-table dynamic | grep 72b7
2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:42 ago

Why flooding to MLAG peer? Say if there is one singly connected host but this arp request was hashed to the other MLAG peer. So we need this step to reach these orphan clients. 

Please note that both MLAG peers assign the srcHost's MAC to interface Vx1. Basically, MLAG peer never learns the MAC via peer link. Instead, they use MLAG message to sync MAC learning. 



Step 5. dstHost unicasts ARP reply back to srcHost

Since VtepX1 nad VtepX2 already learn srcMAC, it knows how to unicast it back to srcVtep

ckp342.vtepX2#sh mac address-table dynamic
Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:03 ago

Arista L3 Leaf/Spine Step by Step (2b) - Recirculation

Topology:

Recirculation

In the previous blog, we see the host1 and host2 can NOT ping vtep3's SVI - VLAN2000 within the same VLAN. But host3 can reach this SVI. Why this happens?

This is because the Trident II ASIC doesn't support routing with overlay, recirculation channel is needed to loop the VXLAN inter-VLAN traffic back to the pipeline for routing lookup on some Arista switches like 7050QX. 

Bridging or Routing?

Based on the dstMAC, the ASIC determines the incoming packets to go to bridging or routing. The ping/ICMP packets from host2 to vtep3 will proceed vxlan decapsulation. Because its dstMAC is to routerMAC of vtep3, it is routing. So T2 can't handle vxlan decap and routing in 1 pass, thus ping failed

But if host2 pings host3, after vxlan decap, the packets are bridged. 

Similar host3 pings vtep3, the packets don't go thru vxlan decap, so ping is good. 

Which platform needs recirculation?

Only all Trident-2 or TH based platforms have this limitation. From above topology, other vteps like Jericho, T2+ doesn't need this. 

How to tell the chip model? The best way is to ask the account engineer who serves your account. Another way is to run the following CLI (based on my own experiences, if you know a better one please comment here. thanks!)

7280QR-C36-F(config)#sh platform fap

.....

Jericho0   !!! clearly this is a Jericho-based

7050QX-32-F#show platform fap

% Invalid input !!! FAP = Sand/Petra/Arad/Jericho, not supported

7050QX-32-F#show platform trident sys !! well this is a Trident 

         Slice              Chip       ModId    GenId

----------------- ----------------- ----------- -----

   FixedSystem       Linecard0/0           1        1  

------------------------------------------------------

Front panel vs internal ports

On the T2 system, the circulation can be done by front panel port and internal ports, depending on the switch model. A T2 chip can support 32 x 40G ports, some platforms like 7050TX-72/96, 7050SX/72/96, 7050S-64 don't use all ports at front panel, while the remaining ports are called internal ports. Using internal ports is definitely better than front panel, because it doesn't impact your switch connectivity capacity. 

So the next question is, how to tell if this switch has internal ports:-) Use CLI - "show inventory". 

7050SX-64-F.10:36:59(config)#show inventory

System has 81 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         64

  Unconnected      16  !!! has 16 unconnected ports

7050QX-32-F(config)#show inventory

System has 105 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         104 !!! No unconnected ports

Configuration:

Step1: Expose all internal ports (if the system has Unconnected ports under "show inventory", actually we don't need this for vtep3)
mLeafB.cd631.Z(config)#service interface unconnected expose
mLeafB.cd631.Z(config)#switch scheduler oversubscribed

Step2: Configure Recirc-channel (if T2 system)
upp224.vtep3(config)#int recirc-Channel 1
upp224.vtep3(config-if-Re1)#switchport recirculation features vxlan

Step3: Assign physical (front panel or internal) ports to recirc-channel

upp224.vtep3(config-if-Re1)#int et34
upp224.vtep3(config-if-Et34)#traffic-loopback source system device mac
upp224.vtep3(config-if-Et34)#channel-group recirculation 1
upp224.vtep3(config-if-Et34)#


Step4: Verify

upp224.vtep3#sh int recirc-Channel 1
Recirc-Channel1 is up, line protocol is up (connected)
  Hardware is Port-Channel, address is 2899.3a8b.e6fa
  Ethernet MTU 9214 bytes , BW 10000000 kbit
  Full-duplex, 10Gb/s
  Active members in this channel: 1
  ... Ethernet34 , Full-duplex, 10Gb/s
  Fallback mode is: off

Step5: ping from remote hosts

wa466.host2(vrf:host2)#ping 20.0.9.253
PING 20.0.9.253 (20.0.9.253) 72(100) bytes of data.
80 bytes from 20.0.9.253: icmp_seq=1 ttl=64 time=0.218 ms
80 bytes from 20.0.9.253: icmp_seq=2 ttl=64 time=0.150 ms
80 bytes from 20.0.9.253: icmp_seq=3 ttl=64 time=0.109 ms
80 bytes from 20.0.9.253: icmp_seq=4 ttl=64 time=0.107 ms
80 bytes from 20.0.9.253: icmp_seq=5 ttl=64 time=0.106 ms

--- 20.0.9.253 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.138/0.218/0.043 ms, ipg/ewma 0.195/0.175 ms

Notes of EOS VXLAN Chapter

EOS VXLAN Chapter

• Architecture, couple of new concepts with similar one in VLAN world
• VXLAN = VLAN
• VTEP = Switch
• VTI = Port
• VNI = VLAN
• 1 VNI can attach different VLANs on different VTEP
• HER (head-end replication) handles BUM traffic by using flood list
• VXLAN vs MLAG
• From downstream L2 devices perspective, MLAG peers appear as ONE single virtual switch, which is by MLAG protocol AUTOMATICALLY, like STP, system MAC, etc. 
• But from upstream L3 devices, you have to configure MANUALLY to ensure one router, so the following info MUST BE SAME:
• VLAN-VNI mapping
• VTEP IP of loopback address
• Flood VTEP list
• And be careful:
• OSPF ID must be manually configured to avoid duplicated to confuse the routing
• On DCS-7050X platform (Trident II ASIC), need a recir interface to process the packets twice in case of VXLAN routing. 

Arista L3 Leaf/Spine Step by Step (2a) - Vxlan Bridging, configuration

Topology:

Configuration:


As long as you can reach the loopback 10 of each VTEPs, the Vxlan bridging configuration is quite straightforward:

wa465.vtep2#sh run int vxlan 1
interface Vxlan1
   vxlan source-interface Loopback10
   vxlan udp-port 4789
   vxlan vlan 2000 vni 2000
   vxlan flood vtep 200.255.200.7 200.255.200.8 200.255.200.9 200.255.200.12 200.255.200.34 200.255.200.56

Basically, you need to specify the following parameters:

• vxlan source interface. (loopback10 in this case)
• vxlan udp port (by default is 4789)
• vlan and vni mapping
• vxlan vtep flood list. 

Please note that you can include the local VTI in the flood list to keep vxlan interface configuration same across the whole domain. 

Ping tests

We can verify the setup by pinging the host address, and GW phy/virtual address. For example, from wa466.host2 (the host behind the bow-tie MLAG), 

ping 20.0.12.250
ping 20.0.12.251

ping 20.0.12.252

ping 20.0.12.253

!

ping 20.0.34.250

ping 20.0.34.251

ping 20.0.34.252

ping 20.0.34.253

!

ping 20.0.56.250

ping 20.0.56.251

ping 20.0.56.252

ping 20.0.56.253
!

ping 20.0.7.250

ping 20.0.7.253

!

ping 20.0.8.250

ping 20.0.8.253

!

ping 20.0.9.250

ping 20.0.9.253 <<<< failed

....

! all successful except 20.0.9.253

--- 20.0.9.250 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 12ms

rtt min/avg/max/mdev = 0.088/0.230/0.577/0.201 ms, ipg/ewma 3.049/0.424 ms

The ping to all host and GW are successful, except the one from wa466.host2 to Vtep3. But the ping from bn303.host3 to vtep3's SVI works. Why?

bn303.spine.host3(s1)(vrf:host3)#ping 20.0.9.253
PING 20.0.9.253 (20.0.9.253) 72(100) bytes of data.
80 bytes from 20.0.9.253: icmp_seq=1 ttl=64 time=0.214 ms
80 bytes from 20.0.9.253: icmp_seq=2 ttl=64 time=0.096 ms
80 bytes from 20.0.9.253: icmp_seq=3 ttl=64 time=0.100 ms
80 bytes from 20.0.9.253: icmp_seq=4 ttl=64 time=0.086 ms
80 bytes from 20.0.9.253: icmp_seq=5 ttl=64 time=0.087 ms

This is because recirculation, which is required for Vxlan routing on T2 platform. We will cover it in next blog. 

Configuration Files

Arista L3 Leaf/Spine Step by Step (1) - Basic Config (BGP/MLAG)

(updated Dec 20, 2018 for more platforms)

Topology:


 

Following my previous post.  In this post, I cover the basic IP, BGP, and MLAG configuration. Here we use eBGP as routing protocol instead of OSPF or ISIS because it is the most common design in the modern data center. You can check the following 2 links for more details. 

• RFC 7938 - BGP Routing in Data Centers, by Petr Lapukhov
• BGP in an Arista Data Center

Basic eBGP configurations:

All BGP peers import ONLY a /32 network NLRI of loopback10, which is VXLAN source address. So no network p2p ethernet NRLI nor "redistribute connect". As a result, the routing table size is very minimal and only ping from local loopback to remote routers' works. 

router bgp 65041
   router-id 200.255.255.12
   maximum-paths 16 ecmp 16
   ....
   network 200.255.200.12/32   !!! loopback10's host addr.
!
interface Loopback10
   ip address 200.255.200.12/32

!!! direct ping doesn't work

ckp342.vtepX2#ping 200.255.200.56
PING 200.255.200.56 (200.255.200.56) 72(100) bytes of data.

--- 200.255.200.56 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

!!! ping source from loopback 10 works

ckp342.vtepX2#ping 200.255.200.56 source 200.255.200.12
PING 200.255.200.56 (200.255.200.56) from 200.255.200.12 : 72(100) bytes of data.
80 bytes from 200.255.200.56: icmp_seq=1 ttl=63 time=0.230 ms
80 bytes from 200.255.200.56: icmp_seq=2 ttl=63 time=0.140 ms
80 bytes from 200.255.200.56: icmp_seq=3 ttl=63 time=0.128 ms
80 bytes from 200.255.200.56: icmp_seq=4 ttl=63 time=0.109 ms
80 bytes from 200.255.200.56: icmp_seq=5 ttl=63 time=0.127 ms

--- 200.255.200.56 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.109/0.146/0.230/0.045 ms, ipg/ewma 0.208/0.186 ms

If multiple leaves share the AS# (not in this setup),  you need "neighbor spine.v4 allowas-in 3" to accept BGP NLRI between each other. 

router bgp 65010

   router-id 10.255.255.11

   neighbor spine.v4 allowas-in 3

eBGP and MLAG configuration

• The mlag configuration here is quite normal. 
• Only one thing worthy to note that, the 2 mlag peers use the same vxlan source address, shown as below

VtepA1: 

interface Loopback10

   ip address 200.255.200.34/32

!

router bgp 65021

   router-id 200.255.255.3

   network 200.255.200.34/32

!

interface Vxlan1

   vxlan source-interface Loopback10

LeafA2:

interface Loopback10

   ip address 200.255.200.34/32

!

router bgp 65021

   router-id 200.255.255.4

   network 200.255.200.34/32

!

interface Vxlan1

   vxlan source-interface Loopback10

Zip file of all routers' "show run"

Arista/EOS: How to trace/debug BGP Update

[2019/9/24] Besides turning on the bgp update trace, anther quick way is to use tcpdump with -vvv option. For example, 

router bgp 65000
   neighbor ebgp.v6 peer group
   neighbor ebgp.v6 remote-as 65020

   neighbor ebgp.v6 maximum-routes 12000
   vrf v3
      bgp default ipv4-unicast transport ipv6
      bgp default ipv6-unicast
      neighbor fe80::20%Et3/12/4 peer group ebgp.v6


But neighbor fe80::20%Et3/12/4 has no incoming prefixes

yo412.mlagA.13:47:59(config-router-bgp)#sh ip bgp summary vrf v3
BGP summary information for VRF v3
Router identifier 100.17.80.251, local AS number 65000
Neighbor Status Codes: m - Under maintenance
  Description              Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc

  v3#1, v46@6L, L3/eth     fe80::20%Et3/12/4 4  65020             22        24    0    0 00:05:27 Estab   0      0

Enable the tcpdump and clear this session in anther session

yo412.mlagA.13:45:36#bash tcpdump -nvvvi et3_12_4 tcp port 179
yo412.mlagA.13:50:00#clear ip bgp fe80::20%Et3/12/4 vrf v3

Now We can see the contents of bgp update packets

13:44:32.751791 44:4c:a8:8d:10:5f > 00:1c:73:44:58:b6, ethertype IPv6 (0x86dd), length 1485: (class 0xc0, flowlabel 0x5ad94, hlim 1, next-header TCP (6) payload length: 1431) fe80::20.38957 > fe80::1.bgp: Flags [P.], seq 85:1484, ack 101, win 225, options [nop,nop,TS val 1506709860 ecr 3429570], length 1399: BGP
Update Message (2), length: 738
  Origin (1), length: 1, Flags [T]: IGP
    0x0000:  00
  AS Path (2), length: 10, Flags [T]: 65020 65000
    0x0000:  0202 0000 fdfc 0000 fde8
  Multi-Protocol Reach NLRI (14), length: 694, Flags [OE]:
    AFI: IPv6 (2), SAFI: Unicast (1)
    nexthop: fe80::20, fe80::20, nh-length: 32, no SNPA
      100:13:22::/64

How to trace/debug BGP updates

This page is to provide a detailed procedure on how to enable BGP trace in gated to debug incoming/outgoing bgp updates. 

Step 1: Find the trace facility

In the gated implementation, the bgp is included in the Ribd agent. So there is no such bgp agent, you have to search the exact trace facility under Rib. 

Arista.7050QX(config)#show trace Rib | grep -i bgp

.....

GatedBgpHelper       enabled  ............

Rib::Bgp             enabled  ............

Rib::Bgp::Keepalive  enabled  ............

Rib::Bgp::Normal     enabled  ............

Rib::Bgp::Notification enabled  ............

Rib::Bgp::Open       enabled  ............

Rib::Bgp::Policy     enabled  ............

Rib::Bgp::Route      enabled  ............

.....

Rib::Bgpwhite::Keepalive enabled  ............

Rib::Bgpwhite::Normal enabled  ............

Rib::Bgpwhite::Notification enabled  ............

Rib::Bgpwhite::Open  enabled  ............

Rib::Bgpwhite::Policy enabled  ............

Rib::Bgpwhite::Route enabled  ............

Rib::Bgpwhite::State enabled  ............

Rib::Bgpwhite::Task  enabled  ............

Rib::Bgpwhite::Timer enabled  ............

Rib::Bgpwhite::Update enabled  ............

....

From above output, there is separate bgp facilities for BGP process under different vrf. In this example, the one I am looking at is Bgp updates under vrf white. 

Step 2: Enable Trace

Arista.7050QX(config)#trace Rib enable Rib::Bgpwhite::Update all

Step 3: Locate the agent log file.  

Arista.7050QX(config)#bash ls -lt /var/log/agents/*white*

-rw-rw-rw- 1 root root 339387 Jun 21 14:15 /var/log/agents/Rib-vrf-white-4032

Step 4: Trigger bgp update and monitor updates

Tail the agent log

Arista.7050QX(config)#bash tail -f /var/log/agents/Rib-vrf-white-4032

Then trigger the bgp clear from another session or other bgp router

Arista.7050QX#clear ipv6 bgp 130:1:10::10 soft vrf white

Here is the output:

....

14:18:08.130842  VR white BGP PACKET: bgp_send_peer_reachability: bnp 130:1:10::1: send path id 4294967295 for prefix 122.130.10.0[apsend=false, entp=edab4460]

14:18:08.130894  VR white BGP PACKET: bgp_rt_send_message: 130:1:10::1, flags 0x5608880, len 108

14:18:08.130947  VR white

14:18:08.130947  VR white BGP SEND 130:1:10::10+49700 -> 130:1:10::1+179

14:18:08.130994  VR white BGP SEND message type 2 (Update) length 108

14:18:08.131190  VR white BGP SEND flags 0x40 code Origin(1): IGP

14:18:08.131231  VR white BGP SEND flags 0x40 code ASPath(2) length 12: 2222

14:18:08.131281  VR white BGP SEND  {1} (0x40 0x03 0x04 0x82)

.....

Step 5: clear the trace

Arista.7050QX(config)#no trace Rib setting

Reference