Arista EOS - hardware counter feature

Since on Sand platform, Ingress/Egress ipv4 ACL, Qos and PBR counters occupy the same counter engineer, so they will not be able to work together. You have to specify which counter could use this shared engineer. 

yo411.mlagB.profA.11:49:40(config)#hardware counter feature ?
  acl            ACL counter feature
  mpls           MPLS LFIB counter feature
  nexthop        Nexthop counter feature
  pdp            PDP counter feature
  subinterface   Subinterface counter feature
  traffic-class  Traffic-class counter feature
  vlan           VLAN counter feature

Arista/EOS, MLAG ipv6 partial traffic loss

Topology (a typical MLAG network):

[mlagA] ======= [mlagB]
    \              /
     +---[Leaf]---+

Symptom:
1) about 10% L3 ipv6 traffic, WE and SN
2) No L2 or L3 ipv4 traffic loss

Get one problem destination - 2000:120:4d:d::1

1) show ipv6 route is good,
bn303.mlagA.profA.16:17:04(config)#sh ipv6 route 2000:120:4d:d::1
 C    2000:120:4d::/48 [0/1]

       via Vlan2077, directly connected

bn302.mlagB.profA.16:17:03(config-if-Vl2199)#sh ipv6 route 2000:120:4d:d::1
 C    2000:120:4d::/48 [0/1]
       via Vlan2077, directly connected

2) show ipv6 route host is NOT right
bn303.mlagA.profA.16:18:20(config)#sh ipv6 route host | grep 2000:120:4d:d
 A  2000:120:4d:d::1 on Vlan2077
 A  2000:120:4d:d::3 on Vlan2077 <<<<<< missing ::2
 A  2000:120:4d:d::4 on Vlan2077

bn302.mlagB.profA.16:18:20(config-if-Vl2199)#sh ipv6 route host | grep 2000:120:4d:d
 A  2000:120:4d:d::2 on Vlan2077 <<<<<< missing ::1
 A  2000:120:4d:d::3 on Vlan2077
 A  2000:120:4d:d::4 on Vlan2077

Root cause: 
missing configuration - "ip virtual-router mac-address mlag-peer"

Arista EOS Debug Tips on Traffic Loss

1. check what/how many pkts punted to CPU?

- "show cpu counter queue", this is the place we see the software drop

- output is by switch ASIC and cpu queues

- CpuQueueL3DstMiss: pkt destinated to unknown address, like unARP'ed host address

- CpuQueueL3LpmOverflow - ?

2. hardward drop

- show hardware counter drop

3. show platform fap interrup

- check hw interrupts

4. Check drops on which interfaces

- show interface counter discard | nz

Arista EOS: %QOS-3-POLICY_HW_RESOURCE_FULL

When applying Qos policy under port-channel, the system doesn't accept it. 

wa463.bug228215.16:12:21(config-if-Po20)#service-policy type qos input SPTest
% Error: Cannot apply service-policy to Port-Channel20 ()

And show logg displays an error msg:
Mar  8 16:01:25 wa463 SandAcl: %QOS-3-POLICY_HW_RESOURCE_FULL: Insufficient hardware resources to program the input policy-map SPTest.

It is because the TCAM is running out. One possible reason is PDP (per-port data policy) which uses up quite some TCAM. So try the EOS-Int image. 

wa462.bug228215.16:17:04#sh platform jericho acl tcam summary
The total number of TCAM lines per Jericho bank is 2048

========================================================
Jericho0:
========================================================
   Bank   Used           Used %          Used By
      0   2046               99         IP RACLs
      1   1554               75         IP RACLs
   2, 3   2048              100       IPv6 RACLs
   4, 5   2048              100       IPv6 RACLs
   6, 7   2048              100       IPv6 RACLs
   8, 9   2046               99       IPv6 RACLs
  10,11    210               10       IPv6 RACLs
     14     79               61 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 20479

After changing to INT image, this issue is gone. 

wa462.bug228215.16:28:09(config)#int po20
wa462.bug228215.16:28:11(config-if-Po20)#service-policy type qos input SPTest
wa462.bug228215.16:28:13(config-if-Po20)#show ver
Arista DCS-7280CR-48-F
Hardware version:    11.01
Serial number:       JPE16473148
System MAC address:  444c.a897.8c51

Software image version: 4.20.0F-INT-7767198.bloomingtonrel (engineering build)

iptables in EOS

iptables is a Linux firewall utility program, which is leveraged by Arista EOS to control protocol control packets. For example:

Example: sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
-S : List the rules
-A : Append rule
-p : protocol
-j : jump = action

[admin@ck478 ~]$ sudo iptables -S
.....
-A INPUT -p tcp -m tcp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE   <<< MLAG control pkt
-A INPUT -p udp -m udp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE

! add a bgp neighbor 1.1.1.1
ck478.17:51:26(config)#router bgp 65500
ck478.17:51:43(config-router-bgp)#nei 1.1.1.1 remote 65001

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -j ACCEPT   <<< a new rule added for bgp nei 1.1.1.1

! configure bgp ttl security rule
ck478.17:51:51(config-router-bgp)#nei 1.1.1.1 ttl maximum-hops 2

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -m ttl --ttl-lt 253 -j DROP <<< all bgp pkts w/ ttl <253 droped!
-A BGP -s 1.1.1.1/32 -j ACCEPT

strace in EOS

"strace" is a powerful linux debug command and it can be used on Arista EOS. 

Some useful arguments:
-c -- count time and calls
-p pid -- trace process with pid#
-T -- print time spent

Example:
[admin@bn303 ~]$ ps -ef | grep Bgp
root     14926  2044  1 09:55 ?        00:00:09 Bgp

[admin@bn303 ~]$ sudo strace -c -p 14926
Process 14926 attached
^CProcess 14926 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0         1           ioctl
  -nan    0.000000           0         3         3 stat64
  -nan    0.000000           0        46           epoll_ctl
  -nan    0.000000           0        45           epoll_wait
  -nan    0.000000           0         4           send
  -nan    0.000000           0         1           sendmsg
  -nan    0.000000           0         1           recvmsg
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   101         3 total

Arista EOS 4.21 : L2 subinterface

Topology:

[rtr3]
  |
  | et2
  |    eth1.1
[rtr1]==============[rtr2]
  |    eth1.2
  | eth3
  |
[rtr4]

Configuration

interface et1.100
  encap dot1q vlan 100
  vlan id 1000
interface et1.200
  encap dot1q vlan 200
  vlan id 1000
interface eth2
  switch access vlan 1000
interface eth3
  switch mode trunk

L2 bridge domain

• Above configuration is to create a L2 bridge domain of 4 end points - et1.100, et1.200, eth2 (access), eth3 (trunk). 
• 1 bcast from et1.100, will be flood to et1.200, et2, et3
• Vlan mapping is different, which is 1:1 relation. So no vlan 1000 on both et1.100 and et1.200
• et1 must be "no switch". 
• Feature is supported from 4.21.*?

Arista Eos 4.21: BGP DSCP Configuration

By default, the BGP packets' DSCP value is 0x0. Now customer wants to have it configurable. A global value is enough. New configuration/change doesn't tear down existing sessions. This feature is started from 4.21.*?

The configuration is quite simple:

router bgp 1
  bgp transport qos dscp 48

Arista EOS CLIs

Platform Dependent

• Trident hw programming
• show platform trident counter | egrep 'card|drop' | nz
• show platform trident tcam detail | grep -i "LAG E" -A2
• show platform trident L3 shadow my-station
• Trident agent logs:
• qtcat strataL3.qt | grep <prefix> 

Platform Independent:

• evpn:
• show l2rib input bgp
• System:
• show event-monitor mac

Software:

• CVP:
• management api http-commands
•   no shutdown
•   user cvpadmin privilege 15 role admin secret eosuper

Ixia: Custom view of traffic

Custom view:
Filters: 
select Traffic -》flow detective
Traffic Item Equals "Traffic Name"
Show 50 "Worst Performers"

Then select Stat

"ip directed-broadcast" in VxLAN

1. What's the feature of "ip directed-broadcast" for?

• One application is Wake-on-Lan(WOL). A host device like PC can be powered on/resumed remotely. 
• Need hw/BIOS support. When host receives a WOL magic packet, it turns on. 
• Enable it in OS, linux - "sudo ethtool -s eth0 wol g"
• So server (20.1.1.1) sends a bcast packet destined to remote subnet like 10.1.1.255 to 10.1.1.0/24 network. 
• With this feature enabled under SVI, this bcast pkt will be fwded to remote subnet like a ucast pkt. 
• By default is disabled, because of the security concern. 
• This is a legacy feature starting from 2011/12?

2. Configuration and details
interface vlan 2001
  ip directed-broadcast

Says the topology with Vxlan is like

vlan 1001
10.1.1.1/24 [host2]-----+
                        |
vlan 2001               |
20.1.1.1/24 [host1]---[l2vtep]---[l2vtep]---[gw of svi1001/2001]

* host1 sends pkts dstIp=10.1.1.255
* pkts follows vxlan to reach gw
* gw routes pkts back l2vtep as a ucast pkt

3. Caveats

• Arista EOS has a bug - 217001, when the vlan is included vxlan interface, the directed-bcast traffic will be shaped by PDP. 

yr252.23:14:18#sh pl trident l3 software host-table | beg 10.50.51.255

   Entry: 0, HwEntry: 0x6002a50, Type: v4Uc, Vrf:  0, Host: 10.50.51.255/32

Bucket: 687, state: 0x00011

VXLAN Routing with MLAG

VXLAN Routing with MLAG
https://eos.arista.com/vxlan-routing-with-mlag/

• VXLAN routing routes pkt based on IP address in inner header, not outer VXLAN header. 
• 有个示意图，可以帮助理解
• 在SW-1其实就是简单的Inter VLAN SVI routing
• 只是SVI-VLAN 20有个VTEP／VNI，所以可以Learn到DEST MAC of Svr2
• Srv-2‘s GW SVI-VLAN-20 is on VTEP-1/SW-1
• VNI 1020 链接左右的L2 Domain
• Routing Topologies
• Direct Routing - routing at 1st-hop leaf node for ALL subnets. 
• Indirect Routing - only route for ONE subnet, reduce amt of ARP/MAC resource on leaf
• Indirect is a derivative of direct
• Direct Routing
• works by creating anycast IP address:
• Leaf acts as GW, owns and responds ARP req
• 所有Leaf Config same "ip address virtual" and "ip virtual mac"
• ip address virtual 10.10.10.254/24
• No routing over an VLAN interface w/ "ip address virtual"
• VTEP w/ "ip address virtual" will fwd any ARP responses to virtual router MAC to all neighbor VTEPs via HER(head-replication). So neighbor VTEPs host same ARP tables. 
• in MLAG, ARP res to "virtual ip addr" are sync'ed with MLAG peer. 
• Note: ARP sync between MLAG is done via VXLAN agent, hence "ip virtual address" is ONLY supported with VXLAN config
• virtual VTEP:
• 每个Leaf都有Virtual IP addr + MAC, 所以都可以response ARP req. 
• 所以建立一个vVTEP. 不太明白
• ARP Timer
• Serv1 sends ARP req to VTEP1. By routing, VTEP1 would learn MAC of Serv4 via initial ARP req. 
• But not via subsequent bi-directional data traffic, because returning traffic could be ECMP'd to VTEP2, which also routes and rewrite SrcMAc of  inner pkt by VTEP2 mac. 
• To avoid MAC being flush (default timeout is 5 min), it is advised to config ARP aging timeout (default 4 hours) less than MAC timeout. 
• So force a ARP refresh and re-learning MAC. 
• Direct Routing Config:
• VTEP only needs to announce its loopback/end-point into BGP. 
• Then tenant subnets exit only on the leafs, NOT in BGP or on spines. 
• show vxlan address-table
• show mac address-table

VXLAN (2) - RFC 7348

• 明白一个概念, Overlay - overlay L2 connectivity over L3 network
• Inter-VM 需要L2 access mode
• 但是DC Infra都是 L3/IP, 因为ECMP，
• 所以Overlay = provide L2 network over L3 infra
• Bcast/Unknown traffic via Mcast
• 这个在EOS里面没有implement，客户不喜欢this approach; 
• 现在就是简单Flood, 所以Mcast/Bcast/Unknown traffic会被复制多份
• Pkt @ IP/UDP (dest port 4789)

VXLAN Bridging with MLAG

VXLAN Bridging with MLAG

• Key takeaways:
• FH VTEP Encap/Decap
• Routing between MLAG peers
• MLAG peers share the same loopback/VTI address
• https://eos.arista.com/vxlan-with-mlag-configuration-guide/
• Provides remote L2 connectivity between racks or DC;
• Each MLAG domain(2 MLAG peers) has ONE logical VTEP
• Same virtual tunnel ip address (VTI)
• 因为两个MLAG Peer work as ONE physical switch
• MAC Sync:
• For encap/decap traffic, both local and remote MAC address need to be sync'ed between peers via peer-link
• remote = remote MAC associated with remote VTEP ip address. 

Configuration (same on both Mlag peers)

interface loopback1
 ip address 192.168.0.1/32
interface vxlan1
 vlan source-interface loopback 1
 vxlan udp-port 4789
 vxlan vlan 10 vni 10
 vxlan vlan 10 flood 192.168.0.2

MAC, ARP, Traffic例子

• serverA (macA) under MLAG domain 1 (Peer1A和1B)，比方说VLAN 10，sends ARP request
• ARP Req 会被Hash over 1 link of 2-port LAG. 
• Peer1A受到这个ARP req, 有4个Actions
• Act#1: peer1A floods this ARP Req所有本地VLAN 10的端口，因为是Bcast Pkt
• Act#2: peer1A floods it to peer1B，这是给peer1B上面的Singly端口
• peer1B只会flood singly ports，而不会flood dual-home ports
• Act#3: peer1A sync with 1B，peer1B知道 macA 是在Port-channel上面
• 这个Sync是另外的 MLAG signaling, 
• Act#4: peer1A ENCAP ARP in VXLAN and floods all VTEP
• FH GW 负责encap/decap vxlan traffic
• VXLAN pkt is ECMP'ed to spine then to remote Peer2A/B，
• Pear2A/B 和 1A/B一样，share一个VTI address，所以逻辑上是一个
• peer1A ECMP to one spine;
• This spine 有2个path to VTEP 192.168.0.2, 比方说ECMP to peer2A
• Peer2A首到ARP req, DECAP VXLAN pkt and learns MACa in from VTEP 192.168.0.1, 以下是标准的MLAG流程 和 Peer1A很类似
• Act#1: Peer2A flood ARP req all local ports
• Act#2: Peer2A flood it via peer-link for those singly ports on Peer2B
• Act#3: Peer2A sync‘s with Peer2B, MACa from VTEP 192.168.0.1
• peerRemoteDynamic
• ServerB unicasts ARP response to ServerA
• dstMAC = MAC.AAA; srcMAC = MAC.BBB
• Now both peer2A/2B know MAC.AAA is on VTEP 192.168.0.1, and ARP response is encap into VXLAN and routed to peer1B
• ENCAP ARP reply on FH device
• 如果peer1B 收到这个ARP reply, 
• Learns MAC.BBB from VTEP 192.168.0.2, remoteDynamic;
• sync with peer1A
• and pkt fwd down to port-ch 10

Useful CLIs:

• show mac address, Ports里面有Vx1
• show vxlan address-table, 有Mac/Vtep/Port

Switch over 例子，例如Peer1A lose all uplinks

• First Hop MLAG Peer/VTEP，负责encap/decap pkts, 这个是Principle
• 例如MLAG peer1A lost all uplinks，但是device is up running
• 还是Peer1A encap/decap pkts
• 需要Routing between peer via Peerlink
• Best Practise是建议routing on a dedicated VLAN而不是Peerlink VLAN 

Arista EOS - "ip virtual mac-address mlag-peer"

Say, in a mlag environment, 

• The hosts can not understand the vMAC in the ARP packets. For example, some F5 and Netapp devices only check the srcMAC of ARP reply, instead of the srcHwMAC inside the ARP.
• These host hashes the traffic with gateway's system mac to 2 mlag peers. 
• When mlagPeer1 receives packets with dstMAC = mlagPeer2 system MAC, it should forward it to peer2 via peerlink, not good, a totally waste of peerlink
• We can configure "ip virtual-router mac-address mlag-peer", which enables the peer to consume packets destined to peer and route them directly. 

psp111.14:56:21#sh platform trident l3  shadow my-station

My Station Tcam:

--------------------------------------------------------------------------------

Id    Vlan/Mask                                  Mac/Mask       VVVVMACD         T/       ModId/    IngPort/

                                                                4646PRPS       Mask         Mask        Mask

                                                                UUMMLPUC

                                                                CCCCS  D

7         0/0x0       44:4c:a8:93:22:9b/ff:ff:ff:ff:ff:ff       00000000        0/1        0/0x0      0/0x7f

8         0/0x0       44:4c:a8:93:22:9b/ff:ff:ff:ff:ff:ff       11001000        0/0        0/0x0       0/0x0

9         0/0x0       01:00:5e:00:00:00/ff:ff:ff:00:00:00       00000100        0/0        0/0x0       0/0x0

10        0/0x0       44:4c:a8:93:29:d5/ff:ff:ff:ff:ff:ff       11001100        0/0        0/0x0       0/0x0

11        0/0x0       00:dc:00:02:00:01/ff:ff:ff:ff:ff:ff       11001100        0/0        0/0x0       0/0x0

Arista EOS - BGP maintenance mode

Basically the BGP maintenance mode on Eos is an implementation of BGP G-SHUT in RFC 8326. The mechanism is quite simple and effective:

• Add an outbound policy to attach GSHUT community to all prefixes, and it triggers a re-advertisement;
• Add an inbound policy to set LOCAL_PRF = 0 to all incoming prefixes. 
• Wait bgp convergence then shut bgp session

The issue to be solved here is: if backup path is hidden by RR or nodes of an AS, it will trigger relearn routes and put them effective. 

Configuration: (system-level in Arista)

config

maintenance

   unit System

      profile unit System

install source scp:solomon@server/export/images/EOS.swi destination flash:

copy runn start

quiese

reload now force

show ip bgp summary

show ip bgp 0.0.0.0/0 detail (in any leaf to verify GSHUT)

show output (before quiesce)

!! Gshut initiator (dut to reload)

ck421.15:08:47(config-builtin-unit-System)#sh ip bgp neighbors | egrep '^BGP|Updates:'
BGP neighbor is 100.1.11.1, remote AS 65110, external link
                         Sent      Rcvd
    Updates:             2810      2574

!! BGP neighbor

pts321.15:06:07(config)#sh ip bgp 2.2.2.2/32 detail
pts321.15:06:14#sh ip bgp 2.2.2.2/32 det
BGP routing table information for VRF default
Router identifier 100.1.11.1, local AS number 65110
BGP routing table entry for 2.2.2.2/32
 Paths: 5 available
....
  65100 65120
    192.1.0.0 from 192.1.0.0 (169.169.169.1) <<< Initiator 
      Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP, ECMP contributor
      Not best: ECMP-Fast configured

show output (after quiesce)

ck421.15:11:28(config-builtin-unit-System)#sh ip bgp sum
BGP summary information for VRF default
Router identifier 169.169.169.1, local AS number 65100
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
m 192.1.0.1        4  65110            100       103    0    0 01:15:55 Estab  11     11
m 192.1.0.3        4  65110            100       104    0    0 01:15:55 Estab  11     11

ck421.15:11:13(config-builtin-unit-System)#sh ip bgp neighbors | egrep '^BGP|Updates:|Sent.*Rcvd'
BGP neighbor is 100.1.11.1, remote AS 65110, external link
                         Sent      Rcvd
    Updates:             4233      2574  <<< 2810 vs 4233 (resend)

pts321.15:06:20#sh ip bgp 2.2.2.2/32 det
BGP routing table information for VRF default
Router identifier 100.1.11.1, local AS number 65110
BGP routing table entry for 2.2.2.2/32
 Paths: 5 available
....
  65100 65120
    192.1.0.0 from 192.1.0.0 (169.169.169.1)
      Origin IGP, metric -, localpref 0, weight 0, valid, external
      Not best: Local preference
      Community: GSHUT <<<<< 

Arista EOS: tcpdump the VRF interface

Use the Linux name space: 

[admin@dc7050 ~]$ sudo ip netns exec ns-<vrf> tcpdump -i vlan2101 arp

Arista: BGP neighbor next-hop-unchanged doesn't work in gated mode

A bit background, in Arista EOS, there is 2 implementations of BGP process. Default is gated, and late one is multi-agent, which can be enabled via cli - service routing protocols model multi-agent. 

BGP neighbor next-hop-unchanged is only supported in multi-agent mode, not in gated. This feature works in route-map with both modes. 

Arista - traffic disruption during LAG reprogramming

No traffic disruption is expected during LAG reprogramming like adding/removing member ports. There is only one exception - on Sand(Arad/Jericho) platforms, if software Lag is enabled, traffic loss will be seen when # of Lag is changed from 1 to 2,  and vice versa. And software Lag can be disabled by knob - "platform sand lag hardware-only".

Arista J/J+ FlexRoute with Internet Routing Table

https://www.arista.com/assets/data/pdf/Whitepapers/FlexRoute-WP.pdf

• CLI: 
• ip hardware fib optimize prefix-length internet
• show ip bgp summary
• show ip route summary
• show hardware capacity | grep Routing
• Routing HW resources are sliced to fit the route distribution of REAL internet routing table
• If loading a lab configuration like Ixia, you mean see the error message like - "%CAPACITY-1-UTILIZATION_HIGH. LEM table utilization is currently at 91%, crossed threshold 90%"
• For example, if too many /24, it will be moved to LEM
• And another important concept of prefix, route and FIB:
• 3 x internet feeds = 3 x 600K prefixes
• 3 prefixes of same dest = 1 route with 3 NH if ECMP
• 1 route = 1 FIB entry

IP fragmentation on Arad platform

• Fragmented/MTU violated packet is sw forwarded and shaped by CoPP or PDP;
• CoPP = 2~2.5 Mbps
• "no shape" under copp-system-mtu can make it 100Mbps
• PDP = 100 Mbps
• But not a good idea to do it. 

TH/Trident low LPM

On Arista DCS-7260CX-64-F platform, if you see hw resource - LPM/V4Routes running out, have to change the "platform trident forward partition". Agent will restart but doesn't a reboot. 

nv475.08:45:07(config)#sh hardware capacity utilization percent exceed 50
Forwarding Resources Usage

Table   Feature    Chip       Used   Used      Free    Committed    Best Case        High
                           Entries    (%)   Entries      Entries          Max   Watermark
                                                                      Entries
------- ---------- ------ --------- ------ ---------- ------------ ------------ ---------
LPM                           8190   100%         0            0         8190        8190
LPM     V4Routes              8190   100%         0            0         8190        8190

nv475.08:52:45(config)#sh ip route 151.131.34.0

VRF: default
======================================================
WARNING: Some of the routes are not programmed in
hardware, and they are marked with '*'.
======================================================

*B E    151.131.34.0/24 [200/0] via 100.1.0.44, Vlan10

nv476.11:05:47#sh platform trident forwarding-table partition

L2 Table Size: 72k

L3 Host Table Size: 72k

LPM Table Size: 16k, uRPF is disabled

nv475.09:27:35(config)#platform trident forwarding-table partition ?

  0  136k l2 entries, 8k l3 host, 16k lpm entries

  1  104k l2 entries, 40k l3 host, 16k lpm entries

  2  72k l2 entries, 72k l3 host, 16k lpm entries

  3  40k l2 entries, 104k l3 host, 16k lpm entries

  4  8k l2 entries, 8k l3 host, 90k lpm entries <<< this one!

But this change will impact MLAG because lower MAC capacity. 

Take away notes from RFC5549 video by AMX-IX

• https://www.youtube.com/watch?v=uJOtfiHDCMw
• AMS-IX running out /22 ipv4 NH address space
• Solution #1: more address space:
• ARP broadcast too much
• Solution #2: private address by RFC1918, 10/8, 172.16/12, 192.168/16
• Space used by ISP customer
• More specific in IGP
• Break traceroute
• Solution #5: RFC5549
• IPv4 NLRI over IPv6 NH
• 2 ways to implement it:
• v4 in v6 tunnel, add 40B head
• direct forwarding - just replace the MAC, no change;

MLAG Fast Convergence - MAC Redirection/Promption

https://eos.arista.com/eos-4-18-0f/mlag-unicast-convergence/

Problem Description:

Consider the following setup 

• mlagA and mlagB are 2 mlag peers with port-ch 2000;
• The host MAC - 0000:1111:2222 is learnt on MLAG 10 and A is the owner. 
• In another way, MAC 0000:1111:2222 is A's local MAC and B's remote
• A bit background:
• The MAC address and ARP information are all sync'ed during boot-up;
• After that, only MAC table is sync'ed, for example, A tells B that MAC a.b.c is from mlag po10, or singly interface, or remote vtep. 
• So in the MAC table, the MAC has at least 4 states:
• learnedDynamic (local mlag), 
• peerDynamic (remote mlag), 
• learnedRemoteDynamic (vxlan) 
• peerRemoteDynamic (remote vxlan)
• Get this information by command - show mac address mlag-peer

Now saying we have 

• link failure, which has 2 loss: down and up
• when B's po 10 is down, all MAC are re-programmed from po 10 to po 2000, so be MAC move is done one by one in old releases before 4.18.1F. (#1)
• when B's po1 10 is back up, the ACL on peerLink to block BUM traffic immediately to break L2 loop while the MAC move needs time to be completed. (#2)
• node failure, which introduces 3 loss, down, up and delay timeout
• node down, 100s msec loss depending on scale
• node up, 100s msec loss (#3)
• When the peerlink is up, why? At this time, peerB has no uplink or downlink up, which are all in reload-delay. 
• Remember the MAC sync mentioned above? A needs to sync up with B on the MAC, so on A these MAC learnt from B will be flushed!!
• A has to relearn these MAC, 50% all of sudden. Still ok for locally switched packets because the hw flooding kicks in. 
• But bad for Vxlan, which requires software flood for the head-end-replication. 
• reload-delay timeout, 100s msec, actually 2 times
• Need to have iBGP or IGP L3 routing between 2 peers.
• Since peerlink is up fast and much earlier than mlag/non-mlag interface. So when non-mlag or mlag interfaces are up, they can send the traffic to peer link before the optimal path converged. 

Feature and solution

So from 4.18F, a feature called MLAG fast MAC redirection is developed to address above issues. This feature has 2 aspects:

1. MAC redirect, for #1 and #2 loss

• With this feature, the the interface attribute of impacted MACs still point to MLAG po 10 in host table. So, no move at all. 
• Strata and Sand implementation are slightly different but same idea. 
• On Sand, it is to use a recirc channel on each Arad/Jericho chip to recycle the MLAG destined packets over to peer-link. 
• Requirements and limitation:
• Peerlink must be a LAG not Ethernet on Strata. 
• On Sand "platform sand lag hardware-only" must be enabled, I believe only hw LAG can share member port - the recirc channel. 
• MLAG ASU2 cannot co-exist. 

2. MAC address promotion, targeting #3. 

• When 1) peer reboot; 2) hitful restart of fwding plane, the remote MACs (learnt via peer switch) will be flushed, which cause
• 1) before the MAC is re-learnt, packets needs to flooded. Still ok in pure L2/L3 environment because done by hw;
• 2) software forward of Vxlan packets. That's a big issue because it results in drops by CoPP. 
• Why the MACs are flushed?
• when peerB is up, MAC are sync'ed from A to B. These remote MAC are flushed. No MAC, then flooding. 
• Solution: 
• when peerB is down, the ownership of MAC are transfered to peerA
• when peerB recovers, *ALL* MAC are sync'ed from A to B
• Details:
• when peerB is down, peerA enters failover state;
• peerDynamic, peerLearnedRemote to learnedDynamic, learnedRemoteDynamic
• NOT single-leg host

Misc:

• From AD1554:
• If all uplinks are L3 interfaces, then it is preferable to keep non-mlag reload-delay timer < mlag timer, so
• Upstream/L3 up first before downstream/L2;
• In this way, S-N traffic should be no loss. 
• Enabled "reload-delay mode lacp standby", need to have non-mlag timre >= mlag timer,
• MLAG interfaces with LACP are kept warm for LAG membership table, MAC table programming. 
• But upstream/L3 must be up after L2/downstream/mlag interfaces, otherwise S/N traffic are blackhole'd. 
• From AD3152:
• SandL3Unicast - managing NH and ensuring EEDB no change
• SandACL - programming DROP ACL on the peerLinkRecircPort avoid pkts from peerLink back to peerLink
• Assigning LagMemberID is interesting, this peerLinkRecircPort needs a member id. what about overflown?
• LAG member, C/D bit
• C = collecting, D = Distributing
• if static LAG, C/D=True, added to LAG
• if LACP enabled, only C=True, D=True, member can be added. 
• peerLinkRecircPort is always C/D=False/True
• LC removal event
• If all ports on this LC, 
• L3 will have some downtime since losing all ARP entries; 
• L2 should be fine once recirc port is programmed. 
• If at least 1 member on another LC, L2/L3 should be fine
• This is quite complicated! 
• 3/1, 3/36, 4/1 are local member of mlag Po 10
• all 3 ports down, all 3 members retained with C/D=False
• PeerLinkRecicFap is added from either 3/0, 3/2 or 4/0 
• If LC3 is pulled, peerLinkRecirFap 4/0 is added. 
• And a lot of combination of events, LAG config change, member port down...
• 4 events:
• LC removal
• LAG config change: unconfig and change config
• Member port down or cable unplug
• port-ch shutdown
• CLIs:
• show plat trident counter int e27/1
• Drops counts in Vlan boundary = Vlan ID missed. In another word, the VLAN id programming on this interface is not done yet. 

Reference:

• PeerOne Vxlan + MLAG
• AD3398, BG141435/96642
• AD3152
• AD1554

Arista EOS - %HARDWARE-3-DROP_COUNTER_ALERT, DchUnreachables

What does the following log message mean?

May 15 15:45:42 bn302 EventMgr: %HARDWARE-3-DROP_COUNTER_ALERT: Persistent Internal Drop 'DchUnreachables': 792271126642 detected on Fe3600-3/1

And there is high # of drop in the output of "sh hardware counter drop"

bn302.jhm.mlagB.profA0.w.15:47:58#show hardware counter drop
Summary:
Total Adverse (A) Drops: 20098028670058
Total Congestion (C) Drops: 0
Total Packet Processor (P) Drops: 72029
Type  Chip         CounterName                    :           Count : First Occurrence    : Last Occurrence
--------------------------------------------------------------------------------------------------------------
A     Fe3600-4/2   DchUnreachables-1              :    103285456899 : 2018-05-15 15:34:55 : 2018-05-15 15:35:07

Basically when the LCs are not ready and up, the Fabric modules don't know where to forward the packets, they will drop them and increase this counter. It is quite common during system booting up. 

How to use python/yaml

import yaml
import sys

with open(sys.argv[1], 'r') as stream:
   try:
      paramDict = yaml.load(stream)
   except yaml.YAMLError as err:
      print(err)
      sys.ext(0)

print paramDict['Devices']['Switch']['mlagA']['Mgmt']['Host']

============
Devices:
   Switch:
      mlagA:                                 # MLAG Dut1
         Mgmt:                               
            Host: bn303
            AccessMethod:    ssh             # ssh or capi
            SshUsername:     admin           # default admin
            ChassisType:     modular         # fixed or modular
            CapiProtocol:    https

~/py @arst1.sjc> python test.py test.yaml

bn303

Arista L3 Leaf/Spine Step by Step (2d) - VxLAN Bridging, interesting behavior

This post is for the paranoid, who really wants to know how it works under the hood. Based on previous setup, we see 2 interesting behaviors

1. Jericho MLAG peers also receive the arp reply

wa462.vtepB1#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.171267 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.171590 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

wa463.vtepB2#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.172821 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.173117 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

44:4c:a8:97:72:b7 is the MAC address of host2 belonging to vtep2. Why these 2 peers receive this unicast packet? The reason is that, when the VtepX1/X2 the first packet from remote Vtep, it will need some time to create a hw tunnel. During this gap, Vtep has to flood this ucast packet to all Vteps. 

When Vtep has no remote Vxlan MAC

ckp341.vtepX1#sh mac address-table dynamic interface vxlan 1
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
Total Mac Addresses for this criterion: 0

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

So Vtep has no remote vtep

ckp341.vtepX1#show vxlan vtep
Remote VTEPS for Vxlan1:
Total number of remote VTEPS:  0

Let's create a static MAC pointing to remote VTEP. After there is a remote Vtep. 

ckp341.vtepX1(config)#mac address-table static 0000.1111.2222 vlan 2000 interface vxlan 1 vtep 200.255.200.8

ckp341.vtepX1(config)#show vxlan vtep

Remote VTEPS for Vxlan1:

200.255.200.8

Total number of remote VTEPS:  1

Repeat the ping. Now the Mlag peer doesn't receive the unicast packet. 

wa462.vtepB1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

22:22:16.623983 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

2. ONLY 1 T2+ MLAG peer see receive the arp request, and no arp reply.

vtep1 has no arp req

snp262.vtepA1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

^C

0 packets captured

vtepA2 sees arp req

snp263.vtepA2(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

06:27:56.665383 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

^C

1 packet captured

This is because Trident family ASIC pipeline can't handle vxlan decap and routing in 1 pass. The arp req seen on vtepA2 is flooded by vtepA1. From the output of "show mac address mlag-peer", you can see vtepA2 learning this MAC from mlag-peer.

snp263.vtepA2(config)#sh mac address-table mlag-peer

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

2000    0000.1111.2222    STATIC      Vx1

2000    001c.73ff.405e    STATIC      Po2000

2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:25 ago

4094    001c.73ff.405e    STATIC      Po2000

Total Mac Addresses for this criterion: 4