Arista EOS/Networking Tips by Solomon Yang: Hardware

Showing posts with label Hardware. Show all posts

12/03/2021

Arista EOS - CoppSystemEgressTrap drop

On Arista 7280*R* series routers, if you are seeing high # of CoppSystemEgressTrap drop in the output of "show cpu counters queue", the most likely reason is that, there are some packets size are larger than the egress interface configured size.

smv550.08:42:39#show cpu counters queue summary | nz
Aggregate CPU counters:
This indicates the sum of port-dependent queue counters across all input ports.
CoPP Class Pkts Octets DropPkts DropOctets
Aggregate
-------------------------------------------------------------------------------------------------------
CoppSystemBgp 14358 1141461 0 0
CoppSystemEgressTrap 10688 97079104 228017 2071078411

4/25/2020

ECMP in MPLS L3 VPN

Consider the above topology

CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1.

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:1

Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:4364

Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012

via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name VRF ID Table ID

cust_1 15 65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 116707

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries.

PE1#show hardware capacity | grep -i ECMP

ECMP 875 21% 3220 0 4095 875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

1.1.1.1 4 65000 1693 1673 0 0 23:25:15 Estab 1602 1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage.

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP 0 0% 4095 0 4095 875

ECMP Mpls 0 0% 4095 0 4095 0

ECMP Routing 0 0% 4095 0 4095 875

ECMP VxlanOverlay 0 0% 4095 0 4095 0

ECMP VxlanTunnel 0 0% 3891 0 3891 0

Now no ECMP anymore in the sw/hw routing table.

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32830 |0 | - | | - |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels.

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

192.158.115.11 4 65100 1568 1819 0 0 1d00h Estab 800 800

192.168.115.11 4 65100 1749 1827 0 0 1d00h Estab 802 802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels.

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

Local Label: 16694 (with rewrite);

Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

Local Label: 16695 (with rewrite);

Received Label 116390

4/24/2020

Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted.

Feb 4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
Paths: 2 available
65000 65101
192.168.66.7 from 192.168.0.7 (192.168.0.7)
Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Extended Community: Route-Target-AS:65000:1
Remote MPLS label: 116396
Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label | Action | Destination | VID |Outlif | MAC / CPU Code |Index| Index| Action
|186827 |Forward | DROP |0 | - | |16434|358104| -
|186827 |Forward | DROP |0 | - | |16434|358105| -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 93206 99% 1002 4096 98304 93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
5.5.5.5 4 65000 421 405 0 0 05:20:01 Estab 801 801
192.158.115.11 4 8073 381 439 0 0 05:30:45 Estab 800 800
192.168.115.11 4 8073 410 439 0 0 05:29:12 Estab 802 802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels.

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 0 0% 77824 0 81920 0

Here is a list of the Jericho-family models:
7280R/Jericho : 80K
7280R2/Jericho+: 96K

7/18/2019

Arista Modular Switch - simulate LC remove/insert

!! bash cat /persist/sys/Linecard3.cfg
platform module Linecard3 remove
platform module Linecard3 insert

6/28/2019

Arista EOS Tunneling Mechanism (5) - Hw Tunnel

Starting from 4.21.1F, hardware GRE tunnel is supported on Jericho-based platforms. Below is a typical setup of hw tunnel and configuration

You can see, configuration-wise it is just like the software tunnel. How to tell which mode, check "show interface tunnel".

R1#sh int tu200
Tunnel200 is up, line protocol is up (connected)
...
Tunnel TTL 10, Hardware forwarding enabled

Compared with software tunnel, the biggest difference is the hardware programming.

1. TCAM:

R2(config)#sh platform fap tcam summary
Tcam Allocation (Jericho0)
Bank Used By Reserved By
---------- ---------------------------- -----------
0 dbVTT0 -
12 dbSystem -
..

Now configuring an up tunnel interface

R2(config)#int tu100
R2(config-if-Tu100)#tunnel source 100.2.2.2
R2(config-if-Tu100)#tunnel destination 100.3.3.3
R2(config-if-Tu100)#show ip int brief
Interface IP Address Status Protocol MTU
....
Tunnel100 unassigned up up 1476

Now you can see a new Tcam allocation named dbGreTunnel

R2#sh platform fap tcam summary
Tcam Allocation (Jericho0)
Bank Used By Reserved By
---------- ---------------------------- -----------
0 dbVTT0 -
2 dbGreTunnel -
12 dbSystem -

2. Tunnel interface

R1#show platform fap eedb ip-tunnel gre interface tunnel 200
....
| Bank/ | OutLIF | Next | VSI | Encap | TOS | TTL | Source | Destination | OamLIF | OutLIF | Drop |
| Offset | | OutLIF | LSB | Mode | | | IP | IP | Set | Profile | |
|-----------------------------------------------------------------------------------------------------------|
| 2/0 | 0x6000 | 0x4012 | 0 | 2 | 0 | 10 | 100.1.1.1 | 100.4.4.4 | No | 0 | No |

* Bank should match the bank# in "show plat fap tcam summary"

3. Ip route

Below show 220.4.4.0/24 is learned via Tu200

R1(config-if-Tu200)#sh ip route

VRF: default
I L2 220.4.4.0/24 [115/20] via 200.1.4.4, Tunnel200, Static Interface GRE tunnel index 200, dst 100.4.4.4, src 100.1.1.1, TTL 10

R1(config-if-Tu200)#show platform fap ip route | egrep 'VRF|ID|220.4'
|VRF| Destination | | | | | | ECMP| FEC | Tunnel
| ID| Subnet | Cmd | Destination | VID |Outlif | MAC / CPU Code |Index| Index|T Value
|0 |220.4.4.0/24 |ROUTE| FEC 32773 |0 | - | | - |49160 | -

4. FEC

R1(config-if-Tu200)#show platform fap fec all
.....
| | | | | | | |
| ECMP| FEC | | | | | | Tunnel
|Index| Index| Cmd | Destination | VID |Outlif | MAC / CPU Code |T Value
----------------------------------------------------------------------------------
.....
| - |49160 |ROUTE| FEC 32773 |0 | - | | -
| - |32773 |ROUTE| Et21 |12 | - | 44:4c:a8:c1:78:69 |T 100.4.4.4

Limitations:

In above TOI link, there is a list of hw tunnel limitations, for example:

Underlay endpoint (tunnel source/destination) must IPv4 address in default VRF.

Even there is tunnel source/destination ipv6 address option, but not supported on this platform

Overlay can be IPv4/IPv6 under default or non-default VRF
No GRE KA, if one side has incorrect GRE configuration like missing tunnel source, the other end of GRE interface is still up.

5/15/2019

Switch goes into power-forever after losing all FANs

If a switch loses all fans, no matter bad hardware or removing parts, the switch should go into a "power-forever" mode. To exit this hardware protection, you have to unplug ALL power cords and reinsert them. This will reset the hardware logic. Unplug then plug power cord one by one doesn't work, neither the power flap.

5/14/2019

Shutdown the fabric link reporting CRC error

If you are seeing some SerdesCrcError in the output of "show hardware counter drop", that means you probably have some bad fabric hardware.

R1.09:39:22#sh hardware counter drop | grep Crc
A Fe3600-1/1 SerdesCrcErrors-18 : 82 : 2019-05-09 16:16:12 : 2019-05-14 09:31:54
A Fe3600-4/1 SerdesCrcErrors-129 : 739 : 2019-05-09 16:32:43 : 2019-05-13 21:02:26
A Jericho11/0 SerdesCrcErrors-43 : 2444 : 2019-05-09 16:03:03 : 2019-05-13 21:02:20
A Jericho5/3 SerdesCrcErrors-8 : 215410 : 2019-05-09 16:02:05 : 2019-05-13 21:02:10

In this case, you can shut down the problematic fabric link to avoid them.

R1.09:42:32(config)#platform Jericho Jericho5/3 serdes 8 shutdown
R1.09:43:11(config)#platform Jericho Jericho11/0 serdes 43 shutdown
R1.09:43:33(config)#platform Fe3600 Fe3600-1/1 serdes 18 shutdown
R1.09:43:53(config)#platform Fe3600 Fe3600-4/1 serdes 129 shutdown

4/30/2019

Arista optics with DOM support

DOM:

Digital Optical Monitoring.
Per industry standard - SFF-8742, and it gives end users the ability to monitor real-time parameters of SFP, such as in/output power, temperature, laser current and supply voltage
2 Arista optical parts don't support full DOM:

40G AOC doesn't support Tx Power
10G AOC doesn't support DOM

EOS CLIs:

show int eth22/1 transceiver [detail]

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html

upp106#show inv | grep SWDM

22 Arista Networks QSFP-100G-SWDM4 ATH173200003 20

upp106#show int eth 22/1 transceiver

If device is externally calibrated, only calibrated values are printed.

N/A: not applicable, Tx: transmit, Rx: receive.

mA: milliamperes, dBm: decibels (milliwatts).

Bias Optical Optical

Temp Voltage Current Tx Power Rx Power

Port (Celsius) (Volts) (mA) (dBm) (dBm) Last Update

----- --------- -------- -------- -------- -------- -------------------

Et22/1 33.00 3.30 7.69 1.32 0.44 0:00:01 ago

upp106#show int eth 22/1 transceiver de

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.

++ : high alarm, + : high warning, - : low warning, -- : low alarm.

A2D readouts (if they differ), are reported in parentheses.

The threshold values are calibrated.