Arista EOS/Networking Tips by Solomon Yang

5/15/2019

Switch goes into power-forever after losing all FANs

If a switch loses all fans, no matter bad hardware or removing parts, the switch should go into a "power-forever" mode. To exit this hardware protection, you have to unplug ALL power cords and reinsert them. This will reset the hardware logic. Unplug then plug power cord one by one doesn't work, neither the power flap.

5/14/2019

Shutdown the fabric link reporting CRC error

If you are seeing some SerdesCrcError in the output of "show hardware counter drop", that means you probably have some bad fabric hardware.

R1.09:39:22#sh hardware counter drop | grep Crc
A Fe3600-1/1 SerdesCrcErrors-18 : 82 : 2019-05-09 16:16:12 : 2019-05-14 09:31:54
A Fe3600-4/1 SerdesCrcErrors-129 : 739 : 2019-05-09 16:32:43 : 2019-05-13 21:02:26
A Jericho11/0 SerdesCrcErrors-43 : 2444 : 2019-05-09 16:03:03 : 2019-05-13 21:02:20
A Jericho5/3 SerdesCrcErrors-8 : 215410 : 2019-05-09 16:02:05 : 2019-05-13 21:02:10

In this case, you can shut down the problematic fabric link to avoid them.

R1.09:42:32(config)#platform Jericho Jericho5/3 serdes 8 shutdown
R1.09:43:11(config)#platform Jericho Jericho11/0 serdes 43 shutdown
R1.09:43:33(config)#platform Fe3600 Fe3600-1/1 serdes 18 shutdown
R1.09:43:53(config)#platform Fe3600 Fe3600-4/1 serdes 129 shutdown

4/30/2019

Arista MLAG reload-delay timers

In this blog post, I will cover the following MLAG related topics:

What is the MLAG reload-delay and why it is needed
The difference of 2 different timers
How to tune the timer values

After an MLAG peer boot up, all its ports are placed in err-disabled state (except the peer-link) with a reason of "mlag-issu". During the reload-delay, the MLAG agents sync all MAC and ARP information with the active peer.

Another trigger of mlag reload delay is forwarding plane agent restart. On some platforms like TH or T3, a port speed change needs a hitful agent restart, which forces mlag interfaces to transition and starts reload-delay.

From 4.15.2F, the default reload-delay timers are different per platform:

All fixed systems: 300 sec
7500* (Arad/Jericho): 1800 sec (due to long hw initialization time)
7300* (Trident*/TH): 1200 sec

When to start the reload timer? The timers start to tick after the start of MLAG agent. In old releases, it is triggered by the sysdb agent.

Can I lower the reload-delay timers? Yes, but be careful since if the interfaces exit err-disabled mode before sync is done, the mlag peer will blackhole the traffic.

Which value should be used, it is highly based on system and configuration. You can look at the log messages closely.

! this is MLAG agent up and timers are on
Apr 19 11:33:32 localhost Mlag: %AGENT-6-INITIALIZED: Agent 'Mlag' initialized; pid=3030
! LCs power on
Apr 19 11:34:08 R1 NorCalCard: %HARDWARE-6-CARD_POWERED_ON: Card Fabric3 has been powered on. model: 7512R-FM rev: 11.02 serial number: JPE16305615
....
Apr 19 11:34:24 R1 NorCalCard: %HARDWARE-6-CARD_POWERED_ON: Card Linecard11 has been powered on. model: 7500R-36Q-LC rev: 02.01 serial number: JPE16204252
! LC initialization
Apr 19 11:38:40 R1 SandFap: %SAND-6-INIT_SUCCEEDED: Initialization of Linecard12 switch asics succeeded.
....
Apr 19 11:43:12 R1 SandFap: %SAND-6-INIT_SUCCEEDED: Initialization of Linecard13 switch asics succeeded.
! interfaces up
Apr 19 11:43:01 R1 Ebra: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet12/26/3 (mlag.207_leaf_et9/3_et10/3=>et12/17/3), changed state to up
....
Apr 19 11:46:33 R1 Ebra: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet14/34/1 (peerLink=>mlagSec.et14/34/1.100g), changed state to up

This is the messages of a 7512N with 8 J/J+ LCs. So you can see, 13 mins after MLAG agent up, the last peer-link interface is up. Given 5 mins to allow IGP/iBGP session and corresponding hw programming finished, a 20-min (1200 sec) reload-delay timer should be safe with an additional 2-minute buffer.

And there are 2 timers:

MLAG reload-delay timer: this is for all MLAG port-channel links. It can be changed by CLI - "reload-delay mlag <seconds>"
Non-MLAG reload-delay timer: most of the time, they are the timeout values for L3 uplinks. And can be modified by CLI - "reload-delay non-mlag <seconds>"

Before discussing how to tune them, let me give a good example to help your understandings:

Imagine there is a house which has a front door (non-MLAG/uplinks to go out) and a back door (MLAG interfaces/to reach hosts/tenants)
And there is a side to your neighbor (MLAG peer), which shares the same tenants/hosts.
So the first thing to do is to communicate with your neighbor to have all the address information, to know who is where.
With all the knowledge, open which door first?

First of all, the side door is opened first. (so peer-link doesn't have reload-delay and a BGP/IGP peering is required)
During the gap of the front and back door, the traffic will not be lost since there is a side door to exit.
The opening of the back door will have half south-north traffic in, and will go thru the side door. Similar to front door, half north-south traffic.
Most of the time, the MLAG interfaces are facing servers which means south-north traffic is much higher than the opposite traffic.

So it is preferable to configure non-mlag reload delay <= mlag timer.
The ONLY exception is, if "reload-delay mode lacp-standby" is enabled, non-mlag timer > mlag timer. This feature keeps the LACP interfaces up to speed up hardware programming. So if the north-south coming first, the router has to drop them because the MLAG port-channels are not really ready.

Arista EOS: sand hardware-only LAG

CLI:
platform sand lag hardware-only

目的：

一般情况下，1-port LAG是software LAG；
如果增加一个port，变成2-port LAG，就会变成hardware LAG，会有traffic interruption，特别是interface flap!
什么意思啊？很简单，如果是2-port LAG，断了一根线，变成1-port/software LAG，有Traffic impact. 如果这根线在Flapping，traffic就一直受影响
所以索性一上来就是hw-LAG
唯一的影响是，占有的资源(啥资源啊？不知道，LAG table?)会多一些

Arista EOS Memory Util%

https://eos.arista.com/introduction-to-managing-eos-devices-memory-utilisation/

经常Arista Switch会显示Mem Util% at 70%，特别是低端的Switch。其实没有用到，而是Linux Caching System把他们Reserve到Cache和Buffer里面。下面红字的都是Free Mem

------------- show processes top memory once -------------

top - 01:14:08 up 21 days, 1:18, 1 user, load average: 0.62, 0.39, 0.41
Tasks: 310 total, 1 running, 252 sleeping, 0 stopped, 57 zombie
%Cpu(s): 10.3 us, 2.0 sy, 0.0 ni, 87.3 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st
KiB Mem: 3900776 total, 3730136 used, 170640 free, 201908 buffers
KiB Swap: 0 total, 0 used, 0 free, 2034600 cached

ckp348.jhm.ebgp.profB.WORKING.05:46:31#sh proc top mem once
top - 05:46:40 up 25 days, 7:21, 2 users, load average: 0.42, 0.25, 0.24
Tasks: 289 total, 2 running, 287 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.4 us, 0.7 sy, 0.0 ni, 94.1 id, 0.0 wa, 0.2 hi, 0.5 si, 0.0 st
KiB Mem: 7931828 total, 4395336 used, 3536492 free, 218976 buffers
KiB Swap: 0 total, 0 used, 0 free, 2371220 cached

Arista optics with DOM support

DOM:

Digital Optical Monitoring.
Per industry standard - SFF-8742, and it gives end users the ability to monitor real-time parameters of SFP, such as in/output power, temperature, laser current and supply voltage
2 Arista optical parts don't support full DOM:

40G AOC doesn't support Tx Power
10G AOC doesn't support DOM

EOS CLIs:

show int eth22/1 transceiver [detail]

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html

upp106#show inv | grep SWDM

22 Arista Networks QSFP-100G-SWDM4 ATH173200003 20

upp106#show int eth 22/1 transceiver

If device is externally calibrated, only calibrated values are printed.

N/A: not applicable, Tx: transmit, Rx: receive.

mA: milliamperes, dBm: decibels (milliwatts).

Bias Optical Optical

Temp Voltage Current Tx Power Rx Power

Port (Celsius) (Volts) (mA) (dBm) (dBm) Last Update

----- --------- -------- -------- -------- -------- -------------------

Et22/1 33.00 3.30 7.69 1.32 0.44 0:00:01 ago

upp106#show int eth 22/1 transceiver de

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.

++ : high alarm, + : high warning, - : low warning, -- : low alarm.

A2D readouts (if they differ), are reported in parentheses.

The threshold values are calibrated.