Arista EOS/Networking Tips by Solomon Yang: Arista

Showing posts with label Arista. Show all posts

5/27/2021

Troubleshooting Arista EOS EVPN VPWS (1)

This blog provides troubleshooting information for common problems with Arista EVPN MPLS VPWS. It covers the following error:

Status: CLI conflict
Status: Admin down
Status: Interface unavailable
Status: No remote
Status: Unprogrammed local connector

Most of the issues start with down state in the output of "show patch panel" like below:

RR1-PE5-wa465.11:42:56#sh patch panel
Patch Connector Status
---------------------------- ------------------------------------------------------- ------
pbVpws-PE5:e471--PE1:e1312 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312 Down
2: Ethernet47/1

And command - "show patch panel <name> details" shows the detailed error reason:

RR1-PE5-wa465.14:58:02#sh patch panel pbVpws-PE5:e471--PE1:e1312 detail

......

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

Status: Interface unavailable

Local MPLS label: 100004

EVPN VPWS type: port-based

Connector 2: Ethernet47/1

Status: Interface mode

The following error status means a local misconfiguration most likely.

Ethernet side:

Status: CLI conflict

Possible reason: duplicated patch panel entries for the same ethernet interface

Status: Admin down

Check the interface status, to make sure is up

Pseudowire side:

Status: Interface unavailable

Possible reason: make sure "no switchport" under interface

After the local configuration is corrected, check the output of "show bgp evpn route-type auto-dis next-hop 0.0.0.0". This is to check if the locally originated BGP EVPN prefixes are advertised out to BGP peers.

RR1-PE5-wa465.14:58:14#show bgp evpn route-type auto-discovery next-hop 0.0.0.0

....

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

* > RD: 100.250.250.5:100 auto-discovery 50471 0000:0000:0000:0000:0000

- - - 0 i

RR1-PE5-wa465.15:07:10#show bgp evpn route-type auto-discovery next-hop 0.0.0.0 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 50471 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.5:100

Paths: 1 available

Local

- from - (0.0.0.0)

Origin IGP, metric -, localpref -, weight 0, valid, local, best

Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

MPLS label: 100004

The most important information as highlighted above is the ETID/PW ID. And compare it with the definition of PW PE5:e471--PE1:e1312

RR1-PE5-wa465.15:06:29#show bgp evpn instance vpws custAB

...

Pseudowire PE5:e471--PE1:e1312

Status: up

VPWS label: 100004

Local VPWS ID: 50471

Remote VPWS ID: 101312

After this point, you are pretty sure this end of PW is good.

But if you are seeing the error status as "No remote", the device may have an issue with remote PW.

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

Status: No remote

Local MPLS label: 100004

EVPN VPWS type: port-based

Connector 2: Ethernet47/1

Status: Unprogrammed local connector

Now let's check the router knows how to reach PW with VPWS ID 101312.

RR1-PE5-wa465.15:13:06#show bgp evpn route-type auto-discovery | grep 101312

RR1-PE5-wa465.15:13:10#

Clearly, the router has no EVPN AD prefix with ETID 101312, so it doesn't how to reach. The next step is to log in to the other end of PW to figure out why by following the above steps. For example, the remote PW should be from NH 100.250.250.11

PE3-lp232.22:00:03#sh bgp evpn route-type auto-discovery next-hop 100.250.250.11

...

Network Next Hop Metric LocPref Weight Path

* >Ec RD: 100.250.250.11:100 auto-discovery 101331 0000:0000:0000:0000:0000

100.250.250.11 - 100 0 i Or-ID: 100.250.250.11 C-LST: 100.250.250.6

From the above output, the PE1(100.250.250.11) sends out an AD prefix with ETID 101331, not 101311.

And sometimes, you may face traffic issues even all the PW and patch panels are UP!! That's because the BGP EVPN AD prefixes only contain the local ETID/PW id, no the expected remote one. There is no way to ensure consistency.

Reference:

https://eos.arista.com/eos-4-24-2f/evpn-mpls-virtual-private-wire-service-vpws/

6/30/2020

"no-internal-vlan" error for routed interfaces

Creating several L3 routed port-channels and sub-interfaces, but they failed to come up with errdisabled status. The output "show interface status err" displays the following reasons:

yo411.16:22:08(config-if-Po1201)#show int status errdisabled
Port Name Status Reason
-------------- ---------------- ----------------- ---------------------
Et3/12/1 errdisabled port-channel-shutdown
Et4/12/1 errdisabled port-channel-shutdown
Po1201.3 errdisabled no-internal-vlan
Po1201.2 errdisabled no-internal-vlan
Po1201 errdisabled no-internal-vlan

Basically, the "port-ch-shutdown" error was caused by the Po1201 being down. Checked the EOS document, the system will reserve an internal VLAN for any "no switchport" interfaces. And the internal VLAN ranges start from 1006 to 4094 (ref: EOS Manual section 19.4.3)

yo411.16:22:59(config-if-Po1201)#show vlan internal usage
1006 Port-Channel1900.4002
1007 Ethernet3/36/3
1008 Port-Channel1900
1009 Ethernet3/36/1
1010 Port-Channel1900.4003
1011 Ethernet3/36/4
1012 Ethernet3/36/2
1013 Ethernet3/36/4.2
1014 Ethernet3/36/4.3

And internal VLAN assignment stops at 1015.

yo411.16:23:23(config-if-Po1201)#sh vlan 1015
VLAN Name Status Ports
----- -------------------------------- --------- -------------------------------
1015 VLAN1015 suspended

So the root cause is that there is an accidental configuration of vlan 1015 with a suspended state, and this blocks the internal VLAN assignment.

yo411.16:24:19(config)#no vlan 1006 - 1099
yo411.16:24:36(config)#sh int status errdisabled

After removing the VLAN configuration, there is no internal-VLAN error anymore.

yo411.16:24:47(config)#show vlan internal usage
...
1015 Port-Channel1201.2
1016 Port-Channel1201.3
1017 Port-Channel1201

Another way is to specify the internal VLAN range to an unused space (ref: EOS manual section 21.3)

yo411(config)# vlan internal order descending range 4000 4094

5/15/2019

Switch goes into power-forever after losing all FANs

If a switch loses all fans, no matter bad hardware or removing parts, the switch should go into a "power-forever" mode. To exit this hardware protection, you have to unplug ALL power cords and reinsert them. This will reset the hardware logic. Unplug then plug power cord one by one doesn't work, neither the power flap.

4/30/2019

Arista MLAG reload-delay timers

In this blog post, I will cover the following MLAG related topics:

What is the MLAG reload-delay and why it is needed
The difference of 2 different timers
How to tune the timer values

After an MLAG peer boot up, all its ports are placed in err-disabled state (except the peer-link) with a reason of "mlag-issu". During the reload-delay, the MLAG agents sync all MAC and ARP information with the active peer.

Another trigger of mlag reload delay is forwarding plane agent restart. On some platforms like TH or T3, a port speed change needs a hitful agent restart, which forces mlag interfaces to transition and starts reload-delay.

From 4.15.2F, the default reload-delay timers are different per platform:

All fixed systems: 300 sec
7500* (Arad/Jericho): 1800 sec (due to long hw initialization time)
7300* (Trident*/TH): 1200 sec

When to start the reload timer? The timers start to tick after the start of MLAG agent. In old releases, it is triggered by the sysdb agent.

Can I lower the reload-delay timers? Yes, but be careful since if the interfaces exit err-disabled mode before sync is done, the mlag peer will blackhole the traffic.

Which value should be used, it is highly based on system and configuration. You can look at the log messages closely.

! this is MLAG agent up and timers are on
Apr 19 11:33:32 localhost Mlag: %AGENT-6-INITIALIZED: Agent 'Mlag' initialized; pid=3030
! LCs power on
Apr 19 11:34:08 R1 NorCalCard: %HARDWARE-6-CARD_POWERED_ON: Card Fabric3 has been powered on. model: 7512R-FM rev: 11.02 serial number: JPE16305615
....
Apr 19 11:34:24 R1 NorCalCard: %HARDWARE-6-CARD_POWERED_ON: Card Linecard11 has been powered on. model: 7500R-36Q-LC rev: 02.01 serial number: JPE16204252
! LC initialization
Apr 19 11:38:40 R1 SandFap: %SAND-6-INIT_SUCCEEDED: Initialization of Linecard12 switch asics succeeded.
....
Apr 19 11:43:12 R1 SandFap: %SAND-6-INIT_SUCCEEDED: Initialization of Linecard13 switch asics succeeded.
! interfaces up
Apr 19 11:43:01 R1 Ebra: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet12/26/3 (mlag.207_leaf_et9/3_et10/3=>et12/17/3), changed state to up
....
Apr 19 11:46:33 R1 Ebra: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet14/34/1 (peerLink=>mlagSec.et14/34/1.100g), changed state to up

This is the messages of a 7512N with 8 J/J+ LCs. So you can see, 13 mins after MLAG agent up, the last peer-link interface is up. Given 5 mins to allow IGP/iBGP session and corresponding hw programming finished, a 20-min (1200 sec) reload-delay timer should be safe with an additional 2-minute buffer.

And there are 2 timers:

MLAG reload-delay timer: this is for all MLAG port-channel links. It can be changed by CLI - "reload-delay mlag <seconds>"
Non-MLAG reload-delay timer: most of the time, they are the timeout values for L3 uplinks. And can be modified by CLI - "reload-delay non-mlag <seconds>"

Before discussing how to tune them, let me give a good example to help your understandings:

Imagine there is a house which has a front door (non-MLAG/uplinks to go out) and a back door (MLAG interfaces/to reach hosts/tenants)
And there is a side to your neighbor (MLAG peer), which shares the same tenants/hosts.
So the first thing to do is to communicate with your neighbor to have all the address information, to know who is where.
With all the knowledge, open which door first?

First of all, the side door is opened first. (so peer-link doesn't have reload-delay and a BGP/IGP peering is required)
During the gap of the front and back door, the traffic will not be lost since there is a side door to exit.
The opening of the back door will have half south-north traffic in, and will go thru the side door. Similar to front door, half north-south traffic.
Most of the time, the MLAG interfaces are facing servers which means south-north traffic is much higher than the opposite traffic.

So it is preferable to configure non-mlag reload delay <= mlag timer.
The ONLY exception is, if "reload-delay mode lacp-standby" is enabled, non-mlag timer > mlag timer. This feature keeps the LACP interfaces up to speed up hardware programming. So if the north-south coming first, the router has to drop them because the MLAG port-channels are not really ready.

Arista EOS: sand hardware-only LAG

CLI:
platform sand lag hardware-only

目的：

一般情况下，1-port LAG是software LAG；
如果增加一个port，变成2-port LAG，就会变成hardware LAG，会有traffic interruption，特别是interface flap!
什么意思啊？很简单，如果是2-port LAG，断了一根线，变成1-port/software LAG，有Traffic impact. 如果这根线在Flapping，traffic就一直受影响
所以索性一上来就是hw-LAG
唯一的影响是，占有的资源(啥资源啊？不知道，LAG table?)会多一些

Arista EOS Memory Util%

https://eos.arista.com/introduction-to-managing-eos-devices-memory-utilisation/

经常Arista Switch会显示Mem Util% at 70%，特别是低端的Switch。其实没有用到，而是Linux Caching System把他们Reserve到Cache和Buffer里面。下面红字的都是Free Mem

------------- show processes top memory once -------------

top - 01:14:08 up 21 days, 1:18, 1 user, load average: 0.62, 0.39, 0.41
Tasks: 310 total, 1 running, 252 sleeping, 0 stopped, 57 zombie
%Cpu(s): 10.3 us, 2.0 sy, 0.0 ni, 87.3 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st
KiB Mem: 3900776 total, 3730136 used, 170640 free, 201908 buffers
KiB Swap: 0 total, 0 used, 0 free, 2034600 cached

ckp348.jhm.ebgp.profB.WORKING.05:46:31#sh proc top mem once
top - 05:46:40 up 25 days, 7:21, 2 users, load average: 0.42, 0.25, 0.24
Tasks: 289 total, 2 running, 287 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.4 us, 0.7 sy, 0.0 ni, 94.1 id, 0.0 wa, 0.2 hi, 0.5 si, 0.0 st
KiB Mem: 7931828 total, 4395336 used, 3536492 free, 218976 buffers
KiB Swap: 0 total, 0 used, 0 free, 2371220 cached

Arista optics with DOM support

DOM:

Digital Optical Monitoring.
Per industry standard - SFF-8742, and it gives end users the ability to monitor real-time parameters of SFP, such as in/output power, temperature, laser current and supply voltage
2 Arista optical parts don't support full DOM:

40G AOC doesn't support Tx Power
10G AOC doesn't support DOM

EOS CLIs:

show int eth22/1 transceiver [detail]

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html

upp106#show inv | grep SWDM

22 Arista Networks QSFP-100G-SWDM4 ATH173200003 20

upp106#show int eth 22/1 transceiver

If device is externally calibrated, only calibrated values are printed.

N/A: not applicable, Tx: transmit, Rx: receive.

mA: milliamperes, dBm: decibels (milliwatts).

Bias Optical Optical

Temp Voltage Current Tx Power Rx Power

Port (Celsius) (Volts) (mA) (dBm) (dBm) Last Update

----- --------- -------- -------- -------- -------- -------------------

Et22/1 33.00 3.30 7.69 1.32 0.44 0:00:01 ago

upp106#show int eth 22/1 transceiver de

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.

++ : high alarm, + : high warning, - : low warning, -- : low alarm.

A2D readouts (if they differ), are reported in parentheses.

The threshold values are calibrated.