12/03/2021

Arista EOS - CoppSystemEgressTrap drop

On Arista 7280*R* series routers, if you are seeing high # of CoppSystemEgressTrap drop in the output of "show cpu counters queue", the most likely reason is that, there are some packets size are larger than the egress interface configured size. 

smv550.08:42:39#show cpu counters queue summary | nz
Aggregate CPU counters:
This indicates the sum of port-dependent queue counters across all input ports.
CoPP Class                                Pkts             Octets           DropPkts         DropOctets
Aggregate
-------------------------------------------------------------------------------------------------------
CoppSystemBgp                            14358            1141461                  0                  0
CoppSystemEgressTrap                     10688           97079104             228017         2071078411
Posted by at No comments:
Labels: , ,

10/19/2021

Arista MLAG vs unstable STP

Before reloading an MLAG peer switch to perform an upgrade/downgrade, one of the key check-ups is to make sure the STP is restartable. Otherwise, traffic loss is expected. 

bn303.08:17:13(s1)#sh spanning-tree instance detail | grep rest
   Stp agent restartable                      :            False

Oct 19 08:16:15 bn303.aristanetworks.com Stp: %SPANTREE-6-STABLE_CHANGE: Stp state is now not stable
Oct 19 08:17:06 bn303.aristanetworks.com Stp: %SPANTREE-6-STABLE_CHANGE: Stp state is now stable

To find out which ports causing this issue, by using CLI - sh spanning-tree topology status detail, to search most recently changed ports, like

bn303.08:22:51(s1)#sh spanning-tree topology status detail | more
Topology: Cist
  Mapped Vlans: 1,4080-4087
  Cpu:                  forwarding (1 changes, last 7:40:20 ago)
  Ethernet8/16/1:       forwarding (3 changes, last 0:23:10 ago) <<<
  PeerEthernet8/16/1:   forwarding (3 changes, last 0:23:10 ago) <<<

The reason of unstable STP is because, the MLAG switch keeps receiving its own (actually from the active MLAG peer) BPDU and can't form an agreement. The solution can be either, shutting down the port or put it as no-switching port. 
Posted by at No comments:
Labels: ,

8/12/2021

EOS device boots into a weird state - [PyServer ar.Aaa not responding, still trying -- is it running?]

One possible reason is that, there is an unknown registered toggle switch.....

Arista Networks EOS 4.22.4M
localhost login: admin
[PyServer ar.Aaa not responding, still trying -- is it running?]
[PyServer ar.Aaa not responding, still trying -- is it running?]

...<snip> reboot....

Welcome to Arista Networks EOS 4.22.4M
Failed(tg != toggles.end()) Toggle #BfdPeerDeferredDeletion not registered
.....

...reboot....
Press Control-C now to enter Aboot shell
...press ctrl-c here to enter aboot mode...

^CWelcome to Aboot.
Aboot# cd /mnt/flash
Aboot# mv toggle_override toggle_override.old
! get rid of /mnt/flash/toggle_override file
Aboot# reboot
....
.... now it is good....

Posted by at No comments:
Labels: , ,

6/13/2021

MPLS Hashing: Control Word and Entropy Label

  • MPLS Lable doesn't have encapsulated protocol information. So after the bottom label, 2 possibilities:
    • IPv4/v6 packets if L3VPN, the 1st 4-bit is ipVersion, 4=ipv4, 6=ipv6
    • Ethernet frames if L2VPN, the 1st 48-bit is dstMAC
  • Along the LSP, if P routers like to do ECMP by hashing, it has to guess like above to do hashing the 5-tuple fields (src/dstAddr, src/dstPort, protoNum)
  • So the problem happens if the dstMac starts with 4 or 6
The solution for the above issue is MPLS control word (RFC 4385, Feb 2006), a 4-byte all-zero (same length as a label) control word after the bottom label. 
So the MPLS control word is to prevent incorrect hashing. How does the LSR do the load-balancing? The answer is Entropy Label (RFC 6790, Nov 2012)
  • Increase the label stack by 2: 1 for ELI (entropy label indicator = 7), 1 for EL (entropy label)
  • Aristea EOS 4.26.0F: LDP Entropy Label Support
    • 4.25.2F: LSR support
    • 4.26.0F: LER support
    • mpls ldp; entropy-label
  • This feature has a limitation: PHP LSR can't pop ELI + EL labels, so has to rely on LER to pop up [ELI, ELI, VPN] labels. 
  • mpls ldp; pseudowire; pseudowire <name>; label flow

Posted by at No comments:
Labels: , , , , ,

5/27/2021

Troubleshooting Arista EOS EVPN VPWS (1)

This blog provides troubleshooting information for common problems with Arista EVPN MPLS VPWS. It covers the following error:
  • Status: CLI conflict
  • Status: Admin down
  • Status: Interface unavailable
  • Status: No remote
  • Status: Unprogrammed local connector
Most of the issues start with down state in the output of "show patch panel" like below:

RR1-PE5-wa465.11:42:56#sh patch panel
Patch                        Connector                                               Status
---------------------------- ------------------------------------------------------- ------
pbVpws-PE5:e471--PE1:e1312   1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312       Down
                             2: Ethernet47/1

And command - "show patch panel <name> details" shows the detailed error reason:

RR1-PE5-wa465.14:58:02#sh patch panel pbVpws-PE5:e471--PE1:e1312 detail
......
Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down
   Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312
      Status: Interface unavailable
      Local MPLS label: 100004
      EVPN VPWS type: port-based
   Connector 2: Ethernet47/1
      Status: Interface mode

The following error status means a local misconfiguration most likely.
  • Ethernet side:
    • Status: CLI conflict
      • Possible reason: duplicated patch panel entries for the same ethernet interface
    • Status: Admin down
      • Check the interface status, to make sure is up
  • Pseudowire side:
    • Status: Interface unavailable
      • Possible reason: make sure "no switchport" under interface
After the local configuration is corrected, check the output of "show bgp evpn route-type auto-dis next-hop 0.0.0.0". This is to check if the locally originated BGP EVPN prefixes are advertised out to BGP peers. 

RR1-PE5-wa465.14:58:14#show bgp evpn route-type auto-discovery next-hop 0.0.0.0
....
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 100.250.250.5:100 auto-discovery 50471 0000:0000:0000:0000:0000
                                 -                     -       -       0       i

RR1-PE5-wa465.15:07:10#show bgp evpn route-type auto-discovery next-hop 0.0.0.0 detail
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
BGP routing table entry for auto-discovery 50471 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.5:100
 Paths: 1 available
  Local
    - from - (0.0.0.0)
      Origin IGP, metric -, localpref -, weight 0, valid, local, best
      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word
      MPLS label: 100004

The most important information as highlighted above is the ETID/PW ID. And compare it with the definition of PW  PE5:e471--PE1:e1312

RR1-PE5-wa465.15:06:29#show bgp evpn instance vpws custAB
...
    Pseudowire PE5:e471--PE1:e1312
      Status: up
      VPWS label: 100004
      Local VPWS ID: 50471
      Remote VPWS ID: 101312

After this point, you are pretty sure this end of PW is good. 

But if you are seeing the error status as "No remote", the device may have an issue with remote PW. 

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down
   Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312
      Status: No remote
      Local MPLS label: 100004
      EVPN VPWS type: port-based
   Connector 2: Ethernet47/1
      Status: Unprogrammed local connector

Now let's check the router knows how to reach PW with VPWS ID 101312. 

RR1-PE5-wa465.15:13:06#show bgp evpn route-type auto-discovery | grep 101312
RR1-PE5-wa465.15:13:10# 

Clearly, the router has no EVPN AD prefix with ETID 101312, so it doesn't how to reach. The next step is to log in to the other end of PW to figure out why by following the above steps. For example, the remote PW should be from NH 100.250.250.11

PE3-lp232.22:00:03#sh bgp evpn route-type auto-discovery next-hop 100.250.250.11
...
          Network                Next Hop              Metric  LocPref Weight  Path
 * >Ec   RD: 100.250.250.11:100 auto-discovery 101331 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i Or-ID: 100.250.250.11 C-LST: 100.250.250.6

From the above output, the PE1(100.250.250.11) sends out an AD prefix with ETID 101331, not 101311. 

And sometimes, you may face traffic issues even all the PW and patch panels are UP!! That's because the BGP EVPN AD prefixes only contain the local ETID/PW id, no the expected remote one. There is no way to ensure consistency. 

Reference:

Posted by at No comments:
Labels: , , , ,

5/20/2021

"Pending BGP Convergence" and invalid on MPLS/SR/EVPN BGP RR

On a BGP EVPN RR, I am seeing some EVPN routes are marked as "Pending BGP Convergence" as below:

RR1.12:01:04#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
   %     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i
   %     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
                                 100.250.250.12        -       100     0       i

The reason is "invalid" in the output of show bgp evpn detail

RR1.12:10:27#sh bgp evpn route-type auto-discovery detail
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100
 Paths: 1 available
  Local (Received from a RR-client)
    100.250.250.11 from 100.250.250.11 (100.250.250.11)
      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution
      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word
BGP routing table entry for auto-discovery 120100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.12:100
 Paths: 1 available
  Local (Received from a RR-client)
    100.250.250.12 from 100.250.250.12 (100.250.250.12)
      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution
      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

Here is the BGP configuration:

RR1.11:59:52#sh run sec router bgp
router bgp 65000
   router-id 100.250.250.5
   no bgp default ipv4-unicast
   neighbor RRC peer group
   neighbor RRC remote-as 65000
   neighbor RRC update-source Loopback0
   neighbor RRC route-reflector-client
   neighbor RRC send-community extended
   neighbor 100.250.250.11 peer group RRC
   neighbor 100.250.250.12 peer group RRC
   neighbor 100.250.250.21 peer group RRC
   neighbor 100.250.250.22 peer group RRC
   !
   address-family evpn
      neighbor RRC activate

Add "neighbor default encapsulation mpls next-hop-self source-interface Loopback0" under EVPN AF

RR1.12:20:00(config)#router bgp 65000
RR1.12:20:02(config-router-bgp)#address-family evpn
RR1.12:20:04(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface Loopback0

One prefix shows "* >" - valid. One shows no code

RR1.12:20:10#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
         RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i
 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
                                 100.250.250.12        -       100     0       i

Details still shows "invalid"

RR1.12:22:09#sh bgp evpn route-type auto-discovery rd 100.250.250.11:100 detail
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100
 Paths: 1 available
  Local (Received from a RR-client)
    100.250.250.11 from 100.250.250.11 (100.250.250.11)
      Origin IGP, metric -, localpref 100, weight 0, invalid, internal
      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word
      MPLS label: 132768

No tunnel endpoint in "show tunnel rib brief"

RR1.12:22:50#show tunnel rib brief | grep 250.250.1
   100.250.250.1/32               IS-IS SR IPv4       6               65                      115               20
   100.250.250.12/32              IS-IS SR IPv4       4               65                      115               40

The cause is the PE's config, missing node-segment ipv4 configuration

PEa1.12:10:13#sh run int lo0
interface Loopback0
   ip address 100.250.250.11/32
   ipv6 address 2000:100:250:250::11/128
   node-segment ipv6 index 11
   isis enable MplsSR
   isis passive

PEa1.12:25:24#c
PEa1.12:26:08(config)#int lo0
PEa1.12:26:10(config-if-Lo0)#node-segment ipv4 index 11

On RR1, NH emerges in tunnel rib. 

RR1.12:22:56#show tunnel rib brief | grep 250.250.1
   100.250.250.1/32        IS-IS SR IPv4       6               65                      115               20
   100.250.250.11/32       IS-IS SR IPv4       2               65                      115               30
   100.250.250.12/32       IS-IS SR IPv4       4               65                      115               40

And bgp table is good

RR1.12:24:51#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i
 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
                                 100.250.250.12        -       100     0       i

Posted by at No comments:
Labels: , ,

1/18/2021

Arista EOS: RSVP related features

  References:

Posted by at No comments:
Labels:

1/12/2021

Arista EOS: %ETH-4-ERRDISABLE: no-internal-vlan error detected

If you are seeing the following error message 

Jan 12 19:03:19 sn452 Ebra: %ETH-4-ERRDISABLE: no-internal-vlan error detected on Ethernet81.

And the output of "show interface status" indicates some ethernet ports are error-disabled like

sn452.19:00:35(config-if-Et1-104)#show interfaces status errdisabled
   Port        Name             Status         Reason
----------- ---------------- ----------------- ----------------
   Et81                         errdisabled    no-internal-vlan
   Et82                         errdisabled    no-internal-vlan

The reason for the above error is that EOS running out of internal VLAN for routed ports. The EOS needs to assign an internal VLAN for the none switching ports. If you have the following 2 lines of configuration, you may hit this error

switchport default mode routed <<< all ports are routed
vlan internal order descending range 4001 4080 <<< but only 80 VLANs assigned

So the solution is to remove the internal VLAN range. 

Posted by at No comments:
Labels: ,

1/02/2021

Why unprotected ISIS segment routing prefix with TiLFA enabled

 


In the above topology, the output of "show isis segment-routing prefix" on PE11 shows the SID of PE22 is not protected. 

bn303-PEa1.23:58:00(config-router-isis-af)#show isis segment-routing prefix-segments

System ID: 0000.0000.0011 Instance: 'isis-sr'
SR supported Data-plane: MPLS SR Router ID: 200.250.250.1

Node: 10     Proxy-Node: 0      Prefix: 0       Total Segments: 10

Flag Descriptions: R: Re-advertised, N: Node Segment, P: no-PHP
                   E: Explicit-NULL, V: Value, L: Local
Segment status codes: * - Self originated Prefix, L1 - level 1, L2 - level 2
  Prefix                      SID Type       Flags                   System ID       Level Protection
  ------------------------- ----- ---------- ----------------------- --------------- ----- ----------
......
  100.250.250.22/32            22 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0022  L2    unprotected
  100.255.255.31/32            31 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0031  L2    node

The reason the SID of PE22 is "unprotected" is that the PE11 has a 2-way ECMP to the destination. 

bn303-PEa1.23:58:01(config-router-isis-af)#show ip route 100.250.250.22/32
 I L2     100.250.250.22/32 [115/50] via 100.2.11.0, Ethernet3/1/1
                                     via 100.1.11.0, Ethernet10/36/1


Now let's the shutdown the interface between PE11 and P2 to break this ECMP. Now the prefix-segment is proteced now. 

bn303-PEa1.00:14:21(config)#int e3/1/1
bn303-PEa1.00:14:27(config-if-Et3/1/1)#shu

bn303-PEa1.00:14:31#show ip route 100.250.250.22/32


 I L2     100.250.250.22/32 [115/50] via 100.1.11.0, Ethernet10/36/1

bn303-PEa1.00:14:34#show isis segment-routing prefix-segments

Segment status codes: * - Self originated Prefix, L1 - level 1, L2 - level 2
  Prefix                      SID Type       Flags                   System ID       Level Protection
  ------------------------- ----- ---------- ----------------------- --------------- ----- ----------
...
  100.250.250.22/32            22 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0022  L2    node
  100.255.255.31/32            31 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0031  L2    node
Posted by at No comments:
Labels: ,
Subscribe to: Posts (Atom)