Easy-mistake-check-list for Arista EOS

Here is my easy-mistake-check-list for Arista EOS

• Incorrect hardware TCAM profile:
• show hardware tcam profile
• Unprogrammed routes:
• show platform sand l3 summary | grep unprog
• show ip/ipv6 route vrf all | egrep '^*'
• bash top - see high cpu% for ksoftirqd 
• Incorrect dstMac (especially for ixia traffic):
• tcpdump packets and check dstMac
• PIC (bgp additional-path install) not working: 
• PIC only works for directly connected NH. (only old EOS version prior 4.24?)

smv575-AFO-CE1.21:24:29(config)#sh ip route vrf vrf-0400 200.4.0.0/24 detail

 B E      200.4.0.0/24 [200/0] via 100.255.255.101/32, BGP LU tunnel index 15, label 116386

                                  via 100.11.101.1, Port-Channel101, =>PE1, label imp-null(3) <<<< no backup

On the remote ASBR:

   address-family vpn-ipv4

      neighbor AFO-10B activate

      neighbor rmtPE activate

      neighbor default encapsulation mpls next-hop-self source-interface Loopback0

      neighbor default encapsulation mpls next-hop-self received-vpnv4-routes

Remove the above line:

 B E      200.4.0.0/24 [200/0] via 100.11.101.1, Port-Channel101 (egress VRF default) =>PE1, label 116386

                               via 10.11.12.1, Port-Channel1112 (egress VRF default) CE1<=>CE2, label 118365, backup

Vlan translation on Arista EOS

Simple topology:

[vlan 509]==> e17/1:: DCS-7170-32C-F :: po5 [vlan 309]

Requirement:

* ingress traffic with vlan tag 509 at et17/1

* like to translate it to vlan tag 309 on egress intf - po5

Configuration:

interface Port-Channel5

   description egress::po5

   switchport trunk native vlan 399

   switchport trunk allowed vlan 300-321,330,399,503,509

   switchport mode trunk

   switchport vlan translation 309 509

Syntax:

  switchport vlan translation <vlan_map_from> <vlan_map_to>

Please note: the vlan # from and to in above syntax is from the incoming point of view

EVPN VxLAN Centralized Routing: redistribute VARP MAC to prevent unnecessary flooded traffic

https://www.arista.com/en/um-eos/eos-evpn-vxlan-single-gateway-centralized-routing

In the below EVPN VxLAN Centralized Routing topology

There are 2 centralized routers, both have default gateway SVI - Vlan100 (100.1.0.1/24) with a virtual MAC (00dc.0000.0001)

interface Vlan100

   mtu 9214

   ip address virtual 100.1.0.1/24

!

ip virtual-router mac-address 00:dc:00:00:00:01

The hostA (100.1.0.201) pings the default GW address and receives duplicated ICMP response like below:

HostA.10:07:37#ping vrf Vlan100 100.1.0.1

PING 100.1.0.1 (100.1.0.1) 72(100) bytes of data.

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.181 ms

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.318 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.345 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.103 ms

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.119 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.135 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.149 ms (DUP!)

And the tcpdump shows the same, duplicated replies from both gateways:

[admin@HostA ~]$ sudo ip netns exec ns-Vlan100 tcpdump -nvvi vlan100

10:07:46.165380 44:4c:a8:80:c1:c8 > 00:dc:00:00:00:01, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 20298, offset 0, flags [none], proto ICMP (1), length 100) <<< req to VARP MAC 00dc.0000.0001

    100.1.0.201 > 100.1.0.1: ICMP echo request, id 6062, seq 2, length 80

10:07:46.165446 44:4c:a8:a5:11:41 > 44:4c:a8:80:c1:c8, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 8443, offset 0, flags [none], proto ICMP (1), length 100) <<< reply1 from 444c.a8a5.1141

    100.1.0.1 > 100.1.0.201: ICMP echo reply, id 6062, seq 2, length 80

10:07:46.165467 44:4c:a8:a5:11:40 > 44:4c:a8:80:c1:c8, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 60002, offset 0, flags [none], proto ICMP (1), length 100) <<< reply2 from 444c.a8a5.1140

    100.1.0.1 > 100.1.0.201: ICMP echo reply, id 6062, seq 2, length 80

Why does it happen? It is because the L2 VTEP doesn't learn the MAC address

L2VTEPa.10:07:17#sh mac address-table vlan 100

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

 100    444c.a86c.6dd9    DYNAMIC     Vx1        1       0:00:49 ago

 100    444c.a87c.d809    DYNAMIC     Vx1        1       0:34:40 ago

 100    444c.a880.c1c8    DYNAMIC     Po1        1       0:05:44 ago

Thus, no type-2 update for 00dc.0000.0001

L2VTEPa.10:07:22#show bgp evpn route-type mac-ip 00dc.0000.0001

BGP routing table information for VRF default

Router identifier 10.0.0.11, local AS number 65011

Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP

                    c - Contributing to ECMP, % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

L2VTEPa.10:09:42#

Per https://www.arista.com/en/um-eos/eos-evpn-vxlan-single-gateway-centralized-routing, CLI - "redistribute router-mac next-hop vtep primary" to generate type-2 udpate for VARP MAC. 

router bgp 65017

   vlan 100

      rd 10.0.0.17:100

      route-target both 100:100

      redistribute learned

      redistribute router-mac next-hop vtep primary

Now the pings are not dup'ed anymore. 

HostA.10:07:46#ping vrf Vlan100 100.1.0.1

PING 100.1.0.1 (100.1.0.1) 72(100) bytes of data.

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.208 ms

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.155 ms

80 bytes from 100.1.0.1: icmp_seq=3 ttl=64 time=0.111 ms

80 bytes from 100.1.0.1: icmp_seq=4 ttl=64 time=0.093 ms

80 bytes from 100.1.0.1: icmp_seq=5 ttl=64 time=0.160 ms

L2VTEPa.10:19:14#sh mac address-table vlan 100

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

 100    00dc.0000.0001    STATIC      Vx1

L2VTEPa#show bgp evpn route-type mac-ip 00dc.0000.0001

BGP routing table information for VRF default

Router identifier 10.0.0.11, local AS number 65011

Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP

                    c - Contributing to ECMP, % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >Ec    RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

                                 10.0.0.17             -       100     0       65001 65017 i

 *  ec    RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

                                 10.0.0.17             -       100     0       65001 65017 i

 *  ec    RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

                                 10.0.0.17             -       100     0       65001 65017 i

 *  ec    RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

                                 10.0.0.17             -       100     0       65001 65017 i

 * >Ec    RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

                                 10.0.0.18             -       100     0       65001 65018 i

 *  ec    RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

                                 10.0.0.18             -       100     0       65001 65018 i

 *  ec    RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

                                 10.0.0.18             -       100     0       65001 65018 i

 *  ec    RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

                                 10.0.0.18             -       100     0       65001 65018 i

Arista EOS: traffic tool on Linux

https://eos.arista.com/traffic-generator-on-arista/

• iperf
• but seeing time out error? iptables issue? need to check later
• ethxmit

Arista EOS - CoppSystemEgressTrap drop

On Arista 7280*R* series routers, if you are seeing high # of CoppSystemEgressTrap drop in the output of "show cpu counters queue", the most likely reason is that, there are some packets size are larger than the egress interface configured size. 

smv550.08:42:39#show cpu counters queue summary | nz
Aggregate CPU counters:
This indicates the sum of port-dependent queue counters across all input ports.
CoPP Class                                Pkts             Octets           DropPkts         DropOctets
Aggregate
-------------------------------------------------------------------------------------------------------
CoppSystemBgp                            14358            1141461                  0                  0
CoppSystemEgressTrap                     10688           97079104             228017         2071078411

Arista MLAG vs unstable STP

Before reloading an MLAG peer switch to perform an upgrade/downgrade, one of the key check-ups is to make sure the STP is restartable. Otherwise, traffic loss is expected. 

bn303.08:17:13(s1)#sh spanning-tree instance detail | grep rest

   Stp agent restartable                      :            False

Oct 19 08:16:15 bn303.aristanetworks.com Stp: %SPANTREE-6-STABLE_CHANGE: Stp state is now not stable

Oct 19 08:17:06 bn303.aristanetworks.com Stp: %SPANTREE-6-STABLE_CHANGE: Stp state is now stable

To find out which ports causing this issue, by using CLI - sh spanning-tree topology status detail, to search most recently changed ports, like

bn303.08:22:51(s1)#sh spanning-tree topology status detail | more

Topology: Cist

  Mapped Vlans: 1,4080-4087

  Cpu:                  forwarding (1 changes, last 7:40:20 ago)

  Ethernet8/16/1:       forwarding (3 changes, last 0:23:10 ago) <<<

  PeerEthernet8/16/1:   forwarding (3 changes, last 0:23:10 ago) <<<

The reason of unstable STP is because, the MLAG switch keeps receiving its own (actually from the active MLAG peer) BPDU and can't form an agreement. The solution can be either, shutting down the port or put it as no-switching port. 

EOS device boots into a weird state - [PyServer ar.Aaa not responding, still trying -- is it running?]

One possible reason is that, there is an unknown registered toggle switch.....

Arista Networks EOS 4.22.4M
localhost login: admin
[PyServer ar.Aaa not responding, still trying -- is it running?]
[PyServer ar.Aaa not responding, still trying -- is it running?]

...<snip> reboot....

Welcome to Arista Networks EOS 4.22.4M

Failed(tg != toggles.end()) Toggle #BfdPeerDeferredDeletion not registered

.....

...reboot....

Press Control-C now to enter Aboot shell

...press ctrl-c here to enter aboot mode...

^CWelcome to Aboot.

Aboot# cd /mnt/flash

Aboot# mv toggle_override toggle_override.old

! get rid of /mnt/flash/toggle_override file

Aboot# reboot

....

.... now it is good....

MPLS Hashing: Control Word and Entropy Label

A good presentation on this topic: https://ripe66.ripe.net/presentations/141-RIPE66_Understanding_MPLS_Hashing-Snijders.pdf

• MPLS Lable doesn't have encapsulated protocol information. So after the bottom label, 2 possibilities:
• IPv4/v6 packets if L3VPN, the 1st 4-bit is ipVersion, 4=ipv4, 6=ipv6
• Ethernet frames if L2VPN, the 1st 48-bit is dstMAC
• Along the LSP, if P routers like to do ECMP by hashing, it has to guess like above to do hashing the 5-tuple fields (src/dstAddr, src/dstPort, protoNum)
• So the problem happens if the dstMac starts with 4 or 6

The solution for the above issue is MPLS control word (RFC 4385, Feb 2006), a 4-byte all-zero (same length as a label) control word after the bottom label. 

• Arista EOS 4.21.3F: Configurable L2 EVPN MPLS control word
• mpls evpn; label-stack control-word
• Arista EOS 4.22.1F: Control Word support on LDP Pseudowire
• mpls ldp; pseudowire <name>; control-word

So the MPLS control word is to prevent incorrect hashing. How does the LSR do the load-balancing? The answer is Entropy Label (RFC 6790, Nov 2012)

• Increase the label stack by 2: 1 for ELI (entropy label indicator = 7), 1 for EL (entropy label)
• Aristea EOS 4.26.0F: LDP Entropy Label Support
• 4.25.2F: LSR support
• 4.26.0F: LER support
• mpls ldp; entropy-label
• This feature has a limitation: PHP LSR can't pop ELI + EL labels, so has to rely on LER to pop up [ELI, ELI, VPN] labels. 

Another solution is FAT - Flow Aware Transport on LDP pseudowires (4.24.0F) based on RFC 6391

• mpls ldp; pseudowire; pseudowire <name>; label flow

Troubleshooting Arista EOS EVPN VPWS (1)

This blog provides troubleshooting information for common problems with Arista EVPN MPLS VPWS. It covers the following error:

• Status: CLI conflict
• Status: Admin down
• Status: Interface unavailable
• Status: No remote
• Status: Unprogrammed local connector

Most of the issues start with down state in the output of "show patch panel" like below:

RR1-PE5-wa465.11:42:56#sh patch panel
Patch                        Connector                                               Status
---------------------------- ------------------------------------------------------- ------
pbVpws-PE5:e471--PE1:e1312   1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312       Down
                             2: Ethernet47/1

And command - "show patch panel <name> details" shows the detailed error reason:

RR1-PE5-wa465.14:58:02#sh patch panel pbVpws-PE5:e471--PE1:e1312 detail

......

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

   Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

      Status: Interface unavailable

      Local MPLS label: 100004

      EVPN VPWS type: port-based

   Connector 2: Ethernet47/1

      Status: Interface mode

The following error status means a local misconfiguration most likely.

• Ethernet side:
• Status: CLI conflict
• Possible reason: duplicated patch panel entries for the same ethernet interface
• Status: Admin down
• Check the interface status, to make sure is up
• Pseudowire side:
• Status: Interface unavailable
• Possible reason: make sure "no switchport" under interface

After the local configuration is corrected, check the output of "show bgp evpn route-type auto-dis next-hop 0.0.0.0". This is to check if the locally originated BGP EVPN prefixes are advertised out to BGP peers. 

RR1-PE5-wa465.14:58:14#show bgp evpn route-type auto-discovery next-hop 0.0.0.0

....

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.5:100 auto-discovery 50471 0000:0000:0000:0000:0000

                                 -                     -       -       0       i

RR1-PE5-wa465.15:07:10#show bgp evpn route-type auto-discovery next-hop 0.0.0.0 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 50471 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.5:100

 Paths: 1 available

  Local

    - from - (0.0.0.0)

      Origin IGP, metric -, localpref -, weight 0, valid, local, best

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

      MPLS label: 100004

The most important information as highlighted above is the ETID/PW ID. And compare it with the definition of PW  PE5:e471--PE1:e1312

RR1-PE5-wa465.15:06:29#show bgp evpn instance vpws custAB

...

    Pseudowire PE5:e471--PE1:e1312

      Status: up

      VPWS label: 100004

      Local VPWS ID: 50471

      Remote VPWS ID: 101312

After this point, you are pretty sure this end of PW is good. 

But if you are seeing the error status as "No remote", the device may have an issue with remote PW. 

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

   Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

      Status: No remote

      Local MPLS label: 100004

      EVPN VPWS type: port-based

   Connector 2: Ethernet47/1

      Status: Unprogrammed local connector

Now let's check the router knows how to reach PW with VPWS ID 101312. 

RR1-PE5-wa465.15:13:06#show bgp evpn route-type auto-discovery | grep 101312

RR1-PE5-wa465.15:13:10# 

Clearly, the router has no EVPN AD prefix with ETID 101312, so it doesn't how to reach. The next step is to log in to the other end of PW to figure out why by following the above steps. For example, the remote PW should be from NH 100.250.250.11

PE3-lp232.22:00:03#sh bgp evpn route-type auto-discovery next-hop 100.250.250.11

...

          Network                Next Hop              Metric  LocPref Weight  Path

 * >Ec   RD: 100.250.250.11:100 auto-discovery 101331 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i Or-ID: 100.250.250.11 C-LST: 100.250.250.6

From the above output, the PE1(100.250.250.11) sends out an AD prefix with ETID 101331, not 101311. 

And sometimes, you may face traffic issues even all the PW and patch panels are UP!! That's because the BGP EVPN AD prefixes only contain the local ETID/PW id, no the expected remote one. There is no way to ensure consistency. 

Reference:

• https://eos.arista.com/eos-4-24-2f/evpn-mpls-virtual-private-wire-service-vpws/

"Pending BGP Convergence" and invalid on MPLS/SR/EVPN BGP RR

On a BGP EVPN RR, I am seeing some EVPN routes are marked as "Pending BGP Convergence" as below:

RR1.12:01:04#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
   %     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i
   %     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
                                 100.250.250.12        -       100     0       i

The reason is "invalid" in the output of show bgp evpn detail

RR1.12:10:27#sh bgp evpn route-type auto-discovery detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.11 from 100.250.250.11 (100.250.250.11)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

BGP routing table entry for auto-discovery 120100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.12:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.12 from 100.250.250.12 (100.250.250.12)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

Here is the BGP configuration:

RR1.11:59:52#sh run sec router bgp

router bgp 65000

   router-id 100.250.250.5

   no bgp default ipv4-unicast

   neighbor RRC peer group

   neighbor RRC remote-as 65000

   neighbor RRC update-source Loopback0

   neighbor RRC route-reflector-client

   neighbor RRC send-community extended

   neighbor 100.250.250.11 peer group RRC

   neighbor 100.250.250.12 peer group RRC

   neighbor 100.250.250.21 peer group RRC

   neighbor 100.250.250.22 peer group RRC

   !

   address-family evpn

      neighbor RRC activate

Add "neighbor default encapsulation mpls next-hop-self source-interface Loopback0" under EVPN AF

RR1.12:20:00(config)#router bgp 65000

RR1.12:20:02(config-router-bgp)#address-family evpn

RR1.12:20:04(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface Loopback0

One prefix shows "* >" - valid. One shows no code

RR1.12:20:10#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

         RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i

 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

                                 100.250.250.12        -       100     0       i

Details still shows "invalid"

RR1.12:22:09#sh bgp evpn route-type auto-discovery rd 100.250.250.11:100 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.11 from 100.250.250.11 (100.250.250.11)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

      MPLS label: 132768

No tunnel endpoint in "show tunnel rib brief"

RR1.12:22:50#show tunnel rib brief | grep 250.250.1

   100.250.250.1/32               IS-IS SR IPv4       6               65                      115               20

   100.250.250.12/32              IS-IS SR IPv4       4               65                      115               40

The cause is the PE's config, missing node-segment ipv4 configuration

PEa1.12:10:13#sh run int lo0

interface Loopback0

   ip address 100.250.250.11/32

   ipv6 address 2000:100:250:250::11/128

   node-segment ipv6 index 11

   isis enable MplsSR

   isis passive

PEa1.12:25:24#c

PEa1.12:26:08(config)#int lo0

PEa1.12:26:10(config-if-Lo0)#node-segment ipv4 index 11

On RR1, NH emerges in tunnel rib. 

RR1.12:22:56#show tunnel rib brief | grep 250.250.1

   100.250.250.1/32        IS-IS SR IPv4       6               65                      115               20

   100.250.250.11/32       IS-IS SR IPv4       2               65                      115               30

   100.250.250.12/32       IS-IS SR IPv4       4               65                      115               40

And bgp table is good

RR1.12:24:51#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i

 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

                                 100.250.250.12        -       100     0       i

Arista EOS: RSVP related features

  References:

• https://eos.arista.com/eos-4-25-1f/rsvp-te-ler/
• https://eos.arista.com/eos-4-23-2f/rsvp-te-lsr/
• https://eos.arista.com/eos-4-23-1f/bgp-nexthop-resolution-ribs-evpn-and-ipv4-6-labeled-unicast-support/
• https://eos.arista.com/eos-4-22-1f/bgp-nexthop-resolution-ribs-mpls-vpn-support/
• https://eos.arista.com/eos-4-22-0f/user-defined-tunnel-ribs/

Arista EOS: %ETH-4-ERRDISABLE: no-internal-vlan error detected

If you are seeing the following error message 

Jan 12 19:03:19 sn452 Ebra: %ETH-4-ERRDISABLE: no-internal-vlan error detected on Ethernet81.

And the output of "show interface status" indicates some ethernet ports are error-disabled like

sn452.19:00:35(config-if-Et1-104)#show interfaces status errdisabled
   Port        Name             Status         Reason
----------- ---------------- ----------------- ----------------
   Et81                         errdisabled    no-internal-vlan
   Et82                         errdisabled    no-internal-vlan

The reason for the above error is that EOS running out of internal VLAN for routed ports. The EOS needs to assign an internal VLAN for the none switching ports. If you have the following 2 lines of configuration, you may hit this error

switchport default mode routed <<< all ports are routed
vlan internal order descending range 4001 4080 <<< but only 80 VLANs assigned

So the solution is to remove the internal VLAN range. 

Why unprotected ISIS segment routing prefix with TiLFA enabled

 

In the above topology, the output of "show isis segment-routing prefix" on PE11 shows the SID of PE22 is not protected. 

bn303-PEa1.23:58:00(config-router-isis-af)#show isis segment-routing prefix-segments

System ID: 0000.0000.0011 Instance: 'isis-sr'

SR supported Data-plane: MPLS SR Router ID: 200.250.250.1

Node: 10     Proxy-Node: 0      Prefix: 0       Total Segments: 10

Flag Descriptions: R: Re-advertised, N: Node Segment, P: no-PHP

                   E: Explicit-NULL, V: Value, L: Local

Segment status codes: * - Self originated Prefix, L1 - level 1, L2 - level 2

  Prefix                      SID Type       Flags                   System ID       Level Protection

  ------------------------- ----- ---------- ----------------------- --------------- ----- ----------

......

  100.250.250.22/32            22 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0022  L2    unprotected

  100.255.255.31/32            31 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0031  L2    node

The reason the SID of PE22 is "unprotected" is that the PE11 has a 2-way ECMP to the destination. 

bn303-PEa1.23:58:01(config-router-isis-af)#show ip route 100.250.250.22/32

 I L2     100.250.250.22/32 [115/50] via 100.2.11.0, Ethernet3/1/1

                                     via 100.1.11.0, Ethernet10/36/1

Now let's the shutdown the interface between PE11 and P2 to break this ECMP. Now the prefix-segment is proteced now. 

bn303-PEa1.00:14:21(config)#int e3/1/1

bn303-PEa1.00:14:27(config-if-Et3/1/1)#shu

bn303-PEa1.00:14:31#show ip route 100.250.250.22/32

 I L2     100.250.250.22/32 [115/50] via 100.1.11.0, Ethernet10/36/1

bn303-PEa1.00:14:34#show isis segment-routing prefix-segments

Segment status codes: * - Self originated Prefix, L1 - level 1, L2 - level 2

  Prefix                      SID Type       Flags                   System ID       Level Protection

  ------------------------- ----- ---------- ----------------------- --------------- ----- ----------

...

  100.250.250.22/32            22 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0022  L2    node

  100.255.255.31/32            31 Node       R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0031  L2    node

Arista 7050CX - Packet Forwarding Pipeline

 

Reference: https://www.arista.com/assets/data/pdf/Whitepapers/Arista_7050X_Switch_Architecture.pdf

Arista EOS: how to ping link local address under VRF

Here is the subinterface configuration:

interface Ethernet34.4007
   description =>v7:Sp4-ck478
   mtu 9214
   encapsulation dot1q vlan 4007
   vrf cust
   ipv6 address fe80::1/64 link-local

To ping the remote link-local address, we could use 

RtrA#ping vrf cust ipv6 fe80:: interface et34.4007
PING fe80::(fe80::) from fe80::1%et34.4007 et34.4007: 72 data bytes
80 bytes from fe80::1%et31.4007: icmp_seq=1 ttl=64 time=0.076 ms
80 bytes from fe80::1%et31.4007: icmp_seq=2 ttl=64 time=0.029 ms
80 bytes from fe80::1%et31.4007: icmp_seq=3 ttl=64 time=0.027 ms
80 bytes from fe80::1%et31.4007: icmp_seq=4 ttl=64 time=0.027 ms
80 bytes from fe80::1%et31.4007: icmp_seq=5 ttl=64 time=0.032 ms

Arista EOS: % - Pending BGP convergence when "show bgp evpn"

1. Router(RR) receives 4 bgp evpn prefixes from RRC

wa465-PD.P1-JHM.SR.14:35:52(config-router-bgp-af)#show bgp evpn summary
BGP summary information for VRF default
Router identifier 100.250.250.30, local AS number 65000
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  100.250.250.11   4 65000            500       487    0    0 00:02:15 Estab   4      4
  100.250.250.12   4 65000            504       488    0    0 00:02:15 Estab   4      4

2. But "show bgp evpn" complains about "Pending BGP convergence"

wa465-PD.P1-JHM.SR.14:36:48(config-router-bgp-af)#show bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.30, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

   %     RD: 100.250.250.12:100 auto-discovery 0 0000:0000:0000:1111:0000

                                 100.250.250.12        -       100     0       i

   %     RD: 100.250.250.12:1 auto-discovery 0000:0000:0000:1111:0000

                                 100.250.250.12        -       100     0       i

3. Check the NH, it was resolved in the tunnel rib

wa465-PD.P1-JHM.SR.14:37:33(config-router-bgp-af)#show tunnel rib brief | grep 100.250.250.12

   100.250.250.12/32       IS-IS SR IPv4       3               65                      115               20

4. The cause is the mis-configuration under router bgp evpn address family. The EOS needs to know to resolve the NH by using the mpls NH tunnel rib. 

wa465-PD.P1-JHM.SR.14:36:53(config-router-bgp-af)#router bgp 65000

wa465-PD.P1-JHM.SR.14:36:57(config-router-bgp)#add evpn

wa465-PD.P1-JHM.SR.14:37:00(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface loopback 0

wa465-PD.P1-JHM.SR.14:37:25(config-router-bgp-af)#show bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.30, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.12:100 auto-discovery 0 0000:0000:0000:1111:0000

                                 100.250.250.12        -       100     0       i

 * >     RD: 100.250.250.12:1 auto-discovery 0000:0000:0000:1111:0000

BFD flaps in scale environment

 In a lab scenario, says between 2 routers, there are hundreds of sub-interfaces and a BGP session with BFD on each subinterface. Then BFD flaps is seen. The scale information is as followed:

• 256 subinterfaces
• 256 ebgp session enabled with BFD
• BFD timers are 50ms x 3

Addressing           Type                         Up        Init        Down    AdminDown

-------------------- -------------------- ------------- ----------- ----------- ---------

All                  All                     239 [0]       8 [0]       9 [0]        0 [0]

IPv4                 All                     239 [0]       8 [0]       9 [0]        0 [0]

    single hop       All                     239 [0]       8 [0]       9 [0]        0 [0]

                     normal                  239 [0]       8 [0]       9 [0]        0 [0]

From the above output, you can see 17 of 256 sessions are not up. And HW BFD is enabled (by default in Eos), but no helps. 

ghb289#show bfd hardware utilization
Chip Name          Number Of HW Sessions*    Maximum Number Of HW Sessions*
--------------- ---------------------------- ------------------------------
Jericho0                                0                               200
Jericho1                              128                               200
Jericho2                              128                               200

"show cpu counter queue" indicates high # of drop of CoppSystemBfd class, which means the receiving BFD packets exceeds the b/w limit of Copp.

ghb289#sh cpu counters queue |nz |grep -i bfd

CoppSystemBfd              Et16/2                5584344          390904080                  0                  0
CoppSystemBfd              Et51/2              783529888        57981211712            3985376          294917824

After increasing the shape/bandwidth of copp-system-bfd, there is bfd flaps anymore.

  Class-map: copp-system-bfd (match-any)

       shape : 25000 kbps

       bandwidth : 2500 kbps

Arista EOS Lag/ECMP Hashing Features

References/Manual:

• EOS Manual Section 12.4: Load Balancing Hash Algorithms

References/TOI:

• EOS 4.24.2F TOI - Support for dynamic load balancing on ECMP groups (Trident Only)
• EOS 4.22.1F TOI - ECMP Load Balance Profile Support (Sand/Jericho Only)
• EOS 4.21.3F TOI - Symmetric header selection in LAG load-balancing profiles (Jericho Only)
• EOS 4.21.0F TOI - UDF flex hashing improvements (Trident Only)
• EOS 4.20.6F TOI - LAG hashing enhancement for configuring member-selection method (Jericho Only)
• EOS 4.20.5F TOI - LAG hashing on Ingress Interface
• EOS 4.20.5F TOI - Lag Hashing Key Shift
• EOS 4.20.5F TOI - VxLAN over MPLS LAG hashing optimizations for Bridged ports
• EOS 4.20.1F TOI - LAG and ECMP selectable hashing fields (Jericho Only)
• EOS 4.18.0F TOI - Hardware load-balancing for ingress/egress and fabric/egress replication
• EOS 4.17.2F TOI - Configurable field selection for ECMP hash, (Trident Only)
• EOS 4.17.2F TOI - Disable usage of ingress interface in ECMP hashing
• EOS 4.17.1F TOI - Inner IP hashing for MPLSoGRE
• EOS 4.17.0F TOI - Global LAG Hashing Profiles (Jericho Only)
• EOS 4.17.0F TOI - ECMP Hash Visibility
• EOS 4.15.2F TOI - GRE in LAG hash
• EOS 4.15.2F TOI - LAG/ECMP hashing on TTL/Hop Limit
• EOS 4.15.0F TOI - Dynamic and symmetric LAG hashing
• EOS 4.15.0F TOI - Tap aggregation enhancements

Sample: In this below sample, I like to show how to use the above features to fix an ipv6 multicast unbalanced issue. 

Says, in the above topology, there are 2000 ipv6 multicast streams flowing ixia-harness-[vrrp primary| secondary]-nexthop router. Let's see the output of srnz on VRRP primary. 

1) w/o any hashing config, lowest vs highest = 220 vs 317

ghs259-GW2.21:24:15(config-if-Po2)#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     220.1   0.2%       35

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     316.9   0.3%       50

Et13/1    GW2-HUB2     0:05       0.0   0.0%        0     266.1   0.3%       42

Et14/1    GW2-HUB2     0:05       0.0   0.0%        0     269.3   0.3%       43

Po2       GW2-HUB2     0:05       0.0   0.0%        0    1072.3   0.3%      171

2) hash key shift

Let's try 1 hashing feature - Lag Hashing Key Shift. Create a LB profile - mcast-v6 + key shift, apply it globally. 

ghs259-GW2.21:37:22(config)#load-balance policies

ghs259-GW2.21:37:28(config-load-balance-policies)#   load-balance sand profile mcast-v6

ghs259-GW2.21:37:35(config-sand-load-balance-profile-mcast-v6)#      ecmp hash key shift 3

ghs259-GW2.21:37:57(config)#port-channel load-balance sand profile mcast-v6

ghs259-GW2.21:39:50#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     220.1   0.2%       35

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     316.9   0.3%       50

...

Clearly, not much help. 

3) hash seed

ghs259-GW2.21:42:47(config-load-balance-policies)#load-balance sand profile mcast-v6

! profile mcast-v6 is the current global profile

ghs259-GW2.21:42:52(config-sand-load-balance-profile-mcast-v6)#no ecmp hash key shift

ghs259-GW2.21:43:03(config-sand-load-balance-profile-mcast-v6)#ecmp hash seed 1001

ghs259-GW2.21:43:25(config-sand-load-balance-profile-mcast-v6)#end

ghs259-GW2.21:43:36#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     220.2   0.2%       35

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     316.9   0.3%       50

...

4) hash polynomial + Hardware load-balancing for ingress/egress and fabric/egress replication (4.18.0F)

ghs259-GW2.21:45:26(config)#load-balance policies

ghs259-GW2.21:45:30(config-load-balance-policies)#load-balance sand profile mcast-v6

! profile mcast-v6 is the current global profile

ghs259-GW2.21:45:35(config-sand-load-balance-profile-mcast-v6)#no ecmp hash seed 1001

ghs259-GW2.21:45:42(config-sand-load-balance-profile-mcast-v6)#port-channel hash polynomial 5

ghs259-GW2.21:45:43(config-load-balance-policies)#    port-channel load-balance sand replication egress

ghs259-GW2.21:46:52#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     265.3   0.3%       42

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     272.3   0.3%       43

Nice! The hash polynomial works! Actually EOS 8 polynomial functions, the #5 works. 

Understanding the output of "show qos interface eth#"

In this post, I like to explain the output of the EOS command - "show qos interface eth#" based on my understanding from EOS document. 

wa461.00:58:25#sh qos interfaces e17/1
Ethernet17/1:
   Trust Mode: DSCP
   Default COS: 0
   Default DSCP: 0

   Port shaping rate: disabled
   Burst-size: disabled

  Tx    Bandwidth         Shape Rate               Burst-Size          Priority   ECN/WRED
 Queue  (percent)          (units)                  (units)
 ------------------------------------------------------------------------------------------
   7      - / -       - / -          ( - )           -  /  -           SP / SP       D
   6      - / -       - / -          ( - )           -  /  -           SP / SP       D
   5      - / -       - / -          ( - )           -  /  -           SP / SP       D
   4      - / -       - / -          ( - )           -  /  -           SP / SP       D
   3      - / -       - / -          ( - )           -  /  -           SP / SP       D
   2     20 / 20    1.2 / 1.0        (Gbps)     2048 KB / 2048 KB      RR / RR       D
   1     30 / 30      - / -          ( - )           -  /  -           RR / SP       D
   0     50 / 50      - / -          ( - )           -  /  -           RR / SP       D

Note: Values are displayed as Operational/Configured
Legend:
RR -> Round Robin
SP -> Strict Priority
 - -> Not Applicable / Not Configured
 % -> Percentage of line rate

• Values are displayed as Operational/Configured, like RR/SP which means this Q is configured as strict priority but operational as round-robin. 
• If one queue is configured as no priority (RR), then all the lower queues are changed to RR. 
• In this example, Q 2 is RR, then 0 and 1 are automatically changed to RR. 
• And Q 0 and 1 are RR/SP, which means their configuration are SP by default, but operational mode is RR.
• If both interface and tx-queue are configured with shape, which is effective? 
• From EOS manual chapter 27.5 - Enabling port shaping on an FM6000 interface disables queue shaping internally. Disabling port shaping restores queue shaping as specified in running-config.

interface Ethernet17/1
   speed forced 10000full
   !
   tx-queue 0
      bandwidth percent 50
   !
   tx-queue 1
      bandwidth percent 30
   !
   tx-queue 2
      no priority
      bandwidth percent 20
      shape rate 1000000

• Bandwidth vs shape.  
• Bandwidth% is the b/w percent this RR queue can get. Says the above configuration:
• In the sample below, the interface 17/1 is 10 Gbps interface 
• Q3-7 are the strict priority and, say use total 2 Gbps traffic, which left 8Gbps for Q0-2
• The tx-Q 2 can have 20% of left-over capacity which is 1.6Gbps
• But the shape rate is 1.2Gbps
• So the maximum throughput of tx-Q 2 is 1.2 Gbps, even it is assigned with 1.6Gbps.

EOS: A simple Qos design example

This article - "A Simple Quality of Service Design Example" is a very good starting point for understanding the EOS Qos architecture and starting a Qos design. 

Some points:

• 3 ways in the ingress points to map packets to Tx queues:
• qos cos trust + cos-tc map
• qos dscp trust + dscp-tc map
• service-policy + policy-map
• 3 big categories of traffic:
• network-control = control plane
• latency/jitter sensitive traffic
• best-efforts = scavenger traffic
• qos profile = 
• policy-map for input
• tx-queue set for output
• "no priority" in a tx-queue, all lower queues become RR