MPLS Hashing: Control Word and Entropy Label

A good presentation on this topic: https://ripe66.ripe.net/presentations/141-RIPE66_Understanding_MPLS_Hashing-Snijders.pdf

• MPLS Lable doesn't have encapsulated protocol information. So after the bottom label, 2 possibilities:
• IPv4/v6 packets if L3VPN, the 1st 4-bit is ipVersion, 4=ipv4, 6=ipv6
• Ethernet frames if L2VPN, the 1st 48-bit is dstMAC
• Along the LSP, if P routers like to do ECMP by hashing, it has to guess like above to do hashing the 5-tuple fields (src/dstAddr, src/dstPort, protoNum)
• So the problem happens if the dstMac starts with 4 or 6

The solution for the above issue is MPLS control word (RFC 4385, Feb 2006), a 4-byte all-zero (same length as a label) control word after the bottom label. 

• Arista EOS 4.21.3F: Configurable L2 EVPN MPLS control word
• mpls evpn; label-stack control-word
• Arista EOS 4.22.1F: Control Word support on LDP Pseudowire
• mpls ldp; pseudowire <name>; control-word

So the MPLS control word is to prevent incorrect hashing. How does the LSR do the load-balancing? The answer is Entropy Label (RFC 6790, Nov 2012)

• Increase the label stack by 2: 1 for ELI (entropy label indicator = 7), 1 for EL (entropy label)
• Aristea EOS 4.26.0F: LDP Entropy Label Support
• 4.25.2F: LSR support
• 4.26.0F: LER support
• mpls ldp; entropy-label
• This feature has a limitation: PHP LSR can't pop ELI + EL labels, so has to rely on LER to pop up [ELI, ELI, VPN] labels. 

Another solution is FAT - Flow Aware Transport on LDP pseudowires (4.24.0F) based on RFC 6391

• mpls ldp; pseudowire; pseudowire <name>; label flow

Troubleshooting Arista EOS EVPN VPWS (1)

This blog provides troubleshooting information for common problems with Arista EVPN MPLS VPWS. It covers the following error:

• Status: CLI conflict
• Status: Admin down
• Status: Interface unavailable
• Status: No remote
• Status: Unprogrammed local connector

Most of the issues start with down state in the output of "show patch panel" like below:

RR1-PE5-wa465.11:42:56#sh patch panel
Patch                        Connector                                               Status
---------------------------- ------------------------------------------------------- ------
pbVpws-PE5:e471--PE1:e1312   1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312       Down
                             2: Ethernet47/1

And command - "show patch panel <name> details" shows the detailed error reason:

RR1-PE5-wa465.14:58:02#sh patch panel pbVpws-PE5:e471--PE1:e1312 detail

......

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

   Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

      Status: Interface unavailable

      Local MPLS label: 100004

      EVPN VPWS type: port-based

   Connector 2: Ethernet47/1

      Status: Interface mode

The following error status means a local misconfiguration most likely.

• Ethernet side:
• Status: CLI conflict
• Possible reason: duplicated patch panel entries for the same ethernet interface
• Status: Admin down
• Check the interface status, to make sure is up
• Pseudowire side:
• Status: Interface unavailable
• Possible reason: make sure "no switchport" under interface

After the local configuration is corrected, check the output of "show bgp evpn route-type auto-dis next-hop 0.0.0.0". This is to check if the locally originated BGP EVPN prefixes are advertised out to BGP peers. 

RR1-PE5-wa465.14:58:14#show bgp evpn route-type auto-discovery next-hop 0.0.0.0

....

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.5:100 auto-discovery 50471 0000:0000:0000:0000:0000

                                 -                     -       -       0       i

RR1-PE5-wa465.15:07:10#show bgp evpn route-type auto-discovery next-hop 0.0.0.0 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 50471 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.5:100

 Paths: 1 available

  Local

    - from - (0.0.0.0)

      Origin IGP, metric -, localpref -, weight 0, valid, local, best

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

      MPLS label: 100004

The most important information as highlighted above is the ETID/PW ID. And compare it with the definition of PW  PE5:e471--PE1:e1312

RR1-PE5-wa465.15:06:29#show bgp evpn instance vpws custAB

...

    Pseudowire PE5:e471--PE1:e1312

      Status: up

      VPWS label: 100004

      Local VPWS ID: 50471

      Remote VPWS ID: 101312

After this point, you are pretty sure this end of PW is good. 

But if you are seeing the error status as "No remote", the device may have an issue with remote PW. 

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

   Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

      Status: No remote

      Local MPLS label: 100004

      EVPN VPWS type: port-based

   Connector 2: Ethernet47/1

      Status: Unprogrammed local connector

Now let's check the router knows how to reach PW with VPWS ID 101312. 

RR1-PE5-wa465.15:13:06#show bgp evpn route-type auto-discovery | grep 101312

RR1-PE5-wa465.15:13:10# 

Clearly, the router has no EVPN AD prefix with ETID 101312, so it doesn't how to reach. The next step is to log in to the other end of PW to figure out why by following the above steps. For example, the remote PW should be from NH 100.250.250.11

PE3-lp232.22:00:03#sh bgp evpn route-type auto-discovery next-hop 100.250.250.11

...

          Network                Next Hop              Metric  LocPref Weight  Path

 * >Ec   RD: 100.250.250.11:100 auto-discovery 101331 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i Or-ID: 100.250.250.11 C-LST: 100.250.250.6

From the above output, the PE1(100.250.250.11) sends out an AD prefix with ETID 101331, not 101311. 

And sometimes, you may face traffic issues even all the PW and patch panels are UP!! That's because the BGP EVPN AD prefixes only contain the local ETID/PW id, no the expected remote one. There is no way to ensure consistency. 

Reference:

• https://eos.arista.com/eos-4-24-2f/evpn-mpls-virtual-private-wire-service-vpws/

"Pending BGP Convergence" and invalid on MPLS/SR/EVPN BGP RR

On a BGP EVPN RR, I am seeing some EVPN routes are marked as "Pending BGP Convergence" as below:

RR1.12:01:04#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
   %     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i
   %     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
                                 100.250.250.12        -       100     0       i

The reason is "invalid" in the output of show bgp evpn detail

RR1.12:10:27#sh bgp evpn route-type auto-discovery detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.11 from 100.250.250.11 (100.250.250.11)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

BGP routing table entry for auto-discovery 120100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.12:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.12 from 100.250.250.12 (100.250.250.12)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

Here is the BGP configuration:

RR1.11:59:52#sh run sec router bgp

router bgp 65000

   router-id 100.250.250.5

   no bgp default ipv4-unicast

   neighbor RRC peer group

   neighbor RRC remote-as 65000

   neighbor RRC update-source Loopback0

   neighbor RRC route-reflector-client

   neighbor RRC send-community extended

   neighbor 100.250.250.11 peer group RRC

   neighbor 100.250.250.12 peer group RRC

   neighbor 100.250.250.21 peer group RRC

   neighbor 100.250.250.22 peer group RRC

   !

   address-family evpn

      neighbor RRC activate

Add "neighbor default encapsulation mpls next-hop-self source-interface Loopback0" under EVPN AF

RR1.12:20:00(config)#router bgp 65000

RR1.12:20:02(config-router-bgp)#address-family evpn

RR1.12:20:04(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface Loopback0

One prefix shows "* >" - valid. One shows no code

RR1.12:20:10#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

         RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i

 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

                                 100.250.250.12        -       100     0       i

Details still shows "invalid"

RR1.12:22:09#sh bgp evpn route-type auto-discovery rd 100.250.250.11:100 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.11 from 100.250.250.11 (100.250.250.11)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

      MPLS label: 132768

No tunnel endpoint in "show tunnel rib brief"

RR1.12:22:50#show tunnel rib brief | grep 250.250.1

   100.250.250.1/32               IS-IS SR IPv4       6               65                      115               20

   100.250.250.12/32              IS-IS SR IPv4       4               65                      115               40

The cause is the PE's config, missing node-segment ipv4 configuration

PEa1.12:10:13#sh run int lo0

interface Loopback0

   ip address 100.250.250.11/32

   ipv6 address 2000:100:250:250::11/128

   node-segment ipv6 index 11

   isis enable MplsSR

   isis passive

PEa1.12:25:24#c

PEa1.12:26:08(config)#int lo0

PEa1.12:26:10(config-if-Lo0)#node-segment ipv4 index 11

On RR1, NH emerges in tunnel rib. 

RR1.12:22:56#show tunnel rib brief | grep 250.250.1

   100.250.250.1/32        IS-IS SR IPv4       6               65                      115               20

   100.250.250.11/32       IS-IS SR IPv4       2               65                      115               30

   100.250.250.12/32       IS-IS SR IPv4       4               65                      115               40

And bgp table is good

RR1.12:24:51#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i

 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

                                 100.250.250.12        -       100     0       i

Arista EOS: % - Pending BGP convergence when "show bgp evpn"

1. Router(RR) receives 4 bgp evpn prefixes from RRC

wa465-PD.P1-JHM.SR.14:35:52(config-router-bgp-af)#show bgp evpn summary
BGP summary information for VRF default
Router identifier 100.250.250.30, local AS number 65000
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  100.250.250.11   4 65000            500       487    0    0 00:02:15 Estab   4      4
  100.250.250.12   4 65000            504       488    0    0 00:02:15 Estab   4      4

2. But "show bgp evpn" complains about "Pending BGP convergence"

wa465-PD.P1-JHM.SR.14:36:48(config-router-bgp-af)#show bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.30, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

   %     RD: 100.250.250.12:100 auto-discovery 0 0000:0000:0000:1111:0000

                                 100.250.250.12        -       100     0       i

   %     RD: 100.250.250.12:1 auto-discovery 0000:0000:0000:1111:0000

                                 100.250.250.12        -       100     0       i

3. Check the NH, it was resolved in the tunnel rib

wa465-PD.P1-JHM.SR.14:37:33(config-router-bgp-af)#show tunnel rib brief | grep 100.250.250.12

   100.250.250.12/32       IS-IS SR IPv4       3               65                      115               20

4. The cause is the mis-configuration under router bgp evpn address family. The EOS needs to know to resolve the NH by using the mpls NH tunnel rib. 

wa465-PD.P1-JHM.SR.14:36:53(config-router-bgp-af)#router bgp 65000

wa465-PD.P1-JHM.SR.14:36:57(config-router-bgp)#add evpn

wa465-PD.P1-JHM.SR.14:37:00(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface loopback 0

wa465-PD.P1-JHM.SR.14:37:25(config-router-bgp-af)#show bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.30, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.12:100 auto-discovery 0 0000:0000:0000:1111:0000

                                 100.250.250.12        -       100     0       i

 * >     RD: 100.250.250.12:1 auto-discovery 0000:0000:0000:1111:0000

ECMP in MPLS L3 VPN

Consider the above topology

• CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
• On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
• Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1. 

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

 Paths: 1 available

  65100 65101

    11.11.11.11 from 1.1.1.1 (1.1.1.1)

      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

      Extended Community: Route-Target-AS:65000:1

      Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

 Paths: 1 available

  65100 65101

    11.11.11.11 from 1.1.1.1 (1.1.1.1)

      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

      Extended Community: Route-Target-AS:65000:4364

      Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

 B I      12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

                                 via 1.0.0.8, Ethernet2/1, label 100000

                              via 11.11.11.11/32, LDP tunnel index 1, label 118012

                                 via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name             VRF ID Table ID

cust_1               15        65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

             vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

 ---------------------------------------------------------------------------------------------------------

|                                 Routing Table                                           |              |

|---------------------------------------------------------------------------------------------------------

|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel

| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value

 ---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24       |ROUTE| FEC 32831          |0    |  -    |                   |300  |  D   |M 116707

|15 |12.0.0.0/24       |ROUTE| FEC 32831          |0    |  -    |                   |300  |  D   |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries. 

PE1#show hardware capacity | grep -i ECMP

ECMP                                                   875      21%        3220             0          4095         875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc

  1.1.1.1          4  65000           1693      1673    0    0 23:25:15 Estab   1602   1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage. 

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip 

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP                                                     0       0%        4095             0          4095         875

ECMP              Mpls                                   0       0%        4095             0          4095           0

ECMP              Routing                                0       0%        4095             0          4095         875

ECMP              VxlanOverlay                           0       0%        4095             0          4095           0

ECMP              VxlanTunnel                            0       0%        3891             0          3891           0

Now no ECMP anymore in the sw/hw routing table. 

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

 B I      12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

                                 via 1.0.0.8, Ethernet2/1, label 100000

                              via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

                                 via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

 ---------------------------------------------------------------------------------------------------------

|                                 Routing Table                                           |              |

|---------------------------------------------------------------------------------------------------------

|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel

| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value

 ---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24       |ROUTE| FEC 32830          |0    |  -    |                   |  -  |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels. 

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc

....

  192.158.115.11   4  65100           1568      1819    0    0    1d00h Estab   800    800

  192.168.115.11   4  65100           1749      1827    0    0    1d00h Estab   802    802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

8

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels. 

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

    Local Label: 16694 (with rewrite);

      Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

    Local Label: 16695 (with rewrite);

      Received Label 116390

Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted. 

Feb  4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
 Paths: 2 available
  65000 65101
    192.168.66.7 from 192.168.0.7 (192.168.0.7)
      Origin IGP, metric -, localpref 100, weight 0, valid, external, best
      Extended Community: Route-Target-AS:65000:1
      Remote MPLS label: 116396
      Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label |  Action  |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index| Action
|186827 |Forward   | DROP               |0    |  -    |                   |16434|358104|   -
|186827 |Forward   | DROP               |0    |  -    |                   |16434|358105|   -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table             Feature           Chip              Used     Used        Free     Committed     Best Case        High
EEDB              MplsTunnel        Jericho0          93206       99%       1002          4096         98304        93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  5.5.5.5          4  65000            421       405    0    0 05:20:01 Estab   801    801
  192.158.115.11   4  8073             381       439    0    0 05:30:45 Estab   800    800
  192.168.115.11   4  8073             410       439    0    0 05:29:12 Estab   802    802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels. 

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table             Feature           Chip              Used     Used        Free     Committed     Best Case        High
EEDB              MplsTunnel        Jericho0             0       0%       77824             0         81920           0

Here is a list of the Jericho-family models:
7280R/Jericho  : 80K
7280R2/Jericho+: 96K

Arista MPLS VPN Step by Step (2c) L2VPN - Tcpdump

One powerful tool in Arista/EOS is tcpdump. In this setup, for example, if you see some traffic issue and want to see if the PE sending out the packets, you can do the tcpdump this way:

If you want to check the type 5 pw between CE1 and CE2, 

Step 1: find out the mpls ldp tunnel

PE1(config)#sh patch panel forwarding
In/Out                     Type CW VLAN Status Patch
-------------------------- ---- -- ---- ------ --------------------
....
Et47/3                        5         Up     ce1-ce2-pw5-1
   Label 132770, LDP Tun 5  <<< tunnel 5

Step 2: check mpls ldp tunnel 5 fwding info so have the interface name

PE1(config)#sh mpls ldp tunnel 5
   Index       Endpoint              Nexthop         Interface      Labels
----------- --------------------- --------------- ----------------- ----------
   5           88.255.22.22/32       88.1.11.1       Ethernet3/1    [ 100005 ]

Step 3: create the monitor session to dump packet to cpu

monitor session ldp-et3-1 source Ethernet3/1

monitor session ldp-et3-1 destination Cpu

Step 4: tcpdump the packet

bash tcpdump -nevvi mirror0 ether proto 0x8847 -w /mnt/flash/ldp.pcp

Arista MPLS VPN Step by Step (2c) L2VPN - Type 4 PW, MTU

In the LDP PW configuration, one thing needs to be careful is the MTU. You have to ensure the MTU along the underlying LSP matched. Here is an example:

So we have OSPF and ISIS enabled on VLAN 301/302 on CEs. But only OSPF up, no ISIS neighbor at all

CE1#sh ip os neighbor vrf ce1-pw4-301
Neighbor ID     VRF      Pri State                  Dead Time   Address         Interface
101.255.2.2     ce1-pw4-301 1   FULL/DR                00:00:32    101.101.101.2   Vlan301
CE1#sh isis neighbors vrf ce1-pw4-302
CE1#

Well the ping works as expected, CE1 can ping CE2's vlan 302 address - 102.102.102.2

CE1#ping vrf ce1-pw4-302 102.102.102.2
PING 102.102.102.2 (102.102.102.2) 72(100) bytes of data.
80 bytes from 102.102.102.2: icmp_seq=1 ttl=64 time=0.276 ms
80 bytes from 102.102.102.2: icmp_seq=2 ttl=64 time=0.119 ms
80 bytes from 102.102.102.2: icmp_seq=3 ttl=64 time=0.112 ms
80 bytes from 102.102.102.2: icmp_seq=4 ttl=64 time=0.111 ms
80 bytes from 102.102.102.2: icmp_seq=5 ttl=64 time=0.110 ms

--- 102.102.102.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.110/0.145/0.276/0.066 ms, ipg/ewma 0.202/0.208 ms

Both sides configured with MTU 9000

CE1#sh run int vlan 302
interface Vlan302
   mtu 9000
   vrf forwarding ce1-pw4-302
   ip address 102.102.102.1/24
   isis enable isis-ce1-pw4-302

CE2#sh run int vlan 302
interface Vlan302
   mtu 9000
   vrf forwarding ce2-pw4-302
   ip address 102.102.102.2/24
   isis enable isis-ce2-pw4-302

Let's try ping the destination with size of data plus DF-bit

CE1#ping vrf ce1-pw4-302 102.102.102.2 size 9000 df-bit
PING 102.102.102.2 (102.102.102.2) 8972(9000) bytes of data.
--- 102.102.102.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Ok, it fails but where is the culprit?

PE1 and PE2 are good:

PE1(config-if-Et47/2)#sh ip int brief | grep 88
Ethernet3/1            88.1.11.11/24      up         up              9082
Ethernet45/3           88.3.11.11/24      down       down            9082

PE2(config-if-Et51/1)#sh ip int brief | grep 88
Ethernet33/1           88.4.22.22/24      up         up              9082
Ethernet51/1           88.2.22.22/24      up         up              9082

Let's move on to P routers:

P1#sh ip int brief | grep 88
Ethernet35/1           88.1.4.1/24        up         up              8000
Ethernet51/1           88.1.2.1/24        up         up              8000
Ethernet52/1           88.1.3.1/24        up         up              9082
Ethernet54/1           88.1.11.1/24       up         up              9082

Ok, here is the issue. Let's correct them and ISIS neighbor is up!

CE1#sh isis neighbors vrf ce1-pw4-302

Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id
isis-ce1- ce1-pw4- CE2              L2   Vlan302            44:4c:a8:11:6b:e1 UP    8           CE2.02

So, the reason is, the ISIS pads hello packet upto MTU and the oversized packets will be dropped siliently on underlying P routers. Hence ISIS fails to form neighbor. 

The OSPF just negotiates the MTU of connecting interfaces, which are 9000 in both sides, so ospf neighbor is up w/o any issues. But if you have a large LSA database, and ospf utilize the full MTU to transmit it, you will hit this MTU mismatch later, but not in the neigobor discoverage stage. 

Arista MPLS VPN Step by Step (2b) L2VPN - Type 5 PW

This blog covers type 5 pw configuration. Here is the topology:

Configuration-wise, type 5 is simpler than type 4. And between 2 CEs, it is just a like direct ethernet link. From above topology, there is 2 vlans between CEs over 1 pw. 

PE1 Configuration:

mpls ldp
   router-id 88.255.11.11
   transport-address interface Loopback88
   fec filter prefix-list fec-pfx-list
   no shutdown
   !
   pseudowires
      pseudowire to-ce2-type5-pw
         neighbor 88.255.22.22
         pseudowire-id 303
         mtu 9082
!
patch panel
   patch ce1-ce2-pw5-1
      connector 1 interface Ethernet47/3
      connector 2 pseudowire ldp to-ce2-type5-pw
!
interface Ethernet47/3
   description mpls.ldp5pw.pe1=>ce1.et51/3
   !! PE to CE interface MUST be routed port !!
   no switchport

L2 Control Packets

One different between type4 and 5 is the forwarding behavior of L2 control packets. From above link, the LLDP is forwarded. So "show lldp neighbor" on CE1 shows CE2, not PE1

CE1(config)#ln
Et51/1     PE1                              Ethernet47/1               120
Et51/2     PE1                              Ethernet47/2               120
Et51/3     PE1                              Ethernet47/3               120
Et51/3     CE2                              Ethernet52/3               120

Arista MPLS VPN Step by Step (2a) L2VPN - Type 4 PW

Based on the MPLS LDP configuration completed in the previous session, I will talk about the MPLS LDP type 4 pseudowire configuration.

Regarding the LDP/BGP/IGP configuration, please refer to this blog.

Here is the topology:



Here we have 2 pseudowires: ce1-ce2-pw4-vlan-301 and ce1-ce2-pw4-vlan-302 to L2 connectivity between CE1 and CE2 across LDP cloud. From CEs' perspective, the pw works like a directly connected L2 link. So the VLANs are in same subnet and running IGPs (OSPF/ISIS) on the top of it. 

PE 1 configuration:

interface Ethernet3/1
   description mpls.pe1=>p1:et54/1
   mtu 9082
   no switchport
   ip address 88.1.11.11/24
!
interface Ethernet45/3
   description mpls.pe1=>p3:et45/3
   mtu 9082
   no switchport
   ip address 88.3.11.11/24
!
interface Ethernet47/2
   description mpl1.pe1=>ce1:et51/1
   mtu 9082
   ! pw intf to CE1 must be routed port, otherwise pw down !
   no switchport
!
interface Loopback88
   ip address 88.255.11.11/32
!
mpls ip
!
mpls ldp
   ! mpls ldp config is same as mpls ldp l3vpn !
   router-id 88.255.11.11
   transport-address interface Loopback88
   fec filter prefix-list fec-pfx-list
   no shutdown
   !
   pseudowires
      pseudowire to-ce2-vlan-301
         ! 88.255.22.22 is PE2's ldp/pw loopback ip addr !
         neighbor 88.255.22.22
         pseudowire-id 301
         mtu 9082
      !
      pseudowire to-ce2-vlan-302
         neighbor 88.255.22.22
         pseudowire-id 302
         mtu 9082
!
patch panel
   patch ce1-ce2-pw4-vlan-301
      ! note, e47/2 is cfg'ed as routed port, NOT sw/trunk port !
      connector 1 interface Ethernet47/2 dot1q vlan 301
      connector 2 pseudowire ldp to-ce2-vlan-301
   !
   patch ce1-ce2-pw4-vlan-302
      connector 1 interface Ethernet47/2 dot1q vlan 302
      connector 2 pseudowire ldp to-ce2-vlan-302

PE 2 configuration:

interface Ethernet33/1

   description pe2-p4=>p4.et33/1

   mtu 9082

   no switchport

   ip address 88.4.22.22/24

!

interface Ethernet37/2

   description pe2-ce2=>ce2.et52/2

   mtu 9082

   no switchport

!

interface Ethernet51/1

   description pe2-p2=>p2.et51/1

   mtu 9082

   no switchport

   ip address 88.2.22.22/24

!

interface Loopback88

   ip address 88.255.22.22/32

!

mpls ip

!

mpls ldp

   router-id 88.255.22.22

   transport-address interface Loopback88

   fec filter prefix-list fec-pfx-list

   no shutdown

   !

   pseudowires

      pseudowire to-ce1-vlan-301

         neighbor 88.255.11.11

         pseudowire-id 301

         mtu 9082

      !

      pseudowire to-ce1-vlan-302

         neighbor 88.255.11.11

         pseudowire-id 302

         mtu 9082

!

patch panel

   patch ce1-ce2-pw4-vlan-301

      connector 1 interface Ethernet37/2 dot1q vlan 301

      connector 2 pseudowire ldp to-ce1-vlan-301

   !

   patch ce1-ce2-pw4-vlan-302

      connector 1 interface Ethernet37/2 dot1q vlan 302

      connector 2 pseudowire ldp to-ce1-vlan-302

For the CE's configuration, please note that I use VRF on CE to simulate multiple hosts. 

CE1 configuration:

vrf definition ce1-pw4-301

   rd 301:1

!

vrf definition ce1-pw4-302

   rd 302:1

!

interface Ethernet51/2

   description mpls.ce1=>pe1:et47/2

   ! compare with PE1's et47/2, here is sw + trunk !

   switchport trunk allowed vlan 301-302

   switchport mode trunk

!

interface Vlan301

   vrf forwarding ce1-pw4-301

   ip address 101.101.101.1/24

!

interface Vlan302

   mtu 9000

   vrf forwarding ce1-pw4-302

   ip address 102.102.102.1/24

   isis enable isis-ce1-pw4-302

!

interface Loopback301

   vrf forwarding ce1-pw4-301

   ip address 101.255.1.1/24

!

interface Loopback302

   vrf forwarding ce1-pw4-302

   ip address 102.255.1.1/24

   isis enable isis-ce1-pw4-302

!

ip routing vrf ce1-pw4-301

ip routing vrf ce1-pw4-302

!

router isis isis-ce1-pw4-302 vrf ce1-pw4-302

   net 47.0002.0001.0001.0001.00

   is-type level-2

   !

   address-family ipv4 unicast

!

router ospf 301 vrf ce1-pw4-301

   network 101.0.0.0/8 area 0.0.0.0

   max-lsa 12000

CE2 configuration:

interface Ethernet52/2

   description mpls.ce2=>pe2:et37/2

   switchport trunk allowed vlan 301-302

   switchport mode trunk

!

vrf definition ce2-pw4-301

   rd 301:2

!

vrf definition ce2-pw4-302

   rd 302:2

!

interface Loopback301

   vrf forwarding ce2-pw4-301

   ip address 101.255.2.2/24

!

interface Loopback302

   vrf forwarding ce2-pw4-302

   ip address 102.255.2.2/24

   isis enable isis-ce2-pw4-302

!

interface Vlan301

   vrf forwarding ce2-pw4-301

   ip address 101.101.101.2/24

!

interface Vlan302

   mtu 9000

   vrf forwarding ce2-pw4-302

   ip address 102.102.102.2/24

   isis enable isis-ce2-pw4-302

!

ip routing vrf ce2-pw4-301

ip routing vrf ce2-pw4-302

!

router isis isis-ce2-pw4-302 vrf ce2-pw4-302

   net 47.0002.0002.0002.0002.00

   is-type level-2

   !

   address-family ipv4 unicast

!

router ospf 301 vrf ce2-pw4-301

   network 101.0.0.0/8 area 0.0.0.0

   max-lsa 12000