Arista EOS/Networking Tips by Solomon Yang

4/25/2020

ECMP in MPLS L3 VPN

Consider the above topology

CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1.

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:1

Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:4364

Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012

via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name VRF ID Table ID

cust_1 15 65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 116707

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries.

PE1#show hardware capacity | grep -i ECMP

ECMP 875 21% 3220 0 4095 875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

1.1.1.1 4 65000 1693 1673 0 0 23:25:15 Estab 1602 1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage.

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP 0 0% 4095 0 4095 875

ECMP Mpls 0 0% 4095 0 4095 0

ECMP Routing 0 0% 4095 0 4095 875

ECMP VxlanOverlay 0 0% 4095 0 4095 0

ECMP VxlanTunnel 0 0% 3891 0 3891 0

Now no ECMP anymore in the sw/hw routing table.

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32830 |0 | - | | - |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels.

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

192.158.115.11 4 65100 1568 1819 0 0 1d00h Estab 800 800

192.168.115.11 4 65100 1749 1827 0 0 1d00h Estab 802 802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels.

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

Local Label: 16694 (with rewrite);

Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

Local Label: 16695 (with rewrite);

Received Label 116390

4/24/2020

Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted.

Feb 4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
Paths: 2 available
65000 65101
192.168.66.7 from 192.168.0.7 (192.168.0.7)
Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Extended Community: Route-Target-AS:65000:1
Remote MPLS label: 116396
Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label | Action | Destination | VID |Outlif | MAC / CPU Code |Index| Index| Action
|186827 |Forward | DROP |0 | - | |16434|358104| -
|186827 |Forward | DROP |0 | - | |16434|358105| -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 93206 99% 1002 4096 98304 93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
5.5.5.5 4 65000 421 405 0 0 05:20:01 Estab 801 801
192.158.115.11 4 8073 381 439 0 0 05:30:45 Estab 800 800
192.168.115.11 4 8073 410 439 0 0 05:29:12 Estab 802 802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels.

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 0 0% 77824 0 81920 0

Here is a list of the Jericho-family models:
7280R/Jericho : 80K
7280R2/Jericho+: 96K

3/07/2020

Arista EOS MTU Behavior (3) - L3 Routed Ethernet Ports, IPv4

Now we switched to L3 routed ethernet (IPv4) ports:

MTU enforcement is for egress traffic
Hardware programs the MTU value (default value is 1500, in this case, 2000) and checks against forwarding packet length (in this case, it is ipv4 packet).
For IPv4 packets, if the length is more than MTU, the packets will be fragmented by CPU.
And the number of fragmented and dropped packets are reflected by CoPPEgressTrap counter in the output of "show cpu counters queue summary | nz"

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system,

Use EOS tool - Ethxmit on the generator to send packets

--ip-dst pointing to the destination address on the receiver device
--size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port

Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

For using ethxmit:

--size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC.
If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC

No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output.

Trident-based system would use 9214.

R2#sh int e47/1 | grep -i mtu
Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
mtu 100
switchport access vlan 100

3/03/2020

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

1. MTU enforcement happens on the egress interfaces.

So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces

2. Arista MTU is layer-3 MTU on Jericho chipset:

2.1 The MTU enforcement is only for routed packets, no action on switched traffic
2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface

3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

2/18/2020

MTU value in EOS interop with Cisco XR

Summary:

Arista mtu value under interface is IP MTU value, while Cisco XR's mtu is L2 MTU
So when ISIS peering fails to come up if both sides have same MTU value
XR's mtu = EOS's mtu + 14 (Ethernet header length)

https://eos.arista.com/forum/isis-between-arista-and-ios-xr/

1/29/2020

Tunnel RIB related features

Self-Defined Tunnel Rib TOI (4.22.1F)
Pseudowire Resolution Over User-defined Tunnel RIBs (4.22.1F)

May provide a sample later on how to use these 2 features.

1/17/2020

Arista BFD related TOIs

4.23*

HW BFD

https://eos.arista.com/eos-4-23-1f/hardware-accelerated-bfd-transmit/

4.22*

BFD C-bit

https://eos.arista.com/eos-4-22-1f/bfd-control-plane-independent-interoperability/

BFD Authentication

https://eos.arista.com/eos-4-22-0f/bfd-authentication/

4.20*

BFD SSO:

https://eos.arista.com/eos-4-18-2f/bfd-sso/
4.18.2F, starting with Arad Modular
4.20.5: Add Jericho

BFD for static routes

4.17*

BFD Enhancement:

https://eos.arista.com/eos-4-17-0f/bfd-enchancements/
BFD under VRF

OSPFv3 BFD:

https://eos.arista.com/eos-4-17-0f/ospfv3-bfd

4.15*

ISIS BFD

https://eos.arista.com/eos-4-15-2f/isis-bfd

BFD NSX Monitoring:

https://eos.arista.com/eos-4-15-0f/nsx-sn-health-monitoring/

12/28/2019

%SAND-6-VXLAN_ROUTE_IGNORED error message

If you see the syslog error message - %SAND-6-VXLAN_ROUTE_IGNORED like below (on arista 7280E/R series) , that's because your hardware TCAM profile is not correct.

Dec 28 16:02:16 eVtep2 SandL3Unicast: %SAND-6-VXLAN_ROUTE_IGNORED: VXLAN routes cannot be programmed in the data path with the current TCAM profile

Just change the configuration like:

hardware tcam profile vxlan-routing

11/21/2019

A dummy error - low MTU vs ipv6

After configuring IPv6 under an interface, see the ipv6 neighbor or host resolution doesn't work.

Router1.10:36:22(config-if-Po5)#sh ipv6 interface po5
Port-Channel5 is up, line protocol is up (connected)
IPv6 is stalled, link-local is unassigned
Global unicast address(es):
2000:170:1:170::1, subnet is 2000:170:1:170::/64 [INACTIVE]
No joined group addresses
ND DAD status is unavailable
ND Reachable time is 30000 milliseconds
ND retransmit interval is 1000 milliseconds

The reason is that, I have a line of a left-over configuration of smaller-than-normal mtu size - 1200. The lowest MTU size of IPv6 is 1280.

interface Port-Channel5
description Cdn_Test_upLink
mtu 1200
no switchport
vrf IxCdn
ip address 170.1.170.1/24
ipv6 nd cache expire 60
ipv6 enable
ipv6 address 2000:170:1:170::1/64
ipv6 nd ra disabled all
no mpls ip
no qos trust
spanning-tree bpdufilter enable

After removing this line, everything is fine.

Port-Channel5 is up, line protocol is up (connected)
IPv6 is enabled, link-local is fe80::464c:a8ff:fea5:1140/64
Global unicast address(es):
2000:170:1:170::1, subnet is 2000:170:1:170::/64
Joined group address(es):
ff02::1
ff02::1:ff00:1
ff02::1:ffa5:1140

11/05/2019

Arista EOS Segment Routing (3) - SR Routing #1, Static Route + NHG

The Arista MPLS Segment Routing White Paper(Page 8) lists 4 SR routing solutions:
1) Static label push via NHG
2) Controller via EOS SDK
3) BGP LU with SR
4) SR-TE

This blog gives an example of solution #1.

The configuration is straightforward:

Instead of running a routing protocol between PE1 and PE2, a static route of remote destination (100.255.4.1/32 on PE1) is configured and pointing to NHG.
The NHG defines:

MPLS encapsulation
Push a label of 900004
The NH is 10.1.2.2 for out-interface and destination MAC.
Obviously, 2 NHGs are needed to 2 way traffic.

Not like the regular NHG setup, decap group is NOT needed.
P1 and P2 have no idea of destination ip - 100.255.4.1, but use label to forward traffic

PE1#sh ip route 100.255.4.1

VRF: default

Codes: C - connected, S - static, K - kernel,

O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,

E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,

N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,

R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,

O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,

NG - Nexthop Group Static Route, V - VXLAN Control Service,

DH - DHCP client installed default route, M - Martian,

DP - Dynamic Policy Route, L - VRF Leaked

NG 100.255.4.1/32 [1/0] via sr-1-push-label-900004, Nexthop Group ID 1

The output of tcpdump on P2's et42/4 (the interface facing to P1) shows MPLS with the correct label in both directions.

23:10:17.713412 44:4c:a8:97:84:5f > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900001, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 6381, offset 0, flags [none], proto ICMP (1), length 100)

100.255.4.1 > 100.255.1.1: ICMP echo request, id 9258, seq 5, length 80

23:10:17.713473 44:4c:a8:97:8c:51 > 44:4c:a8:97:84:5f, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900004, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 2480, offset 0, flags [none], proto ICMP (1), length 100)

100.255.1.1 > 100.255.4.1: ICMP echo reply, id 9258, seq 5, length 80

Now you have the simplest, but completed ISIS segment-routing solution, control plane + data plane :-) In the real network, you can complete the puzzle by adding,

Have a controller participate in the ISIS domain via the ISIS over GRE tunnel.
As an ISIS neighbor, it can fetch the full ISIS LSDB to have a global view of the network including the segment.
Then it can program the edge router vi CLI/capi to steer the traffic. Or use the EOS SDK for faster program speed (this is the solution #2)

Arista EOS Segment Routing (2) - MPLS Ping to verify SR data plane

With the same topology, to verify the SR data plane, we can use the command - "ping mpls" to have PE1 send a UDP packet with the correct label.

PE1#ping mpls segment-routing ip 4.4.4.4/32 repeat 1
LSP ping to Segment-Routing route 4.4.4.4/32
timeout is 5000ms, interval is 1000ms
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
Reply from 10.3.4.4: seq=1, time=0.53ms, success: egress ok

--- Segment-Routing target fec 4.4.4.4/32 : lspping statistics ---
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
1 packets transmitted, 1 received, 0% packet loss, time 150ms
1 received from 10.3.4.4, rtt min/max/avg 0.530/0.530/0.530 ms

Turning on tcpdump on P1, it shows the following packets:

P1#sh run | grep moni
monitor session sr source Ethernet54/4
monitor session sr destination Cpu

P1#bash tcpdump -nvvvi mirror1 udp or mpls
tcpdump: listening on mirror1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:48:15.934459 44:4c:a8:97:77:21 > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 98: MPLS (label 900004, exp 0, [S], ttl 255)
(tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 80, options (RA))
10.1.2.1.36260 > 127.0.0.1.lsp-ping:
LSP-PINGv1, msg-type: MPLS Echo Request (1), length: 48
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: No return code or return code contained in the Error Code TLV (0)
Return Subcode: (0)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: no timestamp
Target FEC Stack TLV (1), length: 12
Unknown subTLV (31744), length: 6
0x0000: 0404 0404 2002
0x0000: 7c00 0006 0404 0404 2002 0000
13:48:15.934737 44:4c:a8:97:8c:51 > 44:4c:a8:97:77:21, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 36448, offset 0, flags [DF], proto UDP (17), length 60)
10.3.4.4.lsp-ping > 10.1.2.1.36260:
LSP-PINGv1, msg-type: MPLS Echo Reply (2), length: 32
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: Replying router is an egress for the FEC at stack depth 1 (3)
Return Subcode: (1)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: -16:-40:-1.401478080

From the above packet capture,

The icmp echo request is encap'ed in MPLS packet with label 90004 as expected.
The dest address of inner ip is 127.0.0.1. With destination address on network 127, the packet can't be routed if LSP is broken before the final destination and the packet is decap'ed prematurely.
The icmp echo reply is just a native ipv4 packet.

Arista EOS Segment Routing (1) - Simplest Setup/Configuration

Here is the simplest SR setup,
1) Only IPv4, only ipv4 node-segment;
2) No IPv6, no ISIS multi-topology

From the above configuration, we can observe:

The basic ISIS-SR configuration is quite simple:

enable "segment-routing mpls" under router isis to attach SR info to ISIS LSA;
Specify the loopback0 interface with a globally unique node-segment index.
ISIS-SR calculates the node-segment label = mpls label range base + index, so the routers in whole domain use the same label. This is an important characteristic of SR, for anycast.

To verify the SR setup, you can use the following CLI commands:

show mpls label ranges
show isis segment-routing
show isis database PE2.00-00 detail !PE2.00-00 is id
show mpls lfib route
show mpls segment-routing bindings
show platform jericho mpls route

Detailed output of PE1:

PE1#show mpls label ranges

Start End Size Usage

------------------------------------------------

0 15 16 reserved

16 99999 99984 static mpls

100000 116383 16384 ldp (dynamic)

116384 132767 16384 pseudowire (dynamic)

132768 149151 16384 bgp (dynamic)

149152 165535 16384 isis (dynamic)

165536 362143 196608 free (dynamic)

362144 899999 537856 unassigned

900000 965535 65536 isis-sr <<< default ISIS-SR range

900000 965535 65536 bgp-sr

965536 1031071 65536 srlb

1031072 1036287 5216 unassigned

1036288 1048575 12288 l2evpn

PE1#show isis segment-routing

System ID: PE1 Instance: sr

SR supported Data-plane: MPLS SR Router ID: 1.1.1.1

SR Global Block( SRGB ): Base: 900000 Size: 65536

Adj-SID allocation mode: SR-adjacencies

Adj-SID allocation pool: Base: 149152 Size: 16384

All Prefix Segments have : P:0 E:0 V:0 L:0

IS-IS Reachability Algorithm : SPF (0)

Number of IS-IS segment routing capable peers: 3

Self-Originated Segment Statistics:

Node-Segments : 1

Prefix-Segments : 0

Proxy-Node-Segments : 0

Adjacency Segments : 1

PE1#show isis database PE2.00-00 detail

IS-IS Instance: sr VRF: default

IS-IS Level 2 Link State Database

LSPID Seq Num Cksum Life IS Flags

PE2.00-00 12 57553 1049 L2 <>

NLPID: 0xCC(IPv4)

Hostname: PE2

Area address: 49.0001

Interface address: 4.4.4.4

Interface address: 10.3.4.4

IS Neighbor : P2.11 Metric: 10

LAN-Adj-sid: 100000 flags: [ L V ] weight: 0 system ID: 0000.0000.3333

Reachability : 4.4.4.4/32 Metric: 10 Type: 1 Up

SR Prefix-SID: 4 Flags: [ N ] Algorithm: 0

Reachability : 10.3.4.0/24 Metric: 10 Type: 1 Up

Router Capabilities: Router Id: 4.4.4.4 Flags: [ ]

SR Local Block:

SRLB Base: 965536 Range: 65536

Area leader priority: 250 algorithm: 0

SR Capability: Flags: [ I ]

SRGB Base: 900000 Range: 65536

PE1#show mpls lfib route

MPLS forwarding table (Label [metric] Vias) - 4 routes

MPLS next-hop resolution allow default route: False

Via Type Codes:

M - MPLS via, P - Pseudowire via,

I - IP lookup via, V - VLAN via,

VA - EVPN VLAN aware via, ES - EVPN ethernet segment via,

VF - EVPN VLAN flood via, AF - EVPN VLAN aware flood via,

NG - Nexthop group via

Source Codes:

G - gRIBI, S - Static MPLS route,

B2 - BGP L2 EVPN, B3 - BGP L3 VPN,

R - RSVP, LP - LDP pseudowire,

L - LDP, M - MLDP,

IP - IS-IS SR prefix segment, IA - IS-IS SR adjacency segment,

IL - IS-IS SR segment to LDP, LI - LDP to IS-IS SR segment,

BL - BGP LU, ST - SR TE policy,

DE - Debug LFIB

IA 149152 [1]

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900002 [1], 2.2.2.2/32

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900003 [1], 3.3.3.3/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900004 [1], 4.4.4.4/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

PE1#show mpls segment-routing bindings

1.1.1.1/32

Local binding: Label: imp-null

Remote binding: Peer ID: 0000.0000.2222, Label: 900001

2.2.2.2/32

Local binding: Label: 900002

Remote binding: Peer ID: 0000.0000.2222, Label: imp-null

3.3.3.3/32

Local binding: Label: 900003

Remote binding: Peer ID: 0000.0000.2222, Label: 900003

4.4.4.4/32

Local binding: Label: 900004

Remote binding: Peer ID: 0000.0000.2222, Label: 900004

PE1#show platform jericho mpls route

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------

| Mpls Table |

|---------------------------------------------------------------------------------------------|

|---------------------------------------------------------------------------------------------|

|149152 |Et2/4 |1027 | 44:4c:a8:97:8c:51 | PopE pipe auto|32773 |8200 |8 | 0 |

Egress Action Codes:

M - Mpls Tunnel, G - GRE Tunnel, MoG - Mpls-over-GRE Tunnel

ECMP Codes:

D - ECMP is divergent across switching chips

-----------------------------------------------------------------------------------------------

| Mpls Table

|-----------------------------------------------------------------------------------------------

|-----------------------------------------------------------------------------------------------

|900002 |Pop | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900003 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900004 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

10/28/2019

EOS - Find the interfaces with the most flaps

bn302.12:40:24#show int phy | egrep '^Eth' | awk '{print $3, $1}' | sort -rn | head -10
479 Ethernet9/25/2
23 Ethernet9/11/1
5 Ethernet9/34/1
5 Ethernet3/34/1

BGP Oscillation (RFC 3345)

BGP Oscillation is common in the context of MED + RR, because,

BGP only advertise the best path and hide the full list.
Similar to RR, it is also with BGP confederation

RFC 3345 has a very good depiction of how it happens

And I converted it to a google slides for a better view

in RFC, there lists a couple of design options to avoid this kind of churn

In RFC 7964, advertises all available paths by using ADD-PATH
Always compare MED even from different AS;
Don't accept MED
Utilize other BGP attributes higher in the decision process. (a little risky, because if any prefix leaked w/o higher attribute and tied)
Assign a high IGP cost to inter-cluster-link

10/15/2019

AS_PATH and Community regexp tips

1. match certain # of as_path

ip as-path access-list SixAS permit ^._._._._._.$ any

2. community list for some specific ending number, like ending 123. Because there is a space at the end of comm list, so need an underscore _, like

ip community-list regexp Ending123 permit ^.*:.*123_$

10/08/2019

Router Server

References:

Router Server = Internet Exchange (IX) Route Server = eBGP RR

Used in internet exchange
eBGP RR, so no need a full mesh eBGP peering among all parties
Reduces configuration complexity and CPU/memory overhead on border routers

Requirements:

AS_PATH transparency:

RS doesn't append its own AS# in the AS_PATH
RS-client doesn't enforce first AS

Nexthop transparency:

doesn't change NRLI's NH

MED transparency:

doesn't change path MED

Path hiding:

Per RFC 4271, BGP only advertises the best path, and a later update will be considered as an implicit withdrawal of the existing path.
In RFC 7947, it does mention the "path hiding". So the RS only advertises the best path, which could slow down the convergence time

EOS configurations:

service routing protocols model multi-agent << must multi-agent

route-map rtmap-no-change-med permit 10

set metric +0 << keep metric unchanged

router bgp 10

router-id 10.255.255.251

neighbor RS-Client peer group

neighbor RS-Client next-hop-unchanged << NH unchanged

neighbor RS-Client as-path prepend-own disabled << no own as#

neighbor RS-Client route-map rtmap-not-change-med out << rtmap

neighbor RS-Client password 7 xLcnzAMGHkI=

neighbor RS-Client send-community

neighbor RS-Client maximum-routes 12000

neighbor 10.10.10.11 peer group RS-Client

neighbor 10.10.10.11 remote-as 11

How many prefixes received? Considering the following scenario:

3 peers advertising ONE prefix
2 route servers
And all 3 peers also have bilateral peering
Then should be 5

Router1.15:19:15#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

Paths: 5 available

12 111

10.10.10.12 from 10.10.10.12 (10.255.255.12) << from R2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP head, ECMP, best, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.11 (10.255.255.11) << from R1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.251 (10.255.255.251) << R1 vis RS1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.252 (10.255.255.252) << R1 via RS2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

13 333 111

10.10.10.13 from 10.10.10.13 (10.255.255.13) << R3

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external

Rx SAFI: Unicast

Let's shut down the private peering to R2. So you can see the RS1 and RS2 don't advertise R2's path.

Router.15:26:31(config-router-bgp)#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

Paths: 4 available

11 111

10.10.10.11 from 10.10.10.252 (10.255.255.252) << RS2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external, best

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.251 (10.255.255.251) << RS1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.11 (10.255.255.11) << R1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

12 111

10.10.10.12 from 10.10.10.12 (10.255.255.12) << R2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

10/03/2019

EOS: A quick alias to show the interfaces with most util%

bn302#bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | more
Mbps Kpps Port
1.6% 4.5% Et9/36/1
1.5% 4.3% Et3/36/1
0.7% 0.5% Et13/33/1
0.7% 0.4% Et12/36/2

or set it as an alias

srnzsort bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | head -n 6

Updated note: The above command works with interface names/description defined. If not, have to change awk '{print $5, $8, $1}' to awk '{print $4, $7, $1}'

hs486.11:32:15#srnz
Port Name Intvl In Mbps % In Kpps Out Mbps % Out Kpps
Et49/1 0:05 9870.4 100.0% 812 9860.3 99.9% 811

hs486.12:14:45#bash Cli -p15 -c 'srnz' | awk '{print $4, $7, $1}' | sort -r | more
In In Port
100.0% 99.9% Et50/1
100.0% 99.9% Et49/4

How about some interfaces with description, some don't? hm.... Let me think about it..... :-)

9/19/2019

Ping bgp ipv6 link local address

With the feature of "BGP IPv6 link-local peering support", you can establish the bgp peering via ipv6 link-local address. To check the underlying reachability, we can use linux kernel command - ping6

For example,

interface Vlan4001
vrf forwarding vrf3_BGP_v6P_v6Ll
ipv6 address fe80::1/64 link-local
!
router bgp 65100
vrf vrf3_BGP_v6P_v6Ll
rd 65000:3
bgp default ipv4-unicast
bgp default ipv4-unicast transport ipv6
bgp default ipv6-unicast
neighbor fe80::100%Vl4001 remote-as 65500

bn303#bash sudo ip netns exec ns-vrf3_BGP_v6P_v6Ll ping6 -I vlan4001 fe80::100
PING fe80::100(fe80::100) from fe80::1%vlan4001 vlan4001: 56 data bytes
64 bytes from fe80::100%vlan4001: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from fe80::100%vlan4001: icmp_seq=2 ttl=64 time=0.238 ms
.....

Please note that this peering is under VRF, so you have to use the namespace command - "ip netns..."

9/09/2019

Kill list of zombie process on Linux(Eos)

Due to a script bug, one EOS device has quite some zombie process like,

[admin@bn302 flash]$ ps aux | grep 'CliShell -A'
root 28804 0.0 0.0 10504 8580 ? S 10:53 0:00 CliShell -A -p 15 -c show port-channel | json
root 28809 0.0 0.0 10504 8688 ? S 10:53 0:00 CliShell -A -p 15 -c show port-channel | json
root 28826 0.0 0.0 10504 8612 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
root 28835 0.0 0.0 10504 8580 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
root 28844 0.0 0.0 10504 8616 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
....

Instead of killing one by one manually, write a quick shell command to kill all. (killall is not supported in EOS/Linux)

[admin@bn302 flash]$ for pid in $(ps aux | grep 'CliShell -A' | awk '{print $2, $11}' | grep CliShell | awk '{print $1}'); do sudo kill -9 $pid; done