Arista EOS/Networking Tips by Solomon Yang: ECMP

Showing posts with label ECMP. Show all posts

9/30/2020

Arista EOS Lag/ECMP Hashing Features

References/Manual:

EOS Manual Section 12.4: Load Balancing Hash Algorithms

References/TOI:

Sample: In this below sample, I like to show how to use the above features to fix an ipv6 multicast unbalanced issue.

Says, in the above topology, there are 2000 ipv6 multicast streams flowing ixia-harness-[vrrp primary| secondary]-nexthop router. Let's see the output of srnz on VRRP primary.

1) w/o any hashing config, lowest vs highest = 220 vs 317

ghs259-GW2.21:24:15(config-if-Po2)#srnz | grep HUB

Et3/1 GW2-HUB2 0:05 0.0 0.0% 0 220.1 0.2% 35

Et4/1 GW2-HUB2 0:05 0.0 0.0% 0 316.9 0.3% 50

Et13/1 GW2-HUB2 0:05 0.0 0.0% 0 266.1 0.3% 42

Et14/1 GW2-HUB2 0:05 0.0 0.0% 0 269.3 0.3% 43

Po2 GW2-HUB2 0:05 0.0 0.0% 0 1072.3 0.3% 171

2) hash key shift

Let's try 1 hashing feature - Lag Hashing Key Shift. Create a LB profile - mcast-v6 + key shift, apply it globally.

ghs259-GW2.21:37:22(config)#load-balance policies

ghs259-GW2.21:37:28(config-load-balance-policies)# load-balance sand profile mcast-v6

ghs259-GW2.21:37:35(config-sand-load-balance-profile-mcast-v6)# ecmp hash key shift 3

ghs259-GW2.21:37:57(config)#port-channel load-balance sand profile mcast-v6

ghs259-GW2.21:39:50#srnz | grep HUB

Et3/1 GW2-HUB2 0:05 0.0 0.0% 0 220.1 0.2% 35

Et4/1 GW2-HUB2 0:05 0.0 0.0% 0 316.9 0.3% 50

...

Clearly, not much help.

3) hash seed

ghs259-GW2.21:42:47(config-load-balance-policies)#load-balance sand profile mcast-v6

! profile mcast-v6 is the current global profile

ghs259-GW2.21:42:52(config-sand-load-balance-profile-mcast-v6)#no ecmp hash key shift

ghs259-GW2.21:43:03(config-sand-load-balance-profile-mcast-v6)#ecmp hash seed 1001

ghs259-GW2.21:43:25(config-sand-load-balance-profile-mcast-v6)#end

ghs259-GW2.21:43:36#srnz | grep HUB

Et3/1 GW2-HUB2 0:05 0.0 0.0% 0 220.2 0.2% 35

Et4/1 GW2-HUB2 0:05 0.0 0.0% 0 316.9 0.3% 50

...

4) hash polynomial + Hardware load-balancing for ingress/egress and fabric/egress replication (4.18.0F)

ghs259-GW2.21:45:26(config)#load-balance policies

ghs259-GW2.21:45:30(config-load-balance-policies)#load-balance sand profile mcast-v6

! profile mcast-v6 is the current global profile

ghs259-GW2.21:45:35(config-sand-load-balance-profile-mcast-v6)#no ecmp hash seed 1001

ghs259-GW2.21:45:42(config-sand-load-balance-profile-mcast-v6)#port-channel hash polynomial 5

ghs259-GW2.21:45:43(config-load-balance-policies)# port-channel load-balance sand replication egress

ghs259-GW2.21:46:52#srnz | grep HUB

Et3/1 GW2-HUB2 0:05 0.0 0.0% 0 265.3 0.3% 42

Et4/1 GW2-HUB2 0:05 0.0 0.0% 0 272.3 0.3% 43

Nice! The hash polynomial works! Actually EOS 8 polynomial functions, the #5 works.

4/25/2020

ECMP in MPLS L3 VPN

Consider the above topology

CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1.

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:1

Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:4364

Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012

via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name VRF ID Table ID

cust_1 15 65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 116707

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries.

PE1#show hardware capacity | grep -i ECMP

ECMP 875 21% 3220 0 4095 875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

1.1.1.1 4 65000 1693 1673 0 0 23:25:15 Estab 1602 1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage.

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP 0 0% 4095 0 4095 875

ECMP Mpls 0 0% 4095 0 4095 0

ECMP Routing 0 0% 4095 0 4095 875

ECMP VxlanOverlay 0 0% 4095 0 4095 0

ECMP VxlanTunnel 0 0% 3891 0 3891 0

Now no ECMP anymore in the sw/hw routing table.

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32830 |0 | - | | - |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels.

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

192.158.115.11 4 65100 1568 1819 0 0 1d00h Estab 800 800

192.168.115.11 4 65100 1749 1827 0 0 1d00h Estab 802 802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels.

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

Local Label: 16694 (with rewrite);

Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

Local Label: 16695 (with rewrite);

Received Label 116390

2/09/2019

How to detect a problem with 1 ECMP link

The ICMP packets are not hashed over ECMP path, which is shown in a quick test below on Arista EOS 4.20.8M.

But we can use traceroute with multiple probes (Thanks for my buddy Janardhan Madabattular's idea), because the traceroute uses UDP with random port numbers.

And the traceroute has a limitation with 10 probes, so need to run multiple times for high-number-way ECMP existing.

[admin@R1 ~]$ traceroute 3.3.3.3 -s 1.1.1.1 -q 10
traceroute to 3.3.3.3 (3.3.3.3), 30 hops max, 60 byte packets
1 12.12.12.2 (12.12.12.2) 0.147 ms 0.078 ms 0.111 ms 0.069 ms 0.082 ms 0.068 ms 0.098 ms 0.072 ms 0.155 ms 0.089 ms

2 3.3.3.3 (3.3.3.3) 0.116 ms * 0.064 ms * 0.069 ms 0.085 ms 0.167 ms 0.060 ms 0.073 ms 0.062 ms

R1(1.1.1.1)-----R2==(4-way ip ecmp)===R3 (3.3.3.3)

1) I send 10K ping to R3
R1(config-if-Lo0)#ping 3.3.3.3 source 1.1.1.1 repeat 10000

2) R2 has a 4-way ecmp
R2(config)#sh ip route 3.3.3.3
S 3.3.3.3/32 [1/0] via 23.23.1.3, Ethernet6/1/1
via 23.23.2.3, Ethernet6/1/2
via 23.23.3.3, Ethernet6/1/3
via 23.23.4.3, Ethernet6/1/4

3) All 10K goes the 1st link
R2(config)#sh int et6/1/1 - 4 | egrep 'ets input|ets output|^Eth'
Ethernet6/1/1 is up, line protocol is up (connected)
3 packets input, 699 bytes
10003 packets output, 1180696 bytes <<<< R1-R3
Ethernet6/1/2 is up, line protocol is up (connected)
3 packets input, 699 bytes
3 packets output, 696 bytes
Ethernet6/1/3 is up, line protocol is up (connected)
3 packets input, 699 bytes
3 packets output, 696 bytes
Ethernet6/1/4 is up, line protocol is up (connected)
10003 packets input, 1180699 bytes <<<< R3-R1
3 packets output, 696 bytes
....

I believe hashing is based on 5-tuple (proto#, src/dst ip, src/dst port) for tcp/udp. How about ICMP? It misses src/dst port, in stead type/code/checksum in the place. The packet capture shows the checksum is calculated and varied.

1/31/2019

Arista EOS Hashing Features

Here are some Arista EOS hashing features:

4.15.2F, hashing on ttl and hop-limit

Trident2-based systems: 7300X, 7050X, 7250QX. All have 2 hash engines, 1 for port-channel, 1 for ECMP
CLI:

port-channel load-balance trident fields ip ttl
port-channel load-balance trident fields ipv6 hop-limit
show port-channel load-balance trident fields

IP TTL hashing is ON

Unlike other "port-ch load-balance trident fields" commands, this one applies both to both LAG and ECMP hash engines.
Be care of unexpected behavior, for example,

Internet traffic goes to proxy or load-balance servers.
But somehow traffic going a different path results into different TTL values, like VRRP or MLAG.
So if hashing based on ttl, the same traffic stream could end up at different destinations.

Hashing egress interface:

4.17.0
show load-balance destination ingress-interface Ethernet1/1 src-ipv4-address 10.0.0.93 dst-ipv4-address 17.0.0.11 ip-protocol 17 src-l4-port 11001 dst-l4-port 19872
Output Interface: Port-Channel102

Disable Ingress interface in hashing

Arad/Jericho based systems: 7500E/R, 7280E/R
CLI:

no ip load-sharing sand fields ingress-interface

To avoid polarization, use different preset in different layers:

ip load-sharing sand has preset <0-2>

好像是，不同interface会有不同random#
这个for ECMP

LAG hashing on ingress interface

和上面差不多，区别是config放在LB profile里面
for LAG

Global LAG hashing profile

Arad/Jericho, Sand-based systems
CLI:

load-balance policies
load-balance sand profile myProfile
hash seed 50

Dynamic and Symmetric LAG Hashing

From 4-15-0
Arad/Jericho, Sand-based systems
Bidirectional flows go thru same member port of LAG
做法不难，就是异或源和目标地址，再求哈希，当然LB就是不好
CLI:

load-balance policies
load-balance sand profile sym
fields symmetric-hash

1/11/2019

Unbalanced Hashing vs DLB vs RECMP

Broadcom introduced Dynamic Load Balancing in Trident 3 product.

https://www.youtube.com/watch?v=1y25OTyFFrs

里面提到了Unblanced Hashing的3个Reasons:
1. Elephant Flow
2. Link Failures, Why?
3. Hash Polarization

可能based在MIT一个flowlet paper

这个原理不难，多根线路，计算Maximum和Minimum Delay，之差就是一个timeout value;
如果后续packet timeout之后到，就不必hash到固定的线路，而是严格Load Balancing.

一个相关的Arista EOS feature - Resilient ECMP

说起来很简单，就是一根线路断了，不要re-hashing区别的sessions，这样会teardown TCP. 而是just re-hash 故障线路上的session
CLI: ip hardware fib ecmp resilience 10.14.2.0/24 capacity 6 redundancy 4
解释:

network 10.14.2.0/24有6个NextHop
但是建立一个24 Entries的ECMP
如果其中一个NH断了，就把表中这个NH用其他NH替换，其他ECMP NH没有变化
Redundancy = 4，应该就是一根断了，有4个来LB

4/10/2018

Arista EOS: CLI to predict egress LAG/ECMP interface

This is a very useful command when troubleshooting packet loss in an ECMP link. A refresh post of an old post.

arista.EOS.7050SX#sh ip route 22.0.2.91
B E 22.0.2.0/24 [200/0] via 210.100.21.0, Ethernet11/1
via 210.100.22.0, Ethernet21/1
via 210.100.21.12, Vlan3304
via 210.100.21.14, Vlan3305
via 210.100.21.16, Vlan3306
via 210.100.22.12, Vlan3404
via 210.100.22.14, Vlan3405
via 210.100.22.16, Vlan3406
via 210.100.21.2, Port-Channel1931
via 210.100.21.4, Port-Channel1932
via 210.100.22.2, Port-Channel1941
via 210.100.22.4, Port-Channel1942
via 210.100.21.6, Ethernet12/1.3301
via 210.100.21.8, Port-Channel1933.3302
via 210.100.21.10, Port-Channel1934.3303
via 210.100.22.6, Ethernet22/1.3401
via 210.100.22.8, Port-Channel1943.3402
via 210.100.22.10, Port-Channel1944.3403

arista.EOS.7050SX#sh load-balance destination ip ingress-interface ethernet 36 src-ipv4-addr 20.0.1.100 dst-ipv4-addr 22.0.2.91 ip-protocol 9
Output Interface: Port-Channel1934.3303

arista.EOS.7050SX#sh load-balance destination ip ingress-interface ethernet 36 src-ipv4-addr 20.0.1.100 dst-ipv4-addr 22.0.2.91 ip-protocol 1
Output Interface: Port-Channel1941

show load-balance destination port-channel portchannel [ingress-interface intf] [src-mac srcmac] [dst-mac dstmac] [eth-type ethtype] [src-ip-address src-ip] [dst-ip-address dst-ip] [ip-ttl ttl] [flow-label fl] [ip-protocol proto] [src-l4-port src-port] [dst-l4-port dst-port] [vlan vlan] [non-unicast]