MPLS Hashing: Control Word and Entropy Label

A good presentation on this topic: https://ripe66.ripe.net/presentations/141-RIPE66_Understanding_MPLS_Hashing-Snijders.pdf

• MPLS Lable doesn't have encapsulated protocol information. So after the bottom label, 2 possibilities:
• IPv4/v6 packets if L3VPN, the 1st 4-bit is ipVersion, 4=ipv4, 6=ipv6
• Ethernet frames if L2VPN, the 1st 48-bit is dstMAC
• Along the LSP, if P routers like to do ECMP by hashing, it has to guess like above to do hashing the 5-tuple fields (src/dstAddr, src/dstPort, protoNum)
• So the problem happens if the dstMac starts with 4 or 6

The solution for the above issue is MPLS control word (RFC 4385, Feb 2006), a 4-byte all-zero (same length as a label) control word after the bottom label. 

• Arista EOS 4.21.3F: Configurable L2 EVPN MPLS control word
• mpls evpn; label-stack control-word
• Arista EOS 4.22.1F: Control Word support on LDP Pseudowire
• mpls ldp; pseudowire <name>; control-word

So the MPLS control word is to prevent incorrect hashing. How does the LSR do the load-balancing? The answer is Entropy Label (RFC 6790, Nov 2012)

• Increase the label stack by 2: 1 for ELI (entropy label indicator = 7), 1 for EL (entropy label)
• Aristea EOS 4.26.0F: LDP Entropy Label Support
• 4.25.2F: LSR support
• 4.26.0F: LER support
• mpls ldp; entropy-label
• This feature has a limitation: PHP LSR can't pop ELI + EL labels, so has to rely on LER to pop up [ELI, ELI, VPN] labels. 

Another solution is FAT - Flow Aware Transport on LDP pseudowires (4.24.0F) based on RFC 6391

• mpls ldp; pseudowire; pseudowire <name>; label flow

Arista EOS Lag/ECMP Hashing Features

References/Manual:

• EOS Manual Section 12.4: Load Balancing Hash Algorithms

References/TOI:

• EOS 4.24.2F TOI - Support for dynamic load balancing on ECMP groups (Trident Only)
• EOS 4.22.1F TOI - ECMP Load Balance Profile Support (Sand/Jericho Only)
• EOS 4.21.3F TOI - Symmetric header selection in LAG load-balancing profiles (Jericho Only)
• EOS 4.21.0F TOI - UDF flex hashing improvements (Trident Only)
• EOS 4.20.6F TOI - LAG hashing enhancement for configuring member-selection method (Jericho Only)
• EOS 4.20.5F TOI - LAG hashing on Ingress Interface
• EOS 4.20.5F TOI - Lag Hashing Key Shift
• EOS 4.20.5F TOI - VxLAN over MPLS LAG hashing optimizations for Bridged ports
• EOS 4.20.1F TOI - LAG and ECMP selectable hashing fields (Jericho Only)
• EOS 4.18.0F TOI - Hardware load-balancing for ingress/egress and fabric/egress replication
• EOS 4.17.2F TOI - Configurable field selection for ECMP hash, (Trident Only)
• EOS 4.17.2F TOI - Disable usage of ingress interface in ECMP hashing
• EOS 4.17.1F TOI - Inner IP hashing for MPLSoGRE
• EOS 4.17.0F TOI - Global LAG Hashing Profiles (Jericho Only)
• EOS 4.17.0F TOI - ECMP Hash Visibility
• EOS 4.15.2F TOI - GRE in LAG hash
• EOS 4.15.2F TOI - LAG/ECMP hashing on TTL/Hop Limit
• EOS 4.15.0F TOI - Dynamic and symmetric LAG hashing
• EOS 4.15.0F TOI - Tap aggregation enhancements

Sample: In this below sample, I like to show how to use the above features to fix an ipv6 multicast unbalanced issue. 

Says, in the above topology, there are 2000 ipv6 multicast streams flowing ixia-harness-[vrrp primary| secondary]-nexthop router. Let's see the output of srnz on VRRP primary. 

1) w/o any hashing config, lowest vs highest = 220 vs 317

ghs259-GW2.21:24:15(config-if-Po2)#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     220.1   0.2%       35

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     316.9   0.3%       50

Et13/1    GW2-HUB2     0:05       0.0   0.0%        0     266.1   0.3%       42

Et14/1    GW2-HUB2     0:05       0.0   0.0%        0     269.3   0.3%       43

Po2       GW2-HUB2     0:05       0.0   0.0%        0    1072.3   0.3%      171

2) hash key shift

Let's try 1 hashing feature - Lag Hashing Key Shift. Create a LB profile - mcast-v6 + key shift, apply it globally. 

ghs259-GW2.21:37:22(config)#load-balance policies

ghs259-GW2.21:37:28(config-load-balance-policies)#   load-balance sand profile mcast-v6

ghs259-GW2.21:37:35(config-sand-load-balance-profile-mcast-v6)#      ecmp hash key shift 3

ghs259-GW2.21:37:57(config)#port-channel load-balance sand profile mcast-v6

ghs259-GW2.21:39:50#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     220.1   0.2%       35

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     316.9   0.3%       50

...

Clearly, not much help. 

3) hash seed

ghs259-GW2.21:42:47(config-load-balance-policies)#load-balance sand profile mcast-v6

! profile mcast-v6 is the current global profile

ghs259-GW2.21:42:52(config-sand-load-balance-profile-mcast-v6)#no ecmp hash key shift

ghs259-GW2.21:43:03(config-sand-load-balance-profile-mcast-v6)#ecmp hash seed 1001

ghs259-GW2.21:43:25(config-sand-load-balance-profile-mcast-v6)#end

ghs259-GW2.21:43:36#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     220.2   0.2%       35

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     316.9   0.3%       50

...

4) hash polynomial + Hardware load-balancing for ingress/egress and fabric/egress replication (4.18.0F)

ghs259-GW2.21:45:26(config)#load-balance policies

ghs259-GW2.21:45:30(config-load-balance-policies)#load-balance sand profile mcast-v6

! profile mcast-v6 is the current global profile

ghs259-GW2.21:45:35(config-sand-load-balance-profile-mcast-v6)#no ecmp hash seed 1001

ghs259-GW2.21:45:42(config-sand-load-balance-profile-mcast-v6)#port-channel hash polynomial 5

ghs259-GW2.21:45:43(config-load-balance-policies)#    port-channel load-balance sand replication egress

ghs259-GW2.21:46:52#srnz | grep HUB

Et3/1     GW2-HUB2     0:05       0.0   0.0%        0     265.3   0.3%       42

Et4/1     GW2-HUB2     0:05       0.0   0.0%        0     272.3   0.3%       43

Nice! The hash polynomial works! Actually EOS 8 polynomial functions, the #5 works. 

How to detect a problem with 1 ECMP link

The ICMP packets are not hashed over ECMP path, which is shown in a quick test below on Arista EOS 4.20.8M. 

But we can use traceroute with multiple probes (Thanks for my buddy Janardhan Madabattular's idea), because the traceroute uses UDP with random port numbers.

And the traceroute has a limitation with 10 probes, so need to run multiple times for high-number-way ECMP existing. 

[admin@R1 ~]$ traceroute 3.3.3.3 -s 1.1.1.1 -q 10
traceroute to 3.3.3.3 (3.3.3.3), 30 hops max, 60 byte packets
 1  12.12.12.2 (12.12.12.2)  0.147 ms  0.078 ms  0.111 ms  0.069 ms  0.082 ms  0.068 ms  0.098 ms  0.072 ms  0.155 ms  0.089 ms

 2  3.3.3.3 (3.3.3.3)  0.116 ms *  0.064 ms *  0.069 ms  0.085 ms  0.167 ms  0.060 ms  0.073 ms  0.062 ms

R1(1.1.1.1)-----R2==(4-way ip ecmp)===R3 (3.3.3.3)

1) I send 10K ping to R3
R1(config-if-Lo0)#ping 3.3.3.3 source 1.1.1.1 repeat 10000

2) R2 has a 4-way ecmp
R2(config)#sh ip route 3.3.3.3
 S      3.3.3.3/32 [1/0] via 23.23.1.3, Ethernet6/1/1
                         via 23.23.2.3, Ethernet6/1/2
                         via 23.23.3.3, Ethernet6/1/3
                         via 23.23.4.3, Ethernet6/1/4

3) All 10K goes the 1st link
R2(config)#sh int et6/1/1 - 4 | egrep 'ets input|ets output|^Eth'
Ethernet6/1/1 is up, line protocol is up (connected)
     3 packets input, 699 bytes
     10003 packets output, 1180696 bytes <<<< R1-R3 
Ethernet6/1/2 is up, line protocol is up (connected)
     3 packets input, 699 bytes
     3 packets output, 696 bytes
Ethernet6/1/3 is up, line protocol is up (connected)
     3 packets input, 699 bytes
     3 packets output, 696 bytes
Ethernet6/1/4 is up, line protocol is up (connected)
     10003 packets input, 1180699 bytes  <<<< R3-R1
     3 packets output, 696 bytes
....

I believe hashing is based on 5-tuple (proto#, src/dst ip, src/dst port) for tcp/udp. How about ICMP? It misses src/dst port, in stead type/code/checksum in the place. The packet capture shows the checksum is calculated and varied. 

Arista EOS Hashing Features

Here are some Arista EOS hashing features: 

• 4.15.2F, hashing on ttl and hop-limit
• Trident2-based systems: 7300X, 7050X, 7250QX. All have 2 hash engines, 1 for port-channel, 1 for ECMP
• CLI: 
• port-channel load-balance trident fields ip ttl 
• port-channel load-balance trident fields ipv6 hop-limit
• show port-channel load-balance trident fields
• IP TTL hashing is ON
• Unlike other "port-ch load-balance trident fields" commands, this one applies both to both LAG and ECMP hash engines. 
• Be care of unexpected behavior,  for example,  
• Internet traffic goes to proxy or load-balance servers.
• But somehow traffic going a different path results into different TTL values, like VRRP or MLAG. 
• So if hashing based on ttl, the same traffic stream could end up at different destinations. 
• Hashing egress interface: 
• 4.17.0
• show load-balance destination ingress-interface Ethernet1/1 src-ipv4-address 10.0.0.93 dst-ipv4-address 17.0.0.11 ip-protocol 17 src-l4-port 11001 dst-l4-port 19872
• Output Interface: Port-Channel102
• Disable Ingress interface in hashing
• Arad/Jericho based systems: 7500E/R, 7280E/R
• CLI: 
• no ip load-sharing sand fields ingress-interface
• To avoid polarization, use different preset in different layers:
• ip load-sharing sand has preset <0-2>
• 好像是，不同interface会有不同random#
• 这个for ECMP
• LAG hashing on ingress interface
• 和上面差不多，区别是config放在LB profile里面
• for LAG
• Global LAG hashing profile
• Arad/Jericho, Sand-based systems
• CLI:
• load-balance policies
• load-balance sand profile myProfile
• hash seed 50
• Dynamic and Symmetric LAG Hashing
• From 4-15-0
• Arad/Jericho, Sand-based systems
• Bidirectional flows go thru same member port of LAG
• 做法不难，就是异或源和目标地址，再求哈希，当然LB就是不好
• CLI: 
• load-balance policies
•    load-balance sand profile sym
•    fields symmetric-hash