2/09/2019

How to detect a problem with 1 ECMP link

The ICMP packets are not hashed over ECMP path, which is shown in a quick test below on Arista EOS 4.20.8M. 

But we can use traceroute with multiple probes (Thanks for my buddy Janardhan Madabattular's idea), because the traceroute uses UDP with random port numbers.

And the traceroute has a limitation with 10 probes, so need to run multiple times for high-number-way ECMP existing. 

[admin@R1 ~]$ traceroute 3.3.3.3 -s 1.1.1.1 -q 10
traceroute to 3.3.3.3 (3.3.3.3), 30 hops max, 60 byte packets
 1  12.12.12.2 (12.12.12.2)  0.147 ms  0.078 ms  0.111 ms  0.069 ms  0.082 ms  0.068 ms  0.098 ms  0.072 ms  0.155 ms  0.089 ms

 2  3.3.3.3 (3.3.3.3)  0.116 ms *  0.064 ms *  0.069 ms  0.085 ms  0.167 ms  0.060 ms  0.073 ms  0.062 ms

R1(1.1.1.1)-----R2==(4-way ip ecmp)===R3 (3.3.3.3)

1) I send 10K ping to R3
R1(config-if-Lo0)#ping 3.3.3.3 source 1.1.1.1 repeat 10000

2) R2 has a 4-way ecmp
R2(config)#sh ip route 3.3.3.3
 S      3.3.3.3/32 [1/0] via 23.23.1.3, Ethernet6/1/1
                         via 23.23.2.3, Ethernet6/1/2
                         via 23.23.3.3, Ethernet6/1/3
                         via 23.23.4.3, Ethernet6/1/4

3) All 10K goes the 1st link
R2(config)#sh int et6/1/1 - 4 | egrep 'ets input|ets output|^Eth'
Ethernet6/1/1 is up, line protocol is up (connected)
     3 packets input, 699 bytes
     10003 packets output, 1180696 bytes <<<< R1-R3 
Ethernet6/1/2 is up, line protocol is up (connected)
     3 packets input, 699 bytes
     3 packets output, 696 bytes
Ethernet6/1/3 is up, line protocol is up (connected)
     3 packets input, 699 bytes
     3 packets output, 696 bytes
Ethernet6/1/4 is up, line protocol is up (connected)
     10003 packets input, 1180699 bytes  <<<< R3-R1
     3 packets output, 696 bytes
....

I believe hashing is based on 5-tuple (proto#, src/dst ip, src/dst port) for tcp/udp. How about ICMP? It misses src/dst port, in stead type/code/checksum in the place. The packet capture shows the checksum is calculated and varied. 




2 comments:

  1. you might also consider https://paris-traceroute.net/ for ecmp-rich environment

    ReplyDelete
  2. Thanks for the link. Nice tool, not sure if ou have a python version?

    ReplyDelete