Arista EOS Hashing Features

Here are some Arista EOS hashing features: 

• 4.15.2F, hashing on ttl and hop-limit
• Trident2-based systems: 7300X, 7050X, 7250QX. All have 2 hash engines, 1 for port-channel, 1 for ECMP
• CLI: 
• port-channel load-balance trident fields ip ttl 
• port-channel load-balance trident fields ipv6 hop-limit
• show port-channel load-balance trident fields
• IP TTL hashing is ON
• Unlike other "port-ch load-balance trident fields" commands, this one applies both to both LAG and ECMP hash engines. 
• Be care of unexpected behavior,  for example,  
• Internet traffic goes to proxy or load-balance servers.
• But somehow traffic going a different path results into different TTL values, like VRRP or MLAG. 
• So if hashing based on ttl, the same traffic stream could end up at different destinations. 
• Hashing egress interface: 
• 4.17.0
• show load-balance destination ingress-interface Ethernet1/1 src-ipv4-address 10.0.0.93 dst-ipv4-address 17.0.0.11 ip-protocol 17 src-l4-port 11001 dst-l4-port 19872
• Output Interface: Port-Channel102
• Disable Ingress interface in hashing
• Arad/Jericho based systems: 7500E/R, 7280E/R
• CLI: 
• no ip load-sharing sand fields ingress-interface
• To avoid polarization, use different preset in different layers:
• ip load-sharing sand has preset <0-2>
• 好像是，不同interface会有不同random#
• 这个for ECMP
• LAG hashing on ingress interface
• 和上面差不多，区别是config放在LB profile里面
• for LAG
• Global LAG hashing profile
• Arad/Jericho, Sand-based systems
• CLI:
• load-balance policies
• load-balance sand profile myProfile
• hash seed 50
• Dynamic and Symmetric LAG Hashing
• From 4-15-0
• Arad/Jericho, Sand-based systems
• Bidirectional flows go thru same member port of LAG
• 做法不难，就是异或源和目标地址，再求哈希，当然LB就是不好
• CLI: 
• load-balance policies
•    load-balance sand profile sym
•    fields symmetric-hash

Arista L3 Leaf/Spine Step by Step (3a) - Vxlan Routing, Modes

The following couple of posts will cover the VxLAN routing. 

What's and why VxLAN Routing? With VxLAN bridging, the hosts in the same VLAN 10 can talk to each other. But we need VxLAN routing to have VLAN 10 address to reach the hosts in VLAN 11. 

How to enable VxLAN Routing? Creating a SVI interface on a VLAN which is associated to a VNI

We can achieve this in the following 5 modes:

• Router on a stick - Gateway only on dedicated routers
• Direct Routing - Gateway on all VTEPs
• Indirect Routing - Gateway only on few dedicated VTEPs
• Naked Routing - Regular routing w/o VxLAN encap
• * Hybrid - Combination of direct/indirect and naked routing based on case

Unbalanced Hashing vs DLB vs RECMP

Broadcom introduced Dynamic Load Balancing in Trident 3 product. 

https://www.youtube.com/watch?v=1y25OTyFFrs

里面提到了Unblanced Hashing的3个Reasons:
1. Elephant Flow
2. Link Failures, Why?
3. Hash Polarization 

可能based在MIT一个flowlet paper

• 这个原理不难，多根线路，计算Maximum和Minimum Delay，之差就是一个timeout value;
• 如果后续packet timeout之后到，就不必hash到固定的线路，而是严格Load Balancing. 

一个相关的Arista EOS feature - Resilient ECMP

• 说起来很简单，就是一根线路断了，不要re-hashing区别的sessions，这样会teardown TCP. 而是just re-hash 故障线路上的session
• CLI: ip hardware fib ecmp resilience 10.14.2.0/24 capacity 6 redundancy 4
• 解释:
• network 10.14.2.0/24有6个NextHop
• 但是建立一个24 Entries的ECMP
• 如果其中一个NH断了，就把表中这个NH用其他NH替换，其他ECMP NH没有变化
• Redundancy = 4，应该就是一根断了，有4个来LB

TCP Notes (2)

1. 如何计算 a TCP segment length?
A: ip.len - ip.hdr_len - tcp.hdr_len

2. TCP Flags
* 最常见的是SYN, ACK, FIN
* URG - 比方说，sender希望receiver stop，无论queue里面还有多少data. 
* PSH - 和URG很像，必须high priority的处理
* RST - 本意是，如果port没有open，host就返回一个RST，容易被hacker利用
* SYN - 只有3-way handshake时有
* FIN - FINISH，4-way close? ACK/FIN-ACK-ACK/FIN-ACK

3. TCP window size
* len = 2 Bytes, max = 64K, in Byte
* 还有一个window scale，就是window size左移多少位
* SYN packet里面没有scaled
* 两边可以有不同的scale#, 不需要negotiated

这个是SYN Packet


后面的Data Packet


4. TCP checksum
看下面的图，前面加上IP Header Info, srcAdr, dstArd, reserved, proto#, tcpLen(computed), + tcpHdr, tcpData. 可以cover ip header integrity



5. TCP Timestamps
10-Byte TCP option, type(1B=8); len(=10), TS(len=4), TSEcho(len=4)，例如：

1st packet, TS = 2216538



2nd packet, TSEcho = 2216538. 这样就可以计算RTT了



6. TCP MSS vs IP MTU

这个MSS有点confusing，因为Segment一般是指Header + Data，但是这个TCP MSS说的是Max Data Leng. 这个图很清楚，
* Ethernet Frame size = 1518, 
* IP MTU = 1500 = 1518 - 18, 
* TCP MSS = 1460 = 1500 - 20 (ip.hdr_len) - 20 (tcp.hdr_len)



Reference:
1. firewall.cx
2. tcpipguide.com

Link-debounce timer

https://www.manualslib.com/manual/546511/Dell-Force10-S4810p.html?page=501

Delay the notification of interface up/down events. 主要是防止快速而重复的interface flaps，对上层协议的影响。时间是milisec. 

Configuration:

bn303.spine.host3(s1)(config-if-Et3/1/1)#link-debounce time 1000 1000
bn303.spine.host3(s1)(config-if-Et3/1/1)#int e3/1/1
bn303.spine.host3(s1)(config-if-Et3/1/1)#link-debounce time 1000 1000
bn303.spine.host3(s1)(config-if-Et3/1/1)#

TCP Notes (1)

How TCP Works @ youtube

• TCP fundamentals Part 2
• 2 related TCP windows: 
• CWND - Congestion Window
• RWND - Receive Window
• Only RWND advertized in packets
• minimum of CWND and RWND is the cap of transmission
• CWND = n * MSS, n = 1,2,4...., slowstart
• not advertised, changing
• Example 1, slow backup between 2 local servers
• from the trace, recvr to sender 只发ACKs, (len = 60) 
• window size 每隔一段时间，会从64K一直降到1000 (小于MSS). 然后hold for 190ms. 然后window size back to 64K
• 显示chart - statistics -> TCP stream graph -> Time sequence (tcptrace)
• 足够的证据是server side问题
• Example 2, 是sender application无法fill CWND,
• Example 3, ip tcp mss-adjust问题
• server/client, 2个pcap
• client直接看到seq = 1461, 不是1
• server一直re-send seq = 1, size = 1460 pkts. 
• TCP NoOP option - 很简单，#TcpLen * 4 = Tcp Header Len，而TCP Options Len不全是4的倍数，所以TCP Header需要NoOp pad out header size to a multiple of 4B. 
• TCP option - timestamp
• TSval, EchoTSval, 为了计算RTT and RTO (retrans timeout)
• Some TCP options:
• 0 - end of TCP options
• 1 - NoOP
• 2 - MSS
• 3 - Window Scale
• 4/5 - SACK
• 8 - timestamp

Wireshark Tips (2)

packetbomb.com - troubleshooting MTU issue

• Following the previous post, need column - tcpLen, Seq, nextSeq, Ack, BytesInFlight
• Statistics -> conversation, find the biggest stream 
• couple of TCP retransmission packets in jumbo frame with DF bit, because no ACK
• later sender starts with 512B -> 1024B, then conversation on. 
• clearly, 1 or multiple middle routers can't handle jumbo frames
• filter = icmp, doesn't show any icmp unreachable
• per RFC 1191 - path MTU discovery. The middle routers should return a ICMP destination Unreachable message with code = "frag needed and DF set"
• So these pkts are dropped by either firewall or disabled on the router

Wireshark Tips (1)

Quick refresh:

• Where to capture有讲究
• Capture on host machine不好，不是真正packets on the wire
• Wireshark setup: 
• 必须有time delta, 
• TCP: seq + tcpLen = nextSeq, ACK
• TCP: winSize，必须有，scaleNum in 3-way handshake
• TCP: byteOnTheFlight

Sources:

• A youtube video - Using Wireshark to Solve Real Problems for Real People (by Kary Rogers)
• packetbomb.com 
• packetbomb.com's "Fundamental Course"

Takeaways:

• Packet capture:
• Where matters
• Close to the source of complaint, capture all traffic w/o filter. If doing this in the network, have to figure out a smart way because of too much traffic. 
• Near the servers, if out of the domain, on the edge where the traffic leave
• Both if possible
• How to capture:
• On the client machine: NOT a good idea
• wireshark, tcpdump, windump
• easy but not the real traffic on the wire, why?
• application -- kernel's tcp/ip stack -- pcap (where is wireshark) -- NIC -- network. 
• for example, the packet before segmentation
• SPAN/Mirror
• TAP
• Hub??
• Wireshark setup (on my MAC wireshark 2.14)
• Wireshark -> Preference
• Layout
• Name Resolution: MAC, protoName, IP addr. 
• Protocol -> TCP
• Column:
• [] means data not in packet but calculated 
• 1. Add [time delta from previous displayed frame] to column
• right-click == apply as column
• 2. Add TCP's seq, Len, nextSeq, ACK to column. 
• Easy to see if losing a packet
• 3. Add TCP's windowSize
• Scale # is ONLY in 3-way handshake, x 256
• 4. Add "Bytes in flight"