Trouble-shoot BGP peering issue over GRE tunnel

Starting from 4.21.1F, Arista EOS starts to support the hardware GRE tunnel interface and BGP session over the tunnel on Jericho platforms. Before the tunnel is implemented by nexthop-group and decap group. 

Here is a very simple and straightforward setup of eBGP over GRE tunnel. 

But BGP session fails to come up as shown below:

R1.gts425#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4               4766       257    0    0 00:20:40 Connect 0      0

Tunnel interface is up and works fine. 

R1.gts425#sh int tunnel 100

Tunnel100 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 0101.0101.0800

  Description: tunnel-gre-sand-to-sand

  Internet address is 10.100.100.1/24

  Broadcast address is 255.255.255.255

  Tunnel source 1.1.1.1, destination 4.4.4.4

  Tunnel protocol/transport GRE/IP

   Key disabled, sequencing disabled

   Checksumming of packets disabled

  Tunnel TTL 0, Hardware forwarding not supported

  Tunnel TOS 0

  Path MTU Discovery

  Tunnel transport MTU 1476 bytes

  Up 22 minutes, 2 seconds

Ping with MTU size works totally fine

R1.gts425#ping 10.100.100.4 size 1476

...

80 bytes from 10.100.100.4: icmp_seq=5 ttl=64 time=0.118 ms

--- 10.100.100.4 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.118/0.155/0.295/0.070 ms, ipg/ewma 0.223/0.222 ms

Now let's run the tcpdump on R1 to see if hello packet out

R1.gts425(config-router-bgp)#bash tcpdump -nvvi et21 host 1.1.1.1

tcpdump: listening on et21, link-type EN10MB (Ethernet), capture size 262144 bytes

11:19:19.774224 28:99:3a:8f:91:bf > 44:4c:a8:c1:78:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

11:19:19.774420 44:4c:a8:c1:78:69 > 28:99:3a:8f:91:bf, ethertype IPv4 (0x0800), length 126: (tos 0xc0, ttl 64, id 29177, offset 0, flags [none], proto ICMP (1), length 112)

    10.1.2.2 > 1.1.1.1: ICMP time exceeded in-transit, length 92

(tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

Now we can see the reason clearly. The eBGP TCP session is default with ttl 1 and copied to outer GRE packets, so the packets get TTL expired at 10.1.2.2 which is R2. 

To fix this issue, just need to set TTL under tunnel interface

R1.gts425(config-router-bgp)#int tu 100

R1.gts425(config-if-Tu100)#tunnel ttl 10

R1.gts425(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4                  4         4    0    0 00:00:01 Estab  0      0

Arista MPLS VPN Step by Step (2c) L2VPN - Tcpdump

One powerful tool in Arista/EOS is tcpdump. In this setup, for example, if you see some traffic issue and want to see if the PE sending out the packets, you can do the tcpdump this way:

If you want to check the type 5 pw between CE1 and CE2, 

Step 1: find out the mpls ldp tunnel

PE1(config)#sh patch panel forwarding
In/Out                     Type CW VLAN Status Patch
-------------------------- ---- -- ---- ------ --------------------
....
Et47/3                        5         Up     ce1-ce2-pw5-1
   Label 132770, LDP Tun 5  <<< tunnel 5

Step 2: check mpls ldp tunnel 5 fwding info so have the interface name

PE1(config)#sh mpls ldp tunnel 5
   Index       Endpoint              Nexthop         Interface      Labels
----------- --------------------- --------------- ----------------- ----------
   5           88.255.22.22/32       88.1.11.1       Ethernet3/1    [ 100005 ]

Step 3: create the monitor session to dump packet to cpu

monitor session ldp-et3-1 source Ethernet3/1

monitor session ldp-et3-1 destination Cpu

Step 4: tcpdump the packet

bash tcpdump -nevvi mirror0 ether proto 0x8847 -w /mnt/flash/ldp.pcp

TCP Notes (1)

How TCP Works @ youtube

• TCP fundamentals Part 2
• 2 related TCP windows: 
• CWND - Congestion Window
• RWND - Receive Window
• Only RWND advertized in packets
• minimum of CWND and RWND is the cap of transmission
• CWND = n * MSS, n = 1,2,4...., slowstart
• not advertised, changing
• Example 1, slow backup between 2 local servers
• from the trace, recvr to sender 只发ACKs, (len = 60) 
• window size 每隔一段时间，会从64K一直降到1000 (小于MSS). 然后hold for 190ms. 然后window size back to 64K
• 显示chart - statistics -> TCP stream graph -> Time sequence (tcptrace)
• 足够的证据是server side问题
• Example 2, 是sender application无法fill CWND,
• Example 3, ip tcp mss-adjust问题
• server/client, 2个pcap
• client直接看到seq = 1461, 不是1
• server一直re-send seq = 1, size = 1460 pkts. 
• TCP NoOP option - 很简单，#TcpLen * 4 = Tcp Header Len，而TCP Options Len不全是4的倍数，所以TCP Header需要NoOp pad out header size to a multiple of 4B. 
• TCP option - timestamp
• TSval, EchoTSval, 为了计算RTT and RTO (retrans timeout)
• Some TCP options:
• 0 - end of TCP options
• 1 - NoOP
• 2 - MSS
• 3 - Window Scale
• 4/5 - SACK
• 8 - timestamp

EOS - CPU monitor session

monitor session 1 source Ethernet3/1
monitor session 1 destination Cpu

wa463.vtepB2(config)#show monitor sess

Session 1
------------------------
Source Ports:
  Both:        Et3/1
Destination Ports:
    Cpu :  active (mirror0) <<< use this interface to tcpdump


wa462.vtepB1#bash tcpdump -nvvi mirror0 udp
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:56:32.121346 44:4c:a8:a5:11:40 > 44:4c:a8:97:8c:51, ethertype IPv4 (0x0800), length 164: (tos 0x0, ttl 63, id 47642, offset 0, flags [DF], proto UDP (17), length 150)
    200.255.200.34.62729 > 200.255.200.56.4789: VXLAN, flags [I] (0x08), vni 2000
00:1c:73:ff:3c:30 > 44:4c:a8:97:72:b7, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 32537, offset 0, flags [none], proto ICMP (1), length 100)

Arista EOS: tcpdump the VRF interface

Use the Linux name space: 

[admin@dc7050 ~]$ sudo ip netns exec ns-<vrf> tcpdump -i vlan2101 arp

Tips on EOS tcpdump

Since Arista EOS is based on Linux kernel, it inherits the tcpdump, the powerful debug tool. Here is some tips on how to use it in EOS.

Reference (most contents from) : Using tcpdump for troubleshooting @ Arista.com. A refresh post of old blog. 

1. Use "bash ifconfig" to find out interface names

bn303.jhm.mlagA.profA0.w.16:01:14#bash ifconfig | grep mtu
et10_10_1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9214
et10_10_2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9214
....
vlan200: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
vlan201: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

2. dump pkts on vlan interface

bn303.jhm.mlagA.profA0.w.16:02:34#bash tcpdump -i vlan200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan200, link-type EN10MB (Ethernet), capture size 262144 bytes
16:11:35.518187 de:ad:be:ef:ba:11 (oui Unknown) > 33:33:00:00:00:01 (oui Unknown), ethertype IPv6 (0x86dd), length 86: 2000:22:0:c8::fd > ff02::1: ICMP6, neighbor advertisement, tgt is 2000:22:0:c8::fd, length 32

3. dump pkts on ethernet interface with more info

bn303.jhm.mlagA.profA0.w.16:11:41#bash tcpdump -i et10_10_1 -v -vv
tcpdump: listening on et10_10_1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:12:58.382825 28:99:3a:10:3b:90 (oui Arista Networks) > 01:80:c2:00:00:0e (oui Unknown), ethertype LLDP (0x88cc), length 324: LLDP, length 310
Chassis ID TLV (1), length 7
  Subtype MAC address (4): 44:4c:a8:a5:11:40 (oui Arista Networks)
  0x0000:  0444 4ca8 a511 40
Port ID TLV (2), length 16
  Subtype Interface Name (5): Ethernet10/10/1
  0x0000:  0545 7468 6572 6e65 7431 302f 3130 2f31

4. Run tcpdump directly in EOS

bn303.jhm.mlagA.profA0.w.15:35:13#tcpdump int eth3/1/1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et3_1_1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:54:45.654907 44:4c:a8:97:8c:52 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110

bn303.jhm.mlagA.profA0.w.15:56:37#tcpdump int vlan 2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes
15:57:05.393026 de:ad:be:ef:ba:11 > Broadcast, ethertype ARP (0x0806), length 56: Reply 22.0.2.253 is-at de:ad:be:ef:ba:11, length 42

5. dump ospf packets

# bash tcpdump -ni vlan1224 proto ospf

yo411.15:24:18#bash tcpdump -ni vlan1224 proto ospf
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan1224, link-type EN10MB (Ethernet), capture size 262144 bytes
15:24:31.110294 00:1c:73:c6:b0:0d > 01:00:5e:00:00:05, ethertype IPv4 (0x0800), length 82: 110.2.13.10 > 224.0.0.5: OSPFv2, Hello, length 48

15:24:35.394905 00:1c:73:44:58:d0 > 01:00:5e:00:00:05, ethertype IPv4 (0x0800), length 82: 110.2.13.2 > 224.0.0.5: OSPFv2, Hello, length 48

6. dump mlag udp KA 

[admin@bn302 ~]$ tcpdump -i ma1_1 udp 4432 -vv -X
tcpdump: syntax error in filter expression: syntax error
[admin@bn302 ~]$ tcpdump -i ma1_1 udp port 4432 -vv -X
tcpdump: listening on ma1_1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:03:22.707628 44:4c:a8:34:cc:d0 (oui Arista Networks) > 00:1c:73:ac:36:09 (oui Arista Networks), ethertype IPv4 (0x0800), length 121: (tos 0xe0, ttl 255, id 41931, offset 0, flags [DF], proto UDP (17), length 107)
    bn302-1.sjc.aristanetworks.com.4432 > bn303.sjc.aristanetworks.com.4432: UDP, length 79
0x0000:  45e0 006b a3cb 4000 ff11 715a ac1e 871e  E..k..@...qZ....
0x0010:  ac1e 8620 1150 1150 0057 65e4 0224 7688  .....P.P.We..$v.
0x0020:  ecb6 2000 7564 7048 6561 7274 6265 6174  ....udpHeartbeat
0x0030:  80c0 8000 0000 0100 0000 08ff f000 0000  ................
0x0040:  0000 00ff f000 0000 0000 0040 b24f ef05  ...........@.O..
0x0050:  4984 c84d 0100 0869 6e61 6374 6976 654d  I..M...inactiveM

0x0060:  0200 0800 0001 7900 0000 02              ......y....

7. Other useful options and samples

bash tcpdump -nevvi et1 ether dst host 01:80:c2:00:00:0e
bash tcpdump -nevvi any host 10.1.1.1


[admin@wa466 ~]$ tcpdump -vv ip6 -i et56_1   -w /mnt/flash/bgp.cap
tcpdump: listening on et56_1, link-type EN10MB (Ethernet), capture size 262144 bytes
2 packets captured
...

[admin@wa466 ~]$ tcpdump -vvv -r /mnt/flash/bgp.cap -n | more
reading from file /mnt/flash/v6.bgp.cap, link-type EN10MB (Ethernet)
22:17:32.070318 44:4c:a8:97:72:b7 > 00:12:01:00:00:01, ethertype IPv6 (0x86dd), length 94: (hlim 255, next-header TCP (6) payload length: 40) 2000:88:88
::1.48744 > 2000:88:88::2.bgp: Flags [S], seq 4088296625, win 28800, options [mss 1440,sackOK,TS val 3863837 ecr 0,nop,wscale 7], length 0
22:17:32.070466 00:12:01:00:00:01 > 44:4c:a8:97:72:b7, ethertype IPv6 (0x86dd), length 94: (class 0xc0, hlim 64, next-header TCP (6) payload length: 40)
 2000:88:88::2.bgp > 2000:88:88::1.48744: Flags [S.], seq 3002335018, ack 4088296626, win 14280, options [mss 1440,sackOK,TS val 8967227 ecr 3863837,nop
,wscale 9], length 0
22:17:32.070489 44:4c:a8:97:72:b7 > 00:12:01:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header TCP (6) payload length: 32) 2000:88:88
::1.48744 > 2000:88:88::2.bgp: Flags [.], seq 1, ack 1, win 225, options [nop,nop,TS val 3863837 ecr 8967227], length 0
22:17:32.070661 44:4c:a8:97:72:b7 > 00:12:01:00:00:01, ethertype IPv6 (0x86dd), length 141: (class 0xc0, hlim 255, next-header TCP (6) payload length: 8
7) 2000:88:88::1.48744 > 2000:88:88::2.bgp: Flags [P.], seq 1:56, ack 1, win 225, options [nop,nop,TS val 3863837 ecr 8967227], length 55: BGP
Open Message (1), length: 55
  Version 4, my AS 23456, Holdtime 180s, ID 5.5.5.5
  Optional parameters, length: 26
    Option Capabilities Advertisement (2), length: 24
      Multiprotocol Extensions (1), length: 4
AFI IPv6 (2), SAFI Unicast (1)
0x0000:  0002 0001
      Route Refresh (2), length: 0
      Graceful Restart (64), length: 2
Restart Flags: [none], Restart Time 300s
0x0000:  012c
      32-Bit AS Number (65), length: 4
 4 Byte AS 4264492530
0x0000:  fe2e fdf2
      Multiple Paths (69), length: 4
AFI IPv6 (2), SAFI Unicast (1), Send/Receive: Receive
0x0000:  0002 0101