Arista EOS/Networking Tips by Solomon Yang

4/11/2018

MLAG vs LACP fast rate

Arista doesn't recommend LACP rate fast enabled with MLAG, which is not very reliable especially in scale environment.

For example:

mlagA/Primary mlagB/Secondary
\ /
\ /
\ /
mlagLeaf

When either primary or secondary peer reloads, the mlagLeaf will see LAG flaps during this process.

In 4.20.2.1F Release Note, on page 21

LACP PDU fast rate ("lacp rate fast" on an Arista switch) should not be configured on any port in an MLAG pair, or any port connected to an MLAG pair. (13950)

Reference:
bug13950/SR 101110/Old post

Max # of MLAGs supported on Arista switch

One the biggest limitations of # MLAG support on an Arista switch, is the port-id. If you see the following the error message when configuring MLAG port-channels, it means you hit the ceiling.
Mar 25 23:44:55 Arista.EOS Ebra: %SYS-1-PORT_ID_ERROR: Unable to allocate a port ID for interface

Well, this kind of failure is quite rare especially if your devices are only fixed 1RU or 2RU systems or the # of mlag port-channels is below 100. But if your system is a heavily loaded modular system like 7508 or 7512 with all LCs plugged, it may hit you surprisingly.

What is the port-id, why important?

The port-id is used in LACP and STP. In STP, the port-id field is 16-bit, after 4-bit for priority, only 12 bit left for port-id, which is 4096.
2 MLAG peers are considered as 1 virtual switch, which means MLAG agent also needs to assign a port-id for corresponding peer-port and track their status. So now the space is cut half to 2048.
All physical ports need port-id, including ethernet and port-channel.
Considering a 7508N chassis with 8 x 7500R-36CQ-LC LC, how many port-id consumed?

By default, 1 QSFP port reserve 4 port-ids, no matter the speed or up/down status - "transceiver qsfp default-mode 4x10G"
1 36CQ-LC = 4 x 36 = 144 port-ids
8 36CQ-LC = 144 x 8 = 1152
After all, we still have 2048 - 1152 = 896 port-ids available for port-channels.

And still need some for L3 uplink, peer link and management interfaces, say 32.
Now the max # is 896 - 32 = 864

What happens if you have a 7512N with 12 x 36CQ LC?

12 36CQ-LC = 144 x 12 = 1728 port-id
Max# of MLAG port-channel = 2048 - 1728 - 32 = 285
Opps! Surprising, right? A chassis with more slots and more LCs supports much fewer MLAGs.
So the best system for scale MLAG is 7508 chassis with 8 x 36CQ LC.

How to check port-id usage?

Arista.EOS#bash echo "ls -l /ar/Sysdb/interface/eth/portid" | python -m Acons Sysdb | more

Connecting to local unix socket @00002 ...

Connected to process 2244

.....

default collection portId has:

Ethernet10/1/1 !!! local ethernet ports

Ethernet10/1/2

.....

Management1/1 !!! local management interfaces

Management1/2

PeerEthernet10/1/1 !!! remote ethernet ports

PeerEthernet10/1/2

....

PeerPort-Channel1001 !!! remote port-channels

PeerPort-Channel1002

....

Port-Channel1001 !!! local port-channels

Port-Channel1002

4049

A refresh post of 2 old blog posts, 1 and 2.

4/10/2018

Arista EOS: CLI to predict egress LAG/ECMP interface

This is a very useful command when troubleshooting packet loss in an ECMP link. A refresh post of an old post.

arista.EOS.7050SX#sh ip route 22.0.2.91
B E 22.0.2.0/24 [200/0] via 210.100.21.0, Ethernet11/1
via 210.100.22.0, Ethernet21/1
via 210.100.21.12, Vlan3304
via 210.100.21.14, Vlan3305
via 210.100.21.16, Vlan3306
via 210.100.22.12, Vlan3404
via 210.100.22.14, Vlan3405
via 210.100.22.16, Vlan3406
via 210.100.21.2, Port-Channel1931
via 210.100.21.4, Port-Channel1932
via 210.100.22.2, Port-Channel1941
via 210.100.22.4, Port-Channel1942
via 210.100.21.6, Ethernet12/1.3301
via 210.100.21.8, Port-Channel1933.3302
via 210.100.21.10, Port-Channel1934.3303
via 210.100.22.6, Ethernet22/1.3401
via 210.100.22.8, Port-Channel1943.3402
via 210.100.22.10, Port-Channel1944.3403

arista.EOS.7050SX#sh load-balance destination ip ingress-interface ethernet 36 src-ipv4-addr 20.0.1.100 dst-ipv4-addr 22.0.2.91 ip-protocol 9
Output Interface: Port-Channel1934.3303

arista.EOS.7050SX#sh load-balance destination ip ingress-interface ethernet 36 src-ipv4-addr 20.0.1.100 dst-ipv4-addr 22.0.2.91 ip-protocol 1
Output Interface: Port-Channel1941

show load-balance destination port-channel portchannel [ingress-interface intf] [src-mac srcmac] [dst-mac dstmac] [eth-type ethtype] [src-ip-address src-ip] [dst-ip-address dst-ip] [ip-ttl ttl] [flow-label fl] [ip-protocol proto] [src-l4-port src-port] [dst-l4-port dst-port] [vlan vlan] [non-unicast]

4/09/2018

Arista EOS: ASU/ASU2

Why needed:

Requested by tier-1 cloud/internet customers
For the servers connecting to only 1 TOR switch, the software upgrade down time is very critical because of lacking redundant path.

How does it work?

To look deep into the process of an Arista switch upgrade/reload (just a rough one):

1. Kernel reboot
2. EOS starting
3. Hardware drivers are called to refresh HW
4. Control plane is converged and create forwarding information
5. Hw programming

The whole idea is: keep the hw programing intact while starting software; then re-program hw entries to reflect new forwarding decisions.

AUS vs ASU2:

ASU: retain the hw programming and keep forwarding till step 3. Maximum down time is around 30 seconds
ASU2: A big step further, no hw hard reset but a soft one. Plus hw overwrite so maximum down time is lowered to 300 msec. And graceful-restart must be turned on.

Limitation:

only Trident2 platform supported

CLI:

reload fast-boot - ASU
reload hitless - ASU2

Arista EOS, MLAG (3) - MLAG peer gateway

Arista: MLAG peer gateway
Cisco: vPC peer gateway

Why do we need it?

Some vendors like NetApp and F5, they use switch system MAC to construct packets instead of virtual MAC.
Or it is by a feature called fast path, which just flip the source and destination MAC.
when mlagA receives a packet with destination mac of mlagB, it has to forward it to mlag peer via peer link, which is not a favored behavior

Configuration:

ip virtual-router mac mlag-peer

How does it work?

MLAG peer create an additional MY_MAC entry with MLAG peer address.
When it gets packet with this MAC, just route them instead of fwding via peer link

Useful command and supported platforms

T2 and Arad
show ip virtual-router
show platform trident l3 shadom my-station

Arista EOS, MLAG (2) - Dual-Primary Detection

MLAG Dual-Primary Detection

This is a long-waiting MLAG feature and was introduced from EOS 4.20.1F around late 2017. When peer link is down, the secondary takes over primary. But sometime, the problem is only on the peer link and the peer is still alive. Without this feature, there will be dual MLAG primary and caused traffic disruption like bursty traffic loop.

But if you enable this feature, MLAG will communicate with peer via management interface + peer link. Since out-of-band management interface is considered less chance to be clogged, it will prevent above dual primary scenario.

How does it work?

When peer link is down, the secondary takes over primary immediately.
Meanwhile it starts the dual-primary detection.
If the peer receives heartbeats, it concludes a dual-primary found. As a result, secondary peer will disable ALL interfaces to avoid loop.
When peer link is up, it will start MLAG negotiation and recovers

Here is the configuration:

mlag configuration
peer-address heartbeat 172.30.134.180
! heartbeat via management ip address
dual-primary detection delay 10 action errdisable all-interfaces

How to verify:

Arista.EOS#show mlag det
MLAG Configuration:
domain-id : pg.mlag.leaf1
local-interface : Vlan4094
peer-address : 192.168.255.0
peer-link : Port-Channel2000
hb-peer-address : 172.30.134.181
peer-config : consistent

MLAG Status:
state : Active
negotiation status : Connected
peer-link status : Up
local-int status : Up
system-id : 46:4c:a8:97:83:7d
dual-primary detection : Configured

What happen if heartbeat connection has issue? If there is misconfiguration like missing vrf in hb-peer-address line, or out-of-band management network has connection issue, the system will report

Arista.EOS#sh mlag det
MLAG Configuration:
domain-id : pg.mlag.leaf1
......
MLAG Detailed Status:
....
Heartbeat timeouts since reboot : 1
UDP heartbeat alive : False

Arista.EOS#show logg | grep MLAG-3

Apr 10 00:06:28 Arista.EOS Mlag: %MLAG-3-PEER_HEARTBEAT_TIMEOUT: MLAG stopped receiving UDP heartbeats from the peer 172.30.134.180.

MLAG split-brain

When the mlag split brain happens (the trigger is to disconnect the peer link), both leaf could hash BPDU to 1 peer, and the other peer doesn't receive any BPDU, so all ports are in forwarding and causes a loop.

The STP may kick in and put ports in designated-dispute mode. But after 2 x fdWhile timer, another round of negotiation starts and form a bursty loop every 2 seconds.

An interesting RFE 11825.

FHRP/GLBP vs VRRP vs VARP

Reference links:

1. Active-Active by using VARP - "ip virtual address x.x.x.x" w/o mask

Arista recommends VARP, which is Active-Active Redundancy
HSRP and VRRP是Active-Standby Redundancy

Host发送数据包，如果hash到VRRP slave/HSRP standby，是不处理的，而是bridge到对方，这个是MLAG里面最不希望的情况，就是数据走PeerLink
如果想Active-Active，客户必须手工分配Vlan和Master，比较烦

Configuration is quite simple:

ip virtual mac-address feed:dead:beef
interface vlan 101

ip address 10.10.10.2/24
ip virtual address 10.10.10.1

2. Comparison of virtual ip commands

Summary:

MLAG/"ip virtual addr x.x.x.x":

when sending ARP requests to dstHost, it uses its own eth hw MAC!!
so ARP Resonse肯定回到Sender, 即使ARP Response hash到另外一个Peer, eth hw is booked peerLink.

VXLAN+MLAG/"ip addr virtual x.x.x.x"

uses vMAC in the ARP requests to dstHost;
so need ARP sync between mlag peers.

"ip virtual addr x.x.x.x/24" vs "ip addr virtual x.x.x.x/24"

"ip addr virtual" for VXLAN anycast
"ip airtual addr" for non-VXLAN MLAG, 看上面

"ip virtual address <vip>/mask"，无论有没有mask，必须有phy IP.
vlan - "ip virtual address 10.10.10.1" + global - "ip virtual mac aa:bb:cc"

标准的VARP config
host resolves GW

GARP: dMac=ff:ff:ff | sMAC=aa:bb:cc | sndHw=aa:bb:cc | sndIP = vip,10.10.10.1
关键是vMAC aa:bb:cc hosts vIP
ARP reply: 如何一个收到Arp req 的router都可以返回, dMac = hostMAC, 其他和上面一样

GW resolves host:

ARP request: dMAC=ff:ff:ff | sMAC = swMAC | sndHW = swMAC | sndIP = phyIP, 不是vIP
所以会有mlag-peer的问题?
不需要ARP sync, 因为sndHW = swMAC?

vlan - "ip virtual address 10.10.10.1/24" + global - "ip virtual mac aa:bb:cc"

注意是有network mask. 而且phy ip 和 vip不是一个网段
还要增加一个static route - "ip route 10.10.10.0/24 vlan 100"

为了节省phy ip, 如果一个vlan across 100 vtep/mlag, 200 phy ip 就被用掉了。

host resolves GW:

GARP/ARP和前面一样

GW resolves host:

ARP req: dMAC=allFF | sMAC = swMAC | sndHW = vMAC | sndIP = vIP
需要ARP sync, 因为sndHW = vMAC, sw2收到应该sw1的ARP reply，也吃了因为是vMAC?

vlan - "ip address virtual 10.10.10.1/24" + global - "ip virtual mac aa:bb:cc"

不需要phy IP under vlan
专门为VXLAN, = Anycast Gateway
host resolves GW:

没有GARP，为啥？
只能是ARP reply, 里面全virtual, srcMAC = vMAC | sndHW = vMAC | sndIP = vIP

GW resolves host:

也一样，全是vritual address
需要ARP sync

Arista EOS, MLAG (1) - Basic Configuration

Arista MLAG - basic configuration

For the details, please refer above link. Here is the list of needed configurations:

Port-channel for peer link
VLAN for peer link
L3 SVI for peer link VLAN
MLAG configuration:

domain-id (must be matched on both peers)
local-interface (above L3 SVI)
peer-address
peer-link

Useful commands:

show mlag detail
show mlag interface

Arista EOS: trunk group

From the below link of "MLAG - basic configuration", there have the following configuration of MLAG peer link VLAN:

vlan 4094
trunk group peerLink
interface port-ch 2000
switch trunk group peerLink

Why do we need to assign vlan 4094 to trunk group? Quota from the reference of EOS configuration - VLAN,

"VLANs configured in a trunk group are 'pruned' off all ports that are not associated with the trunk group".

So if you have other ethernet or port-channels are configured as trunk, by default ALL VLANs are enabled if you don't specify the VLAN range. In this case, peerLink VLAN is also included. This is not an expected behavior or good practice, since the peerLink carries the critical control plane information.
But if you configure VLAN in trunk group, you have to specify the physical interfaces are associated with this trunk group. This is to provide a casual configuration causing the disruption of peerLink Vlan

4/06/2018

Tips on EOS tcpdump

Since Arista EOS is based on Linux kernel, it inherits the tcpdump, the powerful debug tool. Here is some tips on how to use it in EOS.

Reference (most contents from) : Using tcpdump for troubleshooting @ Arista.com. A refresh post of old blog.

1. Use "bash ifconfig" to find out interface names

bn303.jhm.mlagA.profA0.w.16:01:14#bash ifconfig | grep mtu
et10_10_1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9214
et10_10_2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9214
....
vlan200: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
vlan201: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

2. dump pkts on vlan interface

bn303.jhm.mlagA.profA0.w.16:02:34#bash tcpdump -i vlan200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan200, link-type EN10MB (Ethernet), capture size 262144 bytes
16:11:35.518187 de:ad:be:ef:ba:11 (oui Unknown) > 33:33:00:00:00:01 (oui Unknown), ethertype IPv6 (0x86dd), length 86: 2000:22:0:c8::fd > ff02::1: ICMP6, neighbor advertisement, tgt is 2000:22:0:c8::fd, length 32

3. dump pkts on ethernet interface with more info

bn303.jhm.mlagA.profA0.w.16:11:41#bash tcpdump -i et10_10_1 -v -vv
tcpdump: listening on et10_10_1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:12:58.382825 28:99:3a:10:3b:90 (oui Arista Networks) > 01:80:c2:00:00:0e (oui Unknown), ethertype LLDP (0x88cc), length 324: LLDP, length 310
Chassis ID TLV (1), length 7
Subtype MAC address (4): 44:4c:a8:a5:11:40 (oui Arista Networks)
0x0000: 0444 4ca8 a511 40
Port ID TLV (2), length 16
Subtype Interface Name (5): Ethernet10/10/1
0x0000: 0545 7468 6572 6e65 7431 302f 3130 2f31

4. Run tcpdump directly in EOS

bn303.jhm.mlagA.profA0.w.15:35:13#tcpdump int eth3/1/1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et3_1_1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:54:45.654907 44:4c:a8:97:8c:52 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110

bn303.jhm.mlagA.profA0.w.15:56:37#tcpdump int vlan 2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes
15:57:05.393026 de:ad:be:ef:ba:11 > Broadcast, ethertype ARP (0x0806), length 56: Reply 22.0.2.253 is-at de:ad:be:ef:ba:11, length 42

5. dump ospf packets

# bash tcpdump -ni vlan1224 proto ospf

yo411.15:24:18#bash tcpdump -ni vlan1224 proto ospf
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan1224, link-type EN10MB (Ethernet), capture size 262144 bytes
15:24:31.110294 00:1c:73:c6:b0:0d > 01:00:5e:00:00:05, ethertype IPv4 (0x0800), length 82: 110.2.13.10 > 224.0.0.5: OSPFv2, Hello, length 48

15:24:35.394905 00:1c:73:44:58:d0 > 01:00:5e:00:00:05, ethertype IPv4 (0x0800), length 82: 110.2.13.2 > 224.0.0.5: OSPFv2, Hello, length 48

6. dump mlag udp KA

[admin@bn302 ~]$ tcpdump -i ma1_1 udp 4432 -vv -X
tcpdump: syntax error in filter expression: syntax error
[admin@bn302 ~]$ tcpdump -i ma1_1 udp port 4432 -vv -X
tcpdump: listening on ma1_1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:03:22.707628 44:4c:a8:34:cc:d0 (oui Arista Networks) > 00:1c:73:ac:36:09 (oui Arista Networks), ethertype IPv4 (0x0800), length 121: (tos 0xe0, ttl 255, id 41931, offset 0, flags [DF], proto UDP (17), length 107)
bn302-1.sjc.aristanetworks.com.4432 > bn303.sjc.aristanetworks.com.4432: UDP, length 79
0x0000: 45e0 006b a3cb 4000 ff11 715a ac1e 871e E..k..@...qZ....
0x0010: ac1e 8620 1150 1150 0057 65e4 0224 7688 .....P.P.We..$v.
0x0020: ecb6 2000 7564 7048 6561 7274 6265 6174 ....udpHeartbeat
0x0030: 80c0 8000 0000 0100 0000 08ff f000 0000 ................
0x0040: 0000 00ff f000 0000 0000 0040 b24f ef05 ...........@.O..
0x0050: 4984 c84d 0100 0869 6e61 6374 6976 654d I..M...inactiveM

0x0060: 0200 0800 0001 7900 0000 02 ......y....

7. Other useful options and samples

bash tcpdump -nevvi et1 ether dst host 01:80:c2:00:00:0e
bash tcpdump -nevvi any host 10.1.1.1

[admin@wa466 ~]$ tcpdump -vv ip6 -i et56_1 -w /mnt/flash/bgp.cap
tcpdump: listening on et56_1, link-type EN10MB (Ethernet), capture size 262144 bytes
2 packets captured
...

[admin@wa466 ~]$ tcpdump -vvv -r /mnt/flash/bgp.cap -n | more
reading from file /mnt/flash/v6.bgp.cap, link-type EN10MB (Ethernet)
22:17:32.070318 44:4c:a8:97:72:b7 > 00:12:01:00:00:01, ethertype IPv6 (0x86dd), length 94: (hlim 255, next-header TCP (6) payload length: 40) 2000:88:88
::1.48744 > 2000:88:88::2.bgp: Flags [S], seq 4088296625, win 28800, options [mss 1440,sackOK,TS val 3863837 ecr 0,nop,wscale 7], length 0
22:17:32.070466 00:12:01:00:00:01 > 44:4c:a8:97:72:b7, ethertype IPv6 (0x86dd), length 94: (class 0xc0, hlim 64, next-header TCP (6) payload length: 40)
2000:88:88::2.bgp > 2000:88:88::1.48744: Flags [S.], seq 3002335018, ack 4088296626, win 14280, options [mss 1440,sackOK,TS val 8967227 ecr 3863837,nop
,wscale 9], length 0
22:17:32.070489 44:4c:a8:97:72:b7 > 00:12:01:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header TCP (6) payload length: 32) 2000:88:88
::1.48744 > 2000:88:88::2.bgp: Flags [.], seq 1, ack 1, win 225, options [nop,nop,TS val 3863837 ecr 8967227], length 0
22:17:32.070661 44:4c:a8:97:72:b7 > 00:12:01:00:00:01, ethertype IPv6 (0x86dd), length 141: (class 0xc0, hlim 255, next-header TCP (6) payload length: 8
7) 2000:88:88::1.48744 > 2000:88:88::2.bgp: Flags [P.], seq 1:56, ack 1, win 225, options [nop,nop,TS val 3863837 ecr 8967227], length 55: BGP
Open Message (1), length: 55
Version 4, my AS 23456, Holdtime 180s, ID 5.5.5.5
Optional parameters, length: 26
Option Capabilities Advertisement (2), length: 24
Multiprotocol Extensions (1), length: 4
AFI IPv6 (2), SAFI Unicast (1)
0x0000: 0002 0001
Route Refresh (2), length: 0
Graceful Restart (64), length: 2
Restart Flags: [none], Restart Time 300s
0x0000: 012c
32-Bit AS Number (65), length: 4
4 Byte AS 4264492530
0x0000: fe2e fdf2
Multiple Paths (69), length: 4
AFI IPv6 (2), SAFI Unicast (1), Send/Receive: Receive
0x0000: 0002 0101

Eos: last 100 commands you ran

Starting from bloomington release - 4.20.4F, the following command will print out last 100 commands you ran and duration details.

bn303.jhm.mlagA.profA0.w.14:21:02#bash CliCtrl --print-state | more
Activity Lock owner is: 0
Recently run commands:
TID: 17702, Duration: 0.0779159069061*, Start: 1523050110.22, End: None, Cmd: bash CliCtrl --print-state | more
TID: 17702, Duration: 0.139348983765, Start: 1523049662.16, End: 1523049662.3, Cmd: bash CliCtrl --print-state
TID: 17702, Duration: 0.00744891166687, Start: 1523049649.22, End: 1523049649.23, Cmd: enable
TID: 11106, Duration: 0.0013120174408, Start: 1523046913.33, End: 1523046913.33, Cmd: show monitor device-health category
TID: 11106, Duration: 0.00137400627136, Start: 1523046913.33, End: 1523046913.33, Cmd: show monitor device-health component
TID: 11106, Duration: 0.107572078705, Start: 1523046913.22, End: 1523046913.33, Cmd: show agent event history

3/30/2018

BGP session by loopback vs combo static route

Topology:

Configuration:

Says, in the above topology, there is 2 eBGP sessions:

Between R0 and R1, eBGP session is by loopback address 10.255.255.1 and 10.255.255.2 respectively. No IGP involved, to make it work, we have the following configurations:

On R0:
ip route 10.255.255.1/32 10.0.1.1 ! 10.0.1.1 = R1's intf ip address
ip route 10.255.255.1/32 10.0.1.3 ! 10.0.1.3 = R1's intf ip address
!
router bgp 65000
maximum-paths 128
neighbor 10.255.255.1 remote-as 65001
neighbor 10.255.255.1 ebgp-multihop 2

And between R0 and R2, there is an eBGP session. Also R2 advertises a prefix of 10.0.1.0/24, which is the supernet of 10.0.1.0/31

BGP sessions are all up w/o any issues.

R0(s2)(config)#sh ip bgp sum
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
10.0.2.1 4 65002 202 201 0 0 03:11:40 Estab 1 1
10.255.255.1 4 65001 1867 1887 0 0 01:31:06 Estab 0 0

Issue:

Now, let's disable 1 of 2 back-to-back connections between R0 and R1. And eBGP session between R0 and R1 is down! even the 2nd connection is of no problem.

R0(s2)(config)#int e6/45
R0(s2)(config-if-Et6/45)#shu
R0(s2)(config-if-Et6/45)#sh ip bgp sum
BGP summary information for VRF default
Router identifier 72.129.223.37, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
10.0.2.1 4 65002 205 204 0 0 03:14:06 Estab 1 1
10.255.255.1 4 65001 1914 1933 0 0 00:00:05 Connect

Why? It is because the route to 10.255.255.1 pointing to unexpected interface

R0(s2)(config-if-Et6/45)#sh ip route 10.255.255.1
.......
S 10.255.255.1/32 [1/0] via 10.0.2.1, Ethernet6/9 ! R0/R2 intf
via 10.0.1.3, Ethernet6/46

Remember the static route we created above
ip route 10.255.255.1/32 10.0.1.1 ! 10.0.1.1 = R1's intf ip address

R0(s2)(config-if-Et6/45)#sh ip route 10.0.1.1
B E 10.0.1.0/24 [200/0] via 10.0.2.1, Ethernet6/9 !

Ok, the reason is because, the static route 10.0.1.0/31 is gone after shutting down interface. Then the eBGP route from R2 kicks in and now NH 10.0.1.1 points to R2 after recursive lookup. And it builds an incorrect 2-way ECMP between 10.255.255.1 and 10.255.255.2, which creates a 50% chance of dropping packets. Then BGP is down.

Workaround/Solution:

To solve this issue, we need to use combo static routes to specify the NH and egress interfaces, like:

R0(s2)(config)#sh run | grep 10.255.255.1
ip route 10.255.255.1/32 Ethernet6/45 10.0.1.1
ip route 10.255.255.1/32 Ethernet6/46 10.0.1.3

R0(s2)(config)#int e6/45
R0(s2)(config-if-Et6/45)#shu
R0(s2)(config-if-Et6/45)#sh ip route 10.255.255.1
S 10.255.255.1/32 [1/0] via 10.0.1.3, Ethernet6/46

Details of "combo" static route

3/27/2018

Trident Platform: route not hw programmed

If you see some v4 or v6 routes not hw programmed in the output of "show ip route", you may need to try to adjust hardware routing-table partition.

ck494.17:37:15.(config)#platform trident routing-table partition ?
1 16K IPv4, 6K IPv6 ( prefix len up to /64 ), 1K IPv6 ( any prefix length )
2 16K IPv4, 4K IPv6 ( prefix len up to /64 ), 2K IPv6 ( any prefix length )
3 16K IPv4, 2K IPv6 ( prefix len up to /64 ), 3K IPv6 ( any prefix length )
4 16K IPv4, 0K IPv6 ( prefix len up to /64 ), 4K IPv6 ( any prefix length )

3/21/2018

How many ACLs can fit into Arista 7508E? (1)

About this post:
In this blog, I will discuss and sample number of ACLs can be programmed on an Arista Arad-based DCS-7508E system.

System/Interface:
7508E(s1)(config-if-Et3/1/1)#show module
Module Ports Card Type Model Serial No.
------- ----- ------------------------------- ------------- -----------
1 3 DCS-7500E-SUP Supervisor Module DCS-7500E-SUP JPE13493743
2 3 DCS-7500E-SUP Supervisor Module DCS-7500E-SUP JPE15498473
3 144 36 port 40GbE QSFP+ Linecard 7500E-36Q-LC JPE13874943
4 144 36 port 40GbE QSFP+ Linecard 7500E-36Q-LC JPE13493196

List of contents:
1) Overview of Arista LC/Arad/TCAM banks
2) System starts with zero configuration.
3) I will configure v4/v6 ACL on eth3/13/1 ~ 3/18/4, which are under Arad3/2
4) And check output of "show platform arad..." to monitor the TCAM usage

1. Overview of Arista LC/Arad/TCAM banks

Arista 7500E series is an Arad based switch. Each Arista 7500E-36Q-LC has 6 Arad ASICs, so each chip stands behind 6 QSFP or 24 ethernet interfaces. For example, Arad3/2 maps to eth ports, Et3/13/1 - 3/18/4

7508E(s1)#show platform arad Arad3/2 mapping
Arad3/2 (FapId: 2 BaseSystemCoreId: 2)
Port SysPhyPort Voq Core FapPort OtmPort QPairs Xlge NifPort Qsgmii Serdes
------------------------------------------------------------------------------------------------------------------
.....
Ethernet3/13/1 140 4064 0 2 0 8 5 20 - (20)
Ethernet3/13/2 141 4072 0 3 8 8 - 21 - (21)
.....
Ethernet3/18/4 163 4248 0 25 184 8 - 11 - (11)

And each Arad has 14 TCAM banks, 12 (#0-11) full-size 1K (160-bit entries), 2 (#12, 13) smaller (256x160-bit entries) for system-use only

7508E(s1)(config)#show platform arad Arad3/2 tcam summary
Tcam Allocation (Arad3/2)
Bank Used By Reserved By
---------- ------------------------- -----------
0 dbPdpIp -
0 dbPdpTunnel -
0 dbPdpNonIp -
0 dbPdpIp6 -
0 dbPdpMpls -
1 dbIp6UcastRouting -
2 dbIp6UcastRouting -
3 dbVTT0 -
4 dbIpMcastRouting -
12 dbSystem6 -
12 dbMplsSystem -
12 dbSystem -
12 dbTunnelTermination -
13 dbEgressSystem -

Each bank can hold 1024 TCAM lines. So per Arad can hold total ~12K ACL rule lines in theory. BUT....from above output of "tcam summary", 5 banks (0 ~ 4) are used by Pdp, ipv6 routing, VTT (vlan translation t?) and ipv4 multicast.

In the Bank 0, there is some pdp** rules enabled, which are the default port-base CoPP policy map. And the bank usage is exclusive, so 2 types of TCAM applications can't share bank. Even the PDP is quite small, we can't squeeze other ACLs into this bank.

7508E(s1)#show platform arad Arad3/2 acl tcam summary
The total number of TCAM lines per Arad bank is 1024
========================================================
Arad3/2:
========================================================
Bank Used Used % Used By
0 74 7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 74

After all of these, there is slightly less than 7K lines left. But there is several questions need to be addressed before answering the question in the subject line.
1) Are these ACL rule lines be shared by multiple ports? ethernet or port-ch?
2) How many TCAM lines are consume for v4, v6, MAC, L4 rule?
3) How about the ACL under VLAN(SVI) interfaces?

I will figure them out in later parts.

Step 1: configure ipv4 ACL with 1 rule on 1 eth/routed port

7508E(s1)(config)#ip access-list v4-Acl-1-rule
7508E(s1)(config-acl-v4-Acl-1-rule)#permit ip 192.168.1.0/24 any
7508E(s1)(config-acl-v4-Acl-1-rule)#int et3/13/1
7508E(s1)(config-if-Et3/13/1)#ip access-group v4-Acl-1-rule in

The new ipv4 ACL is programmed into TCAM bank 5. (bank 0 - 4 are already used)

7508E(s1)#sh platform arad Arad3/2 tcam summary
Tcam Allocation (Arad3/2)
Bank Used By Reserved By
---------- ------------------------- -----------
0 dbPdpIp -
....
3 dbVTT0 -
4 dbIpMcastRouting -
5 dbIpPacl -
....

And it takes 2 entries. Why 2 entries? It is because there is 1 default deny any any at the end.

7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
Bank Used Used % Used By
5 2 0 IP PACLs
0 74 7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 76

Similar contents from the o/p of "show plat arad <id> acl tcam detail"

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
ip access-list v4-Acl-1-rule (PACL, 1 rules, 2 entries, direction in, state success)
Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1
Bank Offset Entries
5 0 2

Step 2: add one more line to same ipv4 ACL

7508E(s1)(config)#ip access-list v4-Acl-1-rule
7508E(s1)(config-acl-v4-Acl-1-rule)#perm ip 10.0.0.0/8 any
7508E(s1)(config-acl-v4-Acl-1-rule)#end
7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
Bank Used Used % Used By
5 3 0 IP PACLs
0 74 7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 77

7508E(s1)#sh ip access-lists v4-Acl-1-rule
IP Access List v4-Acl-1-rule
10 permit ip 192.168.1.0/24 any
20 permit ip 10.0.0.0/8 any

So, for the ipv4 ACL, it takes rule + 1 TCAM lines

Step 3: apply same ipv4 ACL to another eth port under same ASIC

7508E(s1)#sh ip access-lists v4-Acl-1-rule summary
IPV4 ACL v4-Acl-1-rule
Total rules configured: 2
Configured on Ingress: Et3/13/1
Active on Ingress: Et3/13/1
7508E(s1)(config)#int et3/13/2

7508E(s1)(config-if-Et3/13/2)#ip access-group v4-Acl-1-rule in

7508E(s1)#sh ip access-lists v4-Acl-1-rule summary
IPV4 ACL v4-Acl-1-rule
Total rules configured: 2
Configured on Ingress: Et3/13/1-2
Active on Ingress: Et3/13/1-2

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
ip access-list v4-Acl-1-rule (PACL, 2 rules, 3 entries, direction in, state success)
Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1, Et3/13/2
Bank Offset Entries

5 0 3

From the above output, we can see if the same ACL applied to other eth ports under same Asic, the rule entries are shared.

Step 4: configure a 1-line ipv6 ACL

7508E(s1)(config)#ipv6 access-list v6-Acl-1
7508E(s1)(config-ipv6-acl-v6-Acl-1)#perm ipv6 2000::192:168:1/120 any
7508E(s1)(config-ipv6-acl-v6-Acl-1)#int e3/13/1
7508E(s1)(config-if-Et3/13/1)#ipv6 en
7508E(s1)(config-if-Et3/13/1)#ipv6 address 2000::192:168:1:1/120
7508E(s1)(config-if-Et3/13/1)#ipv6 access-group v6-Acl-1 in
7508E(s1)(config-if-Et3/13/1)#sh ipv6 access-lists v6-Acl-1 summary
IPV6 ACL v6-Acl-1
Total rules configured: 1
Configured on Ingress: Et3/13/1
Active on Ingress: Et3/13/1

7508E(s1)(config-if-Et3/13/1)#sh platform arad Arad3/2 tcam summary
Tcam Allocation (Arad3/2)
Bank Used By Reserved By
---------- ------------------------- -----------
....
5 dbIpPacl -
6 dbIp6Pacl -
7 dbIp6Pacl -
....

7508E(s1)(config-if-Et3/13/1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
Bank Used Used % Used By
5 3 0 IP PACLs
6, 7 6 0 IPv6 PACLs
0 74 7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 89

Now, we know v6 ACL take more TCAM resources than v4:
1) Occupies 2 banks instead of 1 since the length of address;
2) Uses 6 lines for 1-line v6 ACL, because it needs 4 lines of ICMPv6 for neighbor discovery, plus 1 line of default deny all

Step 5: add 1 more line to v6 ACL

7508E(s1)(config)#ipv6 access-list v6-Acl-1
7508E(s1)(config-ipv6-acl-v6-Acl-1)#perm ipv6 2000::10:0:0/120 any
7508E(s1)(config-ipv6-acl-v6-Acl-1)#end
7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
Bank Used Used % Used By
5 3 0 IP PACLs
6, 7 7 0 IPv6 PACLs
0 74 7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 91

Ok, IPv6 ACLs
1) take 2 continuous banks
2) consume 1 line per rule; plus 5 rules for default rules

Step 6: apply same ACL to another ethernet

7508E(s1)(config)#interface Ethernet3/13/2
7508E(s1)(config-if-Et3/13/2)#ipv en
7508E(s1)(config-if-Et3/13/2)#ipv6 address 2000::192:168:2:1/120
7508E(s1)(config-if-Et3/13/2)# ipv6 access-group v6-Acl-1 in

7508E(s1)#sh ipv6 access-lists summary
....
IPV6 ACL v6-Acl-1
Total rules configured: 2
Configured on Ingress: Et3/13/1-2
Active on Ingress: Et3/13/1-2

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
......
ipv6 access-list v6-Acl-1 (PACL, 2 rules, 7 entries, direction in, state success)
Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1, Et3/13/2
Bank Offset Entries
6, 7 0 7

7508E(s1)#sh platform arad Arad3/2 acl tcam sum
Arad3/2:
5 3 0 IP PACLs
6, 7 7 0 IPv6 PACLs
0 74 7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Summary
1. ACL limitation is by Arad Chip
2. Each Arad chip is equipped with 12 TCAM banks and each holds 1K lines;
3. v4 ACL takes 1 bank and v6 uses 2
4. 1 ACL rule = 1 TCAM line
5. Same ACL applied on different ports can share TCAM lines