Arista EOS: traffic tool on Linux

https://eos.arista.com/traffic-generator-on-arista/

• iperf
• but seeing time out error? iptables issue? need to check later
• ethxmit

Arista EOS: %ETH-4-ERRDISABLE: no-internal-vlan error detected

If you are seeing the following error message 

Jan 12 19:03:19 sn452 Ebra: %ETH-4-ERRDISABLE: no-internal-vlan error detected on Ethernet81.

And the output of "show interface status" indicates some ethernet ports are error-disabled like

sn452.19:00:35(config-if-Et1-104)#show interfaces status errdisabled
   Port        Name             Status         Reason
----------- ---------------- ----------------- ----------------
   Et81                         errdisabled    no-internal-vlan
   Et82                         errdisabled    no-internal-vlan

The reason for the above error is that EOS running out of internal VLAN for routed ports. The EOS needs to assign an internal VLAN for the none switching ports. If you have the following 2 lines of configuration, you may hit this error

switchport default mode routed <<< all ports are routed
vlan internal order descending range 4001 4080 <<< but only 80 VLANs assigned

So the solution is to remove the internal VLAN range. 

EOS: sum up and compare in/egress throughput

Sometimes you want to compare the ingress/egress throughput on a particular router to see if any possible traffic loss (of course, the loss should be large enough like 3% more).  On EOS, srnz (alias srnz Show interface counters rates | nz) is a good alias. But if incoming or outgoing on multiple ports, you have to sum up and compare.

Here is a couple of useful tips and commands.

bn309#srnz
Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et9/1/1   ixia:LC8     0:05       0.0   0.0%        0   13435.8  35.0%     3543
Et9/2/1   ixia:LC8     0:05       0.0   0.0%        0   13435.3  35.0%     3543
...
Et11/6/1  ixia:LC7     0:05   13433.7  35.0%     3543       0.0   0.0%        0
Et11/11/1 ixia:LC7     0:05   13433.7  35.0%     3543       0.0   0.0%        0
Et11/12/1 ixia:LC7     0:05   13434.6  35.0%     3543       0.0   0.0%        0
Et11/13/1 ixia:LC7     0:05   13432.3  35.0%     3543       0.0   0.0%        0
Et11/14/1 ixia:LC7     0:05   13434.4  35.0%     3543       0.0   0.0%        0
Et11/15/1 ixia:LC7     0:05   13434.3  35.0%     3543       0.0   0.0%        0
Et11/16/1 ixia:LC7     0:05   13433.9  35.0%     3543       0.0   0.0%        0

In the above example, you want to compare ingress from ixia:LC7 and egress of ixia:LC8

bn309#srnz | grep LC7 | awk '{s+=$4}END{print s}'
161206  <<< ingress
bn309#srnz | grep LC8 | awk '{s+=$7}END{print s}'
161198  <<< egress

EOS: A quick alias to show the interfaces with most util%

bn302#bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | more
Mbps Kpps Port
1.6% 4.5% Et9/36/1
1.5% 4.3% Et3/36/1
0.7% 0.5% Et13/33/1
0.7% 0.4% Et12/36/2

or set it as an alias

srnzsort bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | head -n 6

Updated note: The above command works with interface names/description defined. If not, have to change awk '{print $5, $8, $1}' to awk '{print $4, $7, $1}'

hs486.11:32:15#srnz
Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et49/1                 0:05    9870.4 100.0%      812    9860.3  99.9%      811


hs486.12:14:45#bash Cli -p15 -c 'srnz' | awk '{print $4, $7, $1}' | sort -r | more
In In Port
100.0% 99.9% Et50/1
100.0% 99.9% Et49/4

How about some interfaces with description, some don't? hm.... Let me think about it..... :-)

Kill list of zombie process on Linux(Eos)

Due to a script bug, one EOS device has quite some zombie process like,

[admin@bn302 flash]$ ps aux  | grep 'CliShell -A'
root     28804  0.0  0.0  10504  8580 ?        S    10:53   0:00 CliShell -A -p 15 -c show port-channel | json
root     28809  0.0  0.0  10504  8688 ?        S    10:53   0:00 CliShell -A -p 15 -c show port-channel | json
root     28826  0.0  0.0  10504  8612 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
root     28835  0.0  0.0  10504  8580 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
root     28844  0.0  0.0  10504  8616 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
....

Instead of killing one by one manually, write a quick shell command to kill all. (killall is not supported in EOS/Linux)

[admin@bn302 flash]$ for pid in $(ps aux  | grep 'CliShell -A' | awk '{print $2, $11}' | grep CliShell | awk '{print $1}'); do sudo kill -9 $pid; done

Arista EOS Memory Util%

https://eos.arista.com/introduction-to-managing-eos-devices-memory-utilisation/

经常Arista Switch会显示Mem Util% at 70%，特别是低端的Switch。其实没有用到，而是Linux Caching System把他们Reserve到Cache和Buffer里面。下面红字的都是Free Mem

------------- show processes top memory once -------------

top - 01:14:08 up 21 days,  1:18,  1 user,  load average: 0.62, 0.39, 0.41
Tasks: 310 total,   1 running, 252 sleeping,   0 stopped,  57 zombie
%Cpu(s): 10.3 us,  2.0 sy,  0.0 ni, 87.3 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
KiB Mem:   3900776 total,  3730136 used,   170640 free,   201908 buffers
KiB Swap:        0 total,        0 used,        0 free,  2034600 cached

ckp348.jhm.ebgp.profB.WORKING.05:46:31#sh proc top mem once
top - 05:46:40 up 25 days,  7:21,  2 users,  load average: 0.42, 0.25, 0.24
Tasks: 289 total,   2 running, 287 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.4 us,  0.7 sy,  0.0 ni, 94.1 id,  0.0 wa,  0.2 hi,  0.5 si,  0.0 st
KiB Mem:   7931828 total,  4395336 used,  3536492 free,   218976 buffers
KiB Swap:        0 total,        0 used,        0 free,  2371220 cached

Arista EOS ASU and ASU2

ASU = Accelerated Software Upgrade

• EOS Docu
• 就是在Control Plane booting的时候，Hardware继续Fwd Traffic based on old state
• 等Control Plane is up and running, refresh hw. 所以Traffic disruption大概是30 seconds. 
• CLI: reload fast-boot
• 还有一个ASU+

ASU2

• 没有Document ?
• 就是加上Protocol GR，例如BGP, OSPF
• Traffic disruption is about 50ms
• CLI: reload hitless

How to see the open bug in Arista EOS

Generally, you can see the list of the open bugs in the software release note. Another way is to go to arista.com, click Support in the menu bar, then select the "Software Bug Portal".

Or if you know the specific bug id, you can get it like https://www.arista.com/en/support/software-bug-portal/bugdetail?bug_id=<bug_id>

Install Arista EOS Swix

Reference:

• Arista's - "Package RPM into an Arista swix"
• Arista Extension ylb.jp

In the Linux world, RPM package is used for software installation. Simply put, the swix (software image extension) is a software package which contains multiple RPM files and a manifest file, which is used to install software or software patch on Arista EOS. 

Generally, a swix is installed in the following steps:

1. copy to local extension directory - "scp, copy"

2. install - "extension"

3. copy them to boot-extension - "copy"

4. if a modular system, install on the standby supervisor - "session"

5. restart agents or reset hardware components if required - "agent terminate"

Additionally, 

1. uninstall - "no extension"

2. remove packet - "delete" 

3. If both sup running mismatched swix, login console and reset boot-extension file

Below is the step by step:

Step 1 - copy

[admin@eosrouter flash]$ scp solomonyang@server:/users/solomonyang/swix/*swix /mnt/flash

patch.swix                                                                            100%   11MB  11.4MB/s   00:00

eosrouter#copy flash:patch.swix extension:

eosrouter#show extensions

Name                          Version/Release      Status      Extension

----------------------------- -------------------- ----------- ---------

patch.swix                    1.0.4 .              A, NI       6

A: available | NA: not available | I: installed | NI: not installed | F: forced

eosrouter#show extensions detail

       Name: patch.swix

    Version: 1.0.4

    Release: blahblah

   Presence: available

     Status: not installed
    Summary: SandFap agent related libs

   Packages:

 Total size: 0 bytes

Description: SandFap agent related libs.

Step 2 - install

eosrouter#extension patch.swix

eosrouter#show extensions

Name                          Version/Release      Status      Extension

----------------------------- -------------------- ----------- ---------

patch.swix                    1.0.4                A, I        6

Step 3 - copy to boot extension

eosrouter#show installed-extensions

patch.swix

eosrouter#show boot-extensions

eosrouter#copy installed-extensions boot-extensions

Copy completed successfully.

eosrouter#show boot-extensions

patch.swix

Step 4 - install them on standby sup

eosrouter#copy flash:patch.swix supervisor-peer:/mnt/flash

Copy completed successfully.

eosrouter#session peer-supervisor

Warning: Permanently added '[127.1.0.2]:3601' (RSA) to the list of known hosts.

WARNING - you are currently logged in to the standby supervisor.

Not all cli commands are available or supported. Configuration mode

is disabled for SSO.

s1:eosrouter#copy flash:patch.swix extension:

Copy completed successfully.

s1:eosrouter#extension patch.swix

s1:eosrouter#copy installed-extensions boot-extensions

Copy completed successfully.

Step 5 - restart agents or reset hardware component

For some software patches, it is required to restart software agents or hardware components to make patch effective. 

eosrouter#show agent names | grep SandFap

SandFap-Linecard3

SandFap-Linecard4

SandFap-Linecard5

eosrouter#agent SandFap-Linecard3 SandFap-Linecard4 SandFap-Linecard5 terminate

SandFap-Linecard3 was terminated

SandFap-Linecard4 was terminated

SandFap-Linecard5 was terminated

Also you need to do the same on the standby sup

How to fix unmatched swix'ed sup

If you run into unmatched swix supervisor due to mis-configuration, the way to fix them is as follows:

1. Login in the console of both supervisor. Sometimes the standby sup is at bad shape, then you have to fix it in Aboot mode

Arista Networks EOS 4.20.5.2F

localhost login: admin

[PyServer ar.Aaa not responding, still trying -- is it running?]

2. Reboot both and press ctrl-C to enter Aboot mode

3. Vi /mnt/flash/boot-extensions to remove swix entries. Then reboot so you will have a clean EOS w/o any extensions. 

Arista EOS: %QOS-3-POLICY_HW_RESOURCE_FULL

When applying Qos policy under port-channel, the system doesn't accept it. 

wa463.bug228215.16:12:21(config-if-Po20)#service-policy type qos input SPTest
% Error: Cannot apply service-policy to Port-Channel20 ()

And show logg displays an error msg:
Mar  8 16:01:25 wa463 SandAcl: %QOS-3-POLICY_HW_RESOURCE_FULL: Insufficient hardware resources to program the input policy-map SPTest.

It is because the TCAM is running out. One possible reason is PDP (per-port data policy) which uses up quite some TCAM. So try the EOS-Int image. 

wa462.bug228215.16:17:04#sh platform jericho acl tcam summary
The total number of TCAM lines per Jericho bank is 2048

========================================================
Jericho0:
========================================================
   Bank   Used           Used %          Used By
      0   2046               99         IP RACLs
      1   1554               75         IP RACLs
   2, 3   2048              100       IPv6 RACLs
   4, 5   2048              100       IPv6 RACLs
   6, 7   2048              100       IPv6 RACLs
   8, 9   2046               99       IPv6 RACLs
  10,11    210               10       IPv6 RACLs
     14     79               61 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 20479

After changing to INT image, this issue is gone. 

wa462.bug228215.16:28:09(config)#int po20
wa462.bug228215.16:28:11(config-if-Po20)#service-policy type qos input SPTest
wa462.bug228215.16:28:13(config-if-Po20)#show ver
Arista DCS-7280CR-48-F
Hardware version:    11.01
Serial number:       JPE16473148
System MAC address:  444c.a897.8c51

Software image version: 4.20.0F-INT-7767198.bloomingtonrel (engineering build)

iptables in EOS

iptables is a Linux firewall utility program, which is leveraged by Arista EOS to control protocol control packets. For example:

Example: sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
-S : List the rules
-A : Append rule
-p : protocol
-j : jump = action

[admin@ck478 ~]$ sudo iptables -S
.....
-A INPUT -p tcp -m tcp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE   <<< MLAG control pkt
-A INPUT -p udp -m udp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE

! add a bgp neighbor 1.1.1.1
ck478.17:51:26(config)#router bgp 65500
ck478.17:51:43(config-router-bgp)#nei 1.1.1.1 remote 65001

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -j ACCEPT   <<< a new rule added for bgp nei 1.1.1.1

! configure bgp ttl security rule
ck478.17:51:51(config-router-bgp)#nei 1.1.1.1 ttl maximum-hops 2

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -m ttl --ttl-lt 253 -j DROP <<< all bgp pkts w/ ttl <253 droped!
-A BGP -s 1.1.1.1/32 -j ACCEPT

VXLAN Routing with MLAG

VXLAN Routing with MLAG
https://eos.arista.com/vxlan-routing-with-mlag/

• VXLAN routing routes pkt based on IP address in inner header, not outer VXLAN header. 
• 有个示意图，可以帮助理解
• 在SW-1其实就是简单的Inter VLAN SVI routing
• 只是SVI-VLAN 20有个VTEP／VNI，所以可以Learn到DEST MAC of Svr2
• Srv-2‘s GW SVI-VLAN-20 is on VTEP-1/SW-1
• VNI 1020 链接左右的L2 Domain
• Routing Topologies
• Direct Routing - routing at 1st-hop leaf node for ALL subnets. 
• Indirect Routing - only route for ONE subnet, reduce amt of ARP/MAC resource on leaf
• Indirect is a derivative of direct
• Direct Routing
• works by creating anycast IP address:
• Leaf acts as GW, owns and responds ARP req
• 所有Leaf Config same "ip address virtual" and "ip virtual mac"
• ip address virtual 10.10.10.254/24
• No routing over an VLAN interface w/ "ip address virtual"
• VTEP w/ "ip address virtual" will fwd any ARP responses to virtual router MAC to all neighbor VTEPs via HER(head-replication). So neighbor VTEPs host same ARP tables. 
• in MLAG, ARP res to "virtual ip addr" are sync'ed with MLAG peer. 
• Note: ARP sync between MLAG is done via VXLAN agent, hence "ip virtual address" is ONLY supported with VXLAN config
• virtual VTEP:
• 每个Leaf都有Virtual IP addr + MAC, 所以都可以response ARP req. 
• 所以建立一个vVTEP. 不太明白
• ARP Timer
• Serv1 sends ARP req to VTEP1. By routing, VTEP1 would learn MAC of Serv4 via initial ARP req. 
• But not via subsequent bi-directional data traffic, because returning traffic could be ECMP'd to VTEP2, which also routes and rewrite SrcMAc of  inner pkt by VTEP2 mac. 
• To avoid MAC being flush (default timeout is 5 min), it is advised to config ARP aging timeout (default 4 hours) less than MAC timeout. 
• So force a ARP refresh and re-learning MAC. 
• Direct Routing Config:
• VTEP only needs to announce its loopback/end-point into BGP. 
• Then tenant subnets exit only on the leafs, NOT in BGP or on spines. 
• show vxlan address-table
• show mac address-table

VXLAN (2) - RFC 7348

• 明白一个概念, Overlay - overlay L2 connectivity over L3 network
• Inter-VM 需要L2 access mode
• 但是DC Infra都是 L3/IP, 因为ECMP，
• 所以Overlay = provide L2 network over L3 infra
• Bcast/Unknown traffic via Mcast
• 这个在EOS里面没有implement，客户不喜欢this approach; 
• 现在就是简单Flood, 所以Mcast/Bcast/Unknown traffic会被复制多份
• Pkt @ IP/UDP (dest port 4789)

Arista EOS - %HARDWARE-3-DROP_COUNTER_ALERT, DchUnreachables

What does the following log message mean?

May 15 15:45:42 bn302 EventMgr: %HARDWARE-3-DROP_COUNTER_ALERT: Persistent Internal Drop 'DchUnreachables': 792271126642 detected on Fe3600-3/1

And there is high # of drop in the output of "sh hardware counter drop"

bn302.jhm.mlagB.profA0.w.15:47:58#show hardware counter drop
Summary:
Total Adverse (A) Drops: 20098028670058
Total Congestion (C) Drops: 0
Total Packet Processor (P) Drops: 72029
Type  Chip         CounterName                    :           Count : First Occurrence    : Last Occurrence
--------------------------------------------------------------------------------------------------------------
A     Fe3600-4/2   DchUnreachables-1              :    103285456899 : 2018-05-15 15:34:55 : 2018-05-15 15:35:07

Basically when the LCs are not ready and up, the Fabric modules don't know where to forward the packets, they will drop them and increase this counter. It is quite common during system booting up. 

Arista L3 Leaf/Spine Step by Step (2d) - VxLAN Bridging, interesting behavior

This post is for the paranoid, who really wants to know how it works under the hood. Based on previous setup, we see 2 interesting behaviors

1. Jericho MLAG peers also receive the arp reply

wa462.vtepB1#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.171267 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.171590 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

wa463.vtepB2#bash tcpdump -nvvi vlan2000 arp
tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:20.172821 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

20:49:20.173117 28:99:3a:3d:68:61 > 44:4c:a8:97:72:b7, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Reply 20.0.12.250 is-at 28:99:3a:3d:68:61, length 42

44:4c:a8:97:72:b7 is the MAC address of host2 belonging to vtep2. Why these 2 peers receive this unicast packet? The reason is that, when the VtepX1/X2 the first packet from remote Vtep, it will need some time to create a hw tunnel. During this gap, Vtep has to flood this ucast packet to all Vteps. 

When Vtep has no remote Vxlan MAC

ckp341.vtepX1#sh mac address-table dynamic interface vxlan 1
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
Total Mac Addresses for this criterion: 0

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

So Vtep has no remote vtep

ckp341.vtepX1#show vxlan vtep
Remote VTEPS for Vxlan1:
Total number of remote VTEPS:  0

Let's create a static MAC pointing to remote VTEP. After there is a remote Vtep. 

ckp341.vtepX1(config)#mac address-table static 0000.1111.2222 vlan 2000 interface vxlan 1 vtep 200.255.200.8

ckp341.vtepX1(config)#show vxlan vtep

Remote VTEPS for Vxlan1:

200.255.200.8

Total number of remote VTEPS:  1

Repeat the ping. Now the Mlag peer doesn't receive the unicast packet. 

wa462.vtepB1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

22:22:16.623983 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

2. ONLY 1 T2+ MLAG peer see receive the arp request, and no arp reply.

vtep1 has no arp req

snp262.vtepA1(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

^C

0 packets captured

vtepA2 sees arp req

snp263.vtepA2(config)#bash tcpdump -nvvi vlan2000 arp

tcpdump: listening on vlan2000, link-type EN10MB (Ethernet), capture size 262144 bytes

06:27:56.665383 44:4c:a8:97:72:b7 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.12.250 tell 20.0.8.250, length 42

^C

1 packet captured

This is because Trident family ASIC pipeline can't handle vxlan decap and routing in 1 pass. The arp req seen on vtepA2 is flooded by vtepA1. From the output of "show mac address mlag-peer", you can see vtepA2 learning this MAC from mlag-peer.

snp263.vtepA2(config)#sh mac address-table mlag-peer

          Mac Address Table

------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move

----    -----------       ----        -----      -----   ---------

2000    0000.1111.2222    STATIC      Vx1

2000    001c.73ff.405e    STATIC      Po2000

2000    444c.a897.72b7    DYNAMIC     Vx1        1       0:00:25 ago

4094    001c.73ff.405e    STATIC      Po2000

Total Mac Addresses for this criterion: 4

Arista L3 Leaf/Spine Step by Step (2b) - Recirculation

Topology:

Recirculation

In the previous blog, we see the host1 and host2 can NOT ping vtep3's SVI - VLAN2000 within the same VLAN. But host3 can reach this SVI. Why this happens?

This is because the Trident II ASIC doesn't support routing with overlay, recirculation channel is needed to loop the VXLAN inter-VLAN traffic back to the pipeline for routing lookup on some Arista switches like 7050QX. 

Bridging or Routing?

Based on the dstMAC, the ASIC determines the incoming packets to go to bridging or routing. The ping/ICMP packets from host2 to vtep3 will proceed vxlan decapsulation. Because its dstMAC is to routerMAC of vtep3, it is routing. So T2 can't handle vxlan decap and routing in 1 pass, thus ping failed

But if host2 pings host3, after vxlan decap, the packets are bridged. 

Similar host3 pings vtep3, the packets don't go thru vxlan decap, so ping is good. 

Which platform needs recirculation?

Only all Trident-2 or TH based platforms have this limitation. From above topology, other vteps like Jericho, T2+ doesn't need this. 

How to tell the chip model? The best way is to ask the account engineer who serves your account. Another way is to run the following CLI (based on my own experiences, if you know a better one please comment here. thanks!)

7280QR-C36-F(config)#sh platform fap

.....

Jericho0   !!! clearly this is a Jericho-based

7050QX-32-F#show platform fap

% Invalid input !!! FAP = Sand/Petra/Arad/Jericho, not supported

7050QX-32-F#show platform trident sys !! well this is a Trident 

         Slice              Chip       ModId    GenId

----------------- ----------------- ----------- -----

   FixedSystem       Linecard0/0           1        1  

------------------------------------------------------

Front panel vs internal ports

On the T2 system, the circulation can be done by front panel port and internal ports, depending on the switch model. A T2 chip can support 32 x 40G ports, some platforms like 7050TX-72/96, 7050SX/72/96, 7050S-64 don't use all ports at front panel, while the remaining ports are called internal ports. Using internal ports is definitely better than front panel, because it doesn't impact your switch connectivity capacity. 

So the next question is, how to tell if this switch has internal ports:-) Use CLI - "show inventory". 

7050SX-64-F.10:36:59(config)#show inventory

System has 81 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         64

  Unconnected      16  !!! has 16 unconnected ports

7050QX-32-F(config)#show inventory

System has 105 ports

  Type             Count

  ---------------- ----

  Management       1

  Switched         104 !!! No unconnected ports

Configuration:

Step1: Expose all internal ports (if the system has Unconnected ports under "show inventory", actually we don't need this for vtep3)
mLeafB.cd631.Z(config)#service interface unconnected expose
mLeafB.cd631.Z(config)#switch scheduler oversubscribed

Step2: Configure Recirc-channel (if T2 system)
upp224.vtep3(config)#int recirc-Channel 1
upp224.vtep3(config-if-Re1)#switchport recirculation features vxlan

Step3: Assign physical (front panel or internal) ports to recirc-channel

upp224.vtep3(config-if-Re1)#int et34
upp224.vtep3(config-if-Et34)#traffic-loopback source system device mac
upp224.vtep3(config-if-Et34)#channel-group recirculation 1
upp224.vtep3(config-if-Et34)#


Step4: Verify

upp224.vtep3#sh int recirc-Channel 1
Recirc-Channel1 is up, line protocol is up (connected)
  Hardware is Port-Channel, address is 2899.3a8b.e6fa
  Ethernet MTU 9214 bytes , BW 10000000 kbit
  Full-duplex, 10Gb/s
  Active members in this channel: 1
  ... Ethernet34 , Full-duplex, 10Gb/s
  Fallback mode is: off

Step5: ping from remote hosts

wa466.host2(vrf:host2)#ping 20.0.9.253
PING 20.0.9.253 (20.0.9.253) 72(100) bytes of data.
80 bytes from 20.0.9.253: icmp_seq=1 ttl=64 time=0.218 ms
80 bytes from 20.0.9.253: icmp_seq=2 ttl=64 time=0.150 ms
80 bytes from 20.0.9.253: icmp_seq=3 ttl=64 time=0.109 ms
80 bytes from 20.0.9.253: icmp_seq=4 ttl=64 time=0.107 ms
80 bytes from 20.0.9.253: icmp_seq=5 ttl=64 time=0.106 ms

--- 20.0.9.253 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.138/0.218/0.043 ms, ipg/ewma 0.195/0.175 ms

Arista L3 Leaf/Spine Step by Step (1) - Basic Config (BGP/MLAG)

(updated Dec 20, 2018 for more platforms)

Topology:


 

Following my previous post.  In this post, I cover the basic IP, BGP, and MLAG configuration. Here we use eBGP as routing protocol instead of OSPF or ISIS because it is the most common design in the modern data center. You can check the following 2 links for more details. 

• RFC 7938 - BGP Routing in Data Centers, by Petr Lapukhov
• BGP in an Arista Data Center

Basic eBGP configurations:

All BGP peers import ONLY a /32 network NLRI of loopback10, which is VXLAN source address. So no network p2p ethernet NRLI nor "redistribute connect". As a result, the routing table size is very minimal and only ping from local loopback to remote routers' works. 

router bgp 65041
   router-id 200.255.255.12
   maximum-paths 16 ecmp 16
   ....
   network 200.255.200.12/32   !!! loopback10's host addr.
!
interface Loopback10
   ip address 200.255.200.12/32

!!! direct ping doesn't work

ckp342.vtepX2#ping 200.255.200.56
PING 200.255.200.56 (200.255.200.56) 72(100) bytes of data.

--- 200.255.200.56 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

!!! ping source from loopback 10 works

ckp342.vtepX2#ping 200.255.200.56 source 200.255.200.12
PING 200.255.200.56 (200.255.200.56) from 200.255.200.12 : 72(100) bytes of data.
80 bytes from 200.255.200.56: icmp_seq=1 ttl=63 time=0.230 ms
80 bytes from 200.255.200.56: icmp_seq=2 ttl=63 time=0.140 ms
80 bytes from 200.255.200.56: icmp_seq=3 ttl=63 time=0.128 ms
80 bytes from 200.255.200.56: icmp_seq=4 ttl=63 time=0.109 ms
80 bytes from 200.255.200.56: icmp_seq=5 ttl=63 time=0.127 ms

--- 200.255.200.56 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.109/0.146/0.230/0.045 ms, ipg/ewma 0.208/0.186 ms

If multiple leaves share the AS# (not in this setup),  you need "neighbor spine.v4 allowas-in 3" to accept BGP NLRI between each other. 

router bgp 65010

   router-id 10.255.255.11

   neighbor spine.v4 allowas-in 3

eBGP and MLAG configuration

• The mlag configuration here is quite normal. 
• Only one thing worthy to note that, the 2 mlag peers use the same vxlan source address, shown as below

VtepA1: 

interface Loopback10

   ip address 200.255.200.34/32

!

router bgp 65021

   router-id 200.255.255.3

   network 200.255.200.34/32

!

interface Vxlan1

   vxlan source-interface Loopback10

LeafA2:

interface Loopback10

   ip address 200.255.200.34/32

!

router bgp 65021

   router-id 200.255.255.4

   network 200.255.200.34/32

!

interface Vxlan1

   vxlan source-interface Loopback10

Zip file of all routers' "show run"

Arista EOS Trouble-shooting ipv4/v6 packet loss

Steps:

• stop traffic and clear hw/sw drop counter
• clear hardware counter drop - show hardware counter drop
• clear counter - show cpu counter queue
• start traffic, and repeat trigger or test case, hold for some time then stop traffic
• check drop counter:
• show hard counter drop
• show cpu counter queue | nz
• repeat this step along the data path if needed

Output:

wa462.bug228215.01:04:40#sh cpu counters queue | nz
....
---------------------------------------------------------------------------------------------------------------
Jericho5.0:
CPU Queue                                         Pkts             Octets           DropPkts         DropOctets

---------------------------------------------------------------------------------------------------------------
CpuQueueLinkLocal                                    5                610                  0                  0
CpuQueueLldp                                        31               8060                  0                  0
CpuQueueBgp                                         60               5370                  0                  0
CpuQueueL3DstMiss                               469817          121682603           89773442        23251321478
CpuQueueL3LpmOverflow                             1693             438487             754666          195458494
Jericho5.1:
CPU Queue                                         Pkts             Octets           DropPkts         DropOctets

CpuQueueL3DstMiss means, 1) unresolved ipv4 arp; 2) unresolved ipv6 neighbor. 

Solution:

After configuring no platform sand ipv6 host-route exact-match", the traffic is recovered. 

Finally it turns out that I missed the vxlan config. 

Max # of MLAGs supported on Arista switch

One the biggest limitations of # MLAG support on an Arista switch, is the port-id. If you see the following the error message when configuring MLAG port-channels, it means you hit the ceiling. 
Mar 25 23:44:55 Arista.EOS Ebra: %SYS-1-PORT_ID_ERROR: Unable to allocate a port ID for interface

Well, this kind of failure is quite rare especially if your devices are only fixed 1RU or 2RU systems or the # of mlag port-channels is below 100. But if your system is a heavily loaded modular system like 7508 or 7512 with all LCs plugged, it may hit you surprisingly. 

What is the port-id, why important?

• The port-id is used in LACP and STP. In STP, the port-id field is 16-bit, after 4-bit for priority, only 12 bit left for port-id, which is 4096. 
• 2 MLAG peers are considered as 1 virtual switch, which means MLAG agent also needs to assign a port-id for corresponding peer-port and track their status. So now the space is cut half to 2048. 
• All physical ports need port-id, including ethernet and port-channel. 
• Considering a 7508N chassis with 8 x 7500R-36CQ-LC LC, how many port-id consumed?
• By default, 1 QSFP port reserve 4 port-ids, no matter the speed or up/down status - "transceiver qsfp default-mode 4x10G"
• 1 36CQ-LC = 4 x 36 = 144 port-ids
• 8 36CQ-LC = 144 x 8 = 1152
• After all, we still have 2048 - 1152 = 896 port-ids available for port-channels. 
• And still need some for L3 uplink, peer link and management interfaces, say 32. 
• Now the max # is 896 - 32 = 864

What happens if you have a 7512N with 12 x 36CQ LC?

• 12 36CQ-LC = 144 x 12 = 1728 port-id
• Max# of MLAG port-channel = 2048 - 1728 - 32 = 285
• Opps! Surprising, right? A chassis with more slots and more LCs supports much fewer MLAGs. 
• So the best system for scale MLAG is 7508 chassis with 8 x 36CQ LC. 

How to check port-id usage?

Arista.EOS#bash echo "ls -l /ar/Sysdb/interface/eth/portid"  | python -m Acons Sysdb | more

Connecting to local unix socket @00002 ...

Connected to process 2244

.....

default collection portId has:

    Ethernet10/1/1       !!! local ethernet ports

    Ethernet10/1/2

.....

    Management1/1        !!! local management interfaces

    Management1/2

    PeerEthernet10/1/1   !!! remote ethernet ports

    PeerEthernet10/1/2

....

    PeerPort-Channel1001 !!! remote port-channels

    PeerPort-Channel1002

....

    Port-Channel1001     !!! local port-channels

    Port-Channel1002

Arista.EOS#bash echo "ls -l /ar/Sysdb/interface/eth/portid"  | python -m Acons Sysdb | egrep "Eth|Man|Peer|Port" | wc -l

4049

A refresh post of 2 old blog posts, 1 and 2.