EOS: alias to sum up total num of received bgp prefixes

EOS-R1#show ip bgp neighbors 

BGP neighbor is 100.101.1.2, remote AS 100, external link

  Prefix Statistics:

                                   Sent      Rcvd     Best Paths     Best ECMP Paths

    IPv4 Unicast:                688000    818876         808929                   0

    IPv6 Unicast:                     0         0              0                   0

If we like to know the total number of rcvd prefix from all bgp peers, here is the alias command could be useful

alias totbgp show ip bgp neighbors | grep "IPv4 Unicast: \s\s" | awk  '{s+=$4}END{print s}'

EOS-R1#totbgp

6001256

EOS: sum up and compare in/egress throughput

Sometimes you want to compare the ingress/egress throughput on a particular router to see if any possible traffic loss (of course, the loss should be large enough like 3% more).  On EOS, srnz (alias srnz Show interface counters rates | nz) is a good alias. But if incoming or outgoing on multiple ports, you have to sum up and compare.

Here is a couple of useful tips and commands.

bn309#srnz
Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et9/1/1   ixia:LC8     0:05       0.0   0.0%        0   13435.8  35.0%     3543
Et9/2/1   ixia:LC8     0:05       0.0   0.0%        0   13435.3  35.0%     3543
...
Et11/6/1  ixia:LC7     0:05   13433.7  35.0%     3543       0.0   0.0%        0
Et11/11/1 ixia:LC7     0:05   13433.7  35.0%     3543       0.0   0.0%        0
Et11/12/1 ixia:LC7     0:05   13434.6  35.0%     3543       0.0   0.0%        0
Et11/13/1 ixia:LC7     0:05   13432.3  35.0%     3543       0.0   0.0%        0
Et11/14/1 ixia:LC7     0:05   13434.4  35.0%     3543       0.0   0.0%        0
Et11/15/1 ixia:LC7     0:05   13434.3  35.0%     3543       0.0   0.0%        0
Et11/16/1 ixia:LC7     0:05   13433.9  35.0%     3543       0.0   0.0%        0

In the above example, you want to compare ingress from ixia:LC7 and egress of ixia:LC8

bn309#srnz | grep LC7 | awk '{s+=$4}END{print s}'
161206  <<< ingress
bn309#srnz | grep LC8 | awk '{s+=$7}END{print s}'
161198  <<< egress

Arista EOS MTU Behavior (3) - L3 Routed Ethernet Ports, IPv4

Now we switched to L3 routed ethernet (IPv4) ports:

• MTU enforcement is for egress traffic
• Hardware programs the MTU value (default value is 1500, in this case, 2000) and checks against forwarding packet length (in this case, it is ipv4 packet). 
• For IPv4 packets, if the length is more than MTU, the packets will be fragmented by CPU. 
• And the number of fragmented and dropped packets are reflected by CoPPEgressTrap counter in the output of "show cpu counters queue summary | nz"

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system, 

• Use EOS tool - Ethxmit on the generator to send packets
• --ip-dst pointing to the destination address on the receiver device
• --size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port
• Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

• For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

• For using ethxmit:
• --size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
• If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC. 
• If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC
• No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output. 
• Trident-based system would use 9214. 

R2#sh int e47/1 | grep -i mtu
  Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
   mtu 100
   switchport access vlan 100

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

• 1. MTU enforcement happens on the egress interfaces. 
• So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces
• 2. Arista MTU is layer-3 MTU on Jericho chipset:
• 2.1 The MTU enforcement is only for routed packets, no action on switched traffic
• 2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface
• 3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

• Arista EOS Manual, Section 19.4 MTU
• Arista 4.23.1F TOI - Support for L3 MTU on 7280R3/7500R3/7800R3

EOS - Find the interfaces with the most flaps

bn302.12:40:24#show int phy | egrep '^Eth' | awk '{print $3, $1}' | sort -rn | head -10
479 Ethernet9/25/2
23 Ethernet9/11/1
5 Ethernet9/34/1
5 Ethernet3/34/1

EOS: A quick alias to show the interfaces with most util%

bn302#bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | more
Mbps Kpps Port
1.6% 4.5% Et9/36/1
1.5% 4.3% Et3/36/1
0.7% 0.5% Et13/33/1
0.7% 0.4% Et12/36/2

or set it as an alias

srnzsort bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | head -n 6

Updated note: The above command works with interface names/description defined. If not, have to change awk '{print $5, $8, $1}' to awk '{print $4, $7, $1}'

hs486.11:32:15#srnz
Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et49/1                 0:05    9870.4 100.0%      812    9860.3  99.9%      811


hs486.12:14:45#bash Cli -p15 -c 'srnz' | awk '{print $4, $7, $1}' | sort -r | more
In In Port
100.0% 99.9% Et50/1
100.0% 99.9% Et49/4

How about some interfaces with description, some don't? hm.... Let me think about it..... :-)

Ping bgp ipv6 link local address

With the feature of "BGP IPv6 link-local peering support", you can establish the bgp peering via ipv6 link-local address. To check the underlying reachability, we can use linux kernel command - ping6

For example,

interface Vlan4001
   vrf forwarding vrf3_BGP_v6P_v6Ll
   ipv6 address fe80::1/64 link-local
!
router bgp 65100
   vrf vrf3_BGP_v6P_v6Ll
      rd 65000:3
      bgp default ipv4-unicast
      bgp default ipv4-unicast transport ipv6
      bgp default ipv6-unicast
      neighbor fe80::100%Vl4001 remote-as 65500

bn303#bash sudo ip netns exec ns-vrf3_BGP_v6P_v6Ll ping6 -I vlan4001 fe80::100
PING fe80::100(fe80::100) from fe80::1%vlan4001 vlan4001: 56 data bytes
64 bytes from fe80::100%vlan4001: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from fe80::100%vlan4001: icmp_seq=2 ttl=64 time=0.238 ms
.....


Please note that this peering is under VRF, so you have to use the namespace command - "ip netns..."

Kill list of zombie process on Linux(Eos)

Due to a script bug, one EOS device has quite some zombie process like,

[admin@bn302 flash]$ ps aux  | grep 'CliShell -A'
root     28804  0.0  0.0  10504  8580 ?        S    10:53   0:00 CliShell -A -p 15 -c show port-channel | json
root     28809  0.0  0.0  10504  8688 ?        S    10:53   0:00 CliShell -A -p 15 -c show port-channel | json
root     28826  0.0  0.0  10504  8612 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
root     28835  0.0  0.0  10504  8580 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
root     28844  0.0  0.0  10504  8616 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
....

Instead of killing one by one manually, write a quick shell command to kill all. (killall is not supported in EOS/Linux)

[admin@bn302 flash]$ for pid in $(ps aux  | grep 'CliShell -A' | awk '{print $2, $11}' | grep CliShell | awk '{print $1}'); do sudo kill -9 $pid; done

Arista Linux Essential (2)

Useful Utilities

• Date/time
• date +%Y%m%d-%H%M%S
• Epoch - Linux born time
• Sort
• du -s /var/* | sort -n
• -n: numeric order
• Cut
• cut -d: -f:1,6 /etc/password
• -c: cut columns
• Diff:
• Cli -p15 -c "show run" | diff -y -w -B --suppress-common-lines - /mnt/flash/startup-config
• -B --ingore-blank-lines
• -y --side-by-side
• -w --ignore-all-space
• regexp:
• greedy vs lazy, ? at the end of pattern
• grep = global reg exp print
• -i: ignore case
• -v: inverse, not-matching
• -A: print # lines AFTER matching
• -B: print # lines BEFORE matching
• sed = powerful stream editor
• remove quotes: sed -e 's/"//g'
• reverse ip address:
• echo "10.20.30.40" | sed -e 's/\([0-9]+\)\.\([0-9]+\)\.\([0-9]+\)\.\([0-9]+\)/\4.\3.\2.\1/'
• reverse A-record to fwd record
• awk
• awk -F: '{print $1, $6}
• alias shmc show int | awk '/^[A-Z]/ { intf=$1 } 
• tar
• tar czvf config.1.gz config.1

[admin@bn303 etc]$ Cli -p15 -c "show run" | diff -y -w -B --suppress-common-lines - /mnt/flash/startup-config

! Command: show running-config        | ! Startup-config last modified at  Wed Nov 28 17:01:44 2018 b

! device: bn303 (DCS-7512N, EOS-4.20.1F)       | ! device: bn303 (DCS-7512N, EOS-4.20.10M)

Arista Linux Essentials (1)

From the course - "Arista Linux Essentials"

Linux Flavors

• Linux Distribution = Distro
• Generally includes:
• Kernel
• Package manager
• GNU tools and libraries
• Documentation
• GUI
• Debian
• All Ubuntu are Debian distro
• Gentoo
• For power users
• ChromeOS is based on Gentoo
• Android:
• Uses a Linux Kernel
• NO GNU tools and libraries, like glibc
• SUSE
• Fedora:
• Red Hat, CentOS
• Oracle's Unix OS
• Not Linux anymore because changing kernel 
• Arista EOS is Fedora Core
• No change in kernel
• ver 18 or 21 now

Bootup

• Boot Loaders:
• Same on all computers
• Multi-stage
• First: BIOS/POST
• Stored in ROM or NVRAM
• Initialize system hw
• MBR (Master Boot Record, Boot Loader) 
• Not OS-specific
• Examples: Coreboot, LILO, GRUB
• @arista switch, Aboot = a mini linux
• Point to VBR
• VBR (Volume Boot Record) .... OS Boot .... Kernel
• OS Boot
• dmesg display kernel message buffer

Arista Boot:

• Power On:
• BIOS
• Active partition
• Aboot -> init
• init -> boot-config
• ctrl - C to stop Aboot calls kexec
• EOS Kernel:
• Aboot calls kexec

EOS Boot Stages

• EOS Stage 1:
• /mnt/flash/persist
• boot hooks - patch, bug fix here
• not change kernel, change boot
• Hw Init:
• FRU initialize
• Cell type config - module/fixed, supervisor
• hw device tree
• EOS Stage 2:
• Kernel modules
• ProcMgr
• SysDB
• Launcher

Init/runLevel:

• scripts in /etc/init.d
• init <runlevel>

Package Management:

• wget - a web client to download files
• in aboot
• curl - more protocol support, http/ftp/imap/scp/....
• not in aboot
• Package Managers:
• different distros use different manager
• Debian/Ubuntu
• uses dpkg
• format is .deb
• advanced tool apt, manage dependencies
• apt-get, manage installation
• Fedors (EOS) 
• uses rpm
• format is .rpm
• yum manage dependencies
• as Fedora 18, yum is replaced by dnf
• sudo dnf install sysstat

sum dropped pkts in "show platform jericho counter drop"

bn302.12:51:12#show platform jericho counters drops | grep "Counts Match" | awk '{print $9}' | awk '{sum += $1} END {print sum}'
52508425

iptables in EOS

iptables is a Linux firewall utility program, which is leveraged by Arista EOS to control protocol control packets. For example:

Example: sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
-S : List the rules
-A : Append rule
-p : protocol
-j : jump = action

[admin@ck478 ~]$ sudo iptables -S
.....
-A INPUT -p tcp -m tcp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE   <<< MLAG control pkt
-A INPUT -p udp -m udp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE

! add a bgp neighbor 1.1.1.1
ck478.17:51:26(config)#router bgp 65500
ck478.17:51:43(config-router-bgp)#nei 1.1.1.1 remote 65001

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -j ACCEPT   <<< a new rule added for bgp nei 1.1.1.1

! configure bgp ttl security rule
ck478.17:51:51(config-router-bgp)#nei 1.1.1.1 ttl maximum-hops 2

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -m ttl --ttl-lt 253 -j DROP <<< all bgp pkts w/ ttl <253 droped!
-A BGP -s 1.1.1.1/32 -j ACCEPT

strace in EOS

"strace" is a powerful linux debug command and it can be used on Arista EOS. 

Some useful arguments:
-c -- count time and calls
-p pid -- trace process with pid#
-T -- print time spent

Example:
[admin@bn303 ~]$ ps -ef | grep Bgp
root     14926  2044  1 09:55 ?        00:00:09 Bgp

[admin@bn303 ~]$ sudo strace -c -p 14926
Process 14926 attached
^CProcess 14926 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0         1           ioctl
  -nan    0.000000           0         3         3 stat64
  -nan    0.000000           0        46           epoll_ctl
  -nan    0.000000           0        45           epoll_wait
  -nan    0.000000           0         4           send
  -nan    0.000000           0         1           sendmsg
  -nan    0.000000           0         1           recvmsg
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   101         3 total