Arista EOS Hashing Features

Here are some Arista EOS hashing features: 

• 4.15.2F, hashing on ttl and hop-limit
• Trident2-based systems: 7300X, 7050X, 7250QX. All have 2 hash engines, 1 for port-channel, 1 for ECMP
• CLI: 
• port-channel load-balance trident fields ip ttl 
• port-channel load-balance trident fields ipv6 hop-limit
• show port-channel load-balance trident fields
• IP TTL hashing is ON
• Unlike other "port-ch load-balance trident fields" commands, this one applies both to both LAG and ECMP hash engines. 
• Be care of unexpected behavior,  for example,  
• Internet traffic goes to proxy or load-balance servers.
• But somehow traffic going a different path results into different TTL values, like VRRP or MLAG. 
• So if hashing based on ttl, the same traffic stream could end up at different destinations. 
• Hashing egress interface: 
• 4.17.0
• show load-balance destination ingress-interface Ethernet1/1 src-ipv4-address 10.0.0.93 dst-ipv4-address 17.0.0.11 ip-protocol 17 src-l4-port 11001 dst-l4-port 19872
• Output Interface: Port-Channel102
• Disable Ingress interface in hashing
• Arad/Jericho based systems: 7500E/R, 7280E/R
• CLI: 
• no ip load-sharing sand fields ingress-interface
• To avoid polarization, use different preset in different layers:
• ip load-sharing sand has preset <0-2>
• 好像是，不同interface会有不同random#
• 这个for ECMP
• LAG hashing on ingress interface
• 和上面差不多，区别是config放在LB profile里面
• for LAG
• Global LAG hashing profile
• Arad/Jericho, Sand-based systems
• CLI:
• load-balance policies
• load-balance sand profile myProfile
• hash seed 50
• Dynamic and Symmetric LAG Hashing
• From 4-15-0
• Arad/Jericho, Sand-based systems
• Bidirectional flows go thru same member port of LAG
• 做法不难，就是异或源和目标地址，再求哈希，当然LB就是不好
• CLI: 
• load-balance policies
•    load-balance sand profile sym
•    fields symmetric-hash

Arista L3 Leaf/Spine Step by Step (3a) - Vxlan Routing, Modes

The following couple of posts will cover the VxLAN routing. 

What's and why VxLAN Routing? With VxLAN bridging, the hosts in the same VLAN 10 can talk to each other. But we need VxLAN routing to have VLAN 10 address to reach the hosts in VLAN 11. 

How to enable VxLAN Routing? Creating a SVI interface on a VLAN which is associated to a VNI

We can achieve this in the following 5 modes:

• Router on a stick - Gateway only on dedicated routers
• Direct Routing - Gateway on all VTEPs
• Indirect Routing - Gateway only on few dedicated VTEPs
• Naked Routing - Regular routing w/o VxLAN encap
• * Hybrid - Combination of direct/indirect and naked routing based on case

Unbalanced Hashing vs DLB vs RECMP

Broadcom introduced Dynamic Load Balancing in Trident 3 product. 

https://www.youtube.com/watch?v=1y25OTyFFrs

里面提到了Unblanced Hashing的3个Reasons:
1. Elephant Flow
2. Link Failures, Why?
3. Hash Polarization 

可能based在MIT一个flowlet paper

• 这个原理不难，多根线路，计算Maximum和Minimum Delay，之差就是一个timeout value;
• 如果后续packet timeout之后到，就不必hash到固定的线路，而是严格Load Balancing. 

一个相关的Arista EOS feature - Resilient ECMP

• 说起来很简单，就是一根线路断了，不要re-hashing区别的sessions，这样会teardown TCP. 而是just re-hash 故障线路上的session
• CLI: ip hardware fib ecmp resilience 10.14.2.0/24 capacity 6 redundancy 4
• 解释:
• network 10.14.2.0/24有6个NextHop
• 但是建立一个24 Entries的ECMP
• 如果其中一个NH断了，就把表中这个NH用其他NH替换，其他ECMP NH没有变化
• Redundancy = 4，应该就是一根断了，有4个来LB

TCP Notes (2)

1. 如何计算 a TCP segment length?
A: ip.len - ip.hdr_len - tcp.hdr_len

2. TCP Flags
* 最常见的是SYN, ACK, FIN
* URG - 比方说，sender希望receiver stop，无论queue里面还有多少data. 
* PSH - 和URG很像，必须high priority的处理
* RST - 本意是，如果port没有open，host就返回一个RST，容易被hacker利用
* SYN - 只有3-way handshake时有
* FIN - FINISH，4-way close? ACK/FIN-ACK-ACK/FIN-ACK

3. TCP window size
* len = 2 Bytes, max = 64K, in Byte
* 还有一个window scale，就是window size左移多少位
* SYN packet里面没有scaled
* 两边可以有不同的scale#, 不需要negotiated

这个是SYN Packet


后面的Data Packet


4. TCP checksum
看下面的图，前面加上IP Header Info, srcAdr, dstArd, reserved, proto#, tcpLen(computed), + tcpHdr, tcpData. 可以cover ip header integrity



5. TCP Timestamps
10-Byte TCP option, type(1B=8); len(=10), TS(len=4), TSEcho(len=4)，例如：

1st packet, TS = 2216538



2nd packet, TSEcho = 2216538. 这样就可以计算RTT了



6. TCP MSS vs IP MTU

这个MSS有点confusing，因为Segment一般是指Header + Data，但是这个TCP MSS说的是Max Data Leng. 这个图很清楚，
* Ethernet Frame size = 1518, 
* IP MTU = 1500 = 1518 - 18, 
* TCP MSS = 1460 = 1500 - 20 (ip.hdr_len) - 20 (tcp.hdr_len)



Reference:
1. firewall.cx
2. tcpipguide.com

Link-debounce timer

https://www.manualslib.com/manual/546511/Dell-Force10-S4810p.html?page=501

Delay the notification of interface up/down events. 主要是防止快速而重复的interface flaps，对上层协议的影响。时间是milisec. 

Configuration:

bn303.spine.host3(s1)(config-if-Et3/1/1)#link-debounce time 1000 1000
bn303.spine.host3(s1)(config-if-Et3/1/1)#int e3/1/1
bn303.spine.host3(s1)(config-if-Et3/1/1)#link-debounce time 1000 1000
bn303.spine.host3(s1)(config-if-Et3/1/1)#

TCP Notes (1)

How TCP Works @ youtube

• TCP fundamentals Part 2
• 2 related TCP windows: 
• CWND - Congestion Window
• RWND - Receive Window
• Only RWND advertized in packets
• minimum of CWND and RWND is the cap of transmission
• CWND = n * MSS, n = 1,2,4...., slowstart
• not advertised, changing
• Example 1, slow backup between 2 local servers
• from the trace, recvr to sender 只发ACKs, (len = 60) 
• window size 每隔一段时间，会从64K一直降到1000 (小于MSS). 然后hold for 190ms. 然后window size back to 64K
• 显示chart - statistics -> TCP stream graph -> Time sequence (tcptrace)
• 足够的证据是server side问题
• Example 2, 是sender application无法fill CWND,
• Example 3, ip tcp mss-adjust问题
• server/client, 2个pcap
• client直接看到seq = 1461, 不是1
• server一直re-send seq = 1, size = 1460 pkts. 
• TCP NoOP option - 很简单，#TcpLen * 4 = Tcp Header Len，而TCP Options Len不全是4的倍数，所以TCP Header需要NoOp pad out header size to a multiple of 4B. 
• TCP option - timestamp
• TSval, EchoTSval, 为了计算RTT and RTO (retrans timeout)
• Some TCP options:
• 0 - end of TCP options
• 1 - NoOP
• 2 - MSS
• 3 - Window Scale
• 4/5 - SACK
• 8 - timestamp

Wireshark Tips (2)

packetbomb.com - troubleshooting MTU issue

• Following the previous post, need column - tcpLen, Seq, nextSeq, Ack, BytesInFlight
• Statistics -> conversation, find the biggest stream 
• couple of TCP retransmission packets in jumbo frame with DF bit, because no ACK
• later sender starts with 512B -> 1024B, then conversation on. 
• clearly, 1 or multiple middle routers can't handle jumbo frames
• filter = icmp, doesn't show any icmp unreachable
• per RFC 1191 - path MTU discovery. The middle routers should return a ICMP destination Unreachable message with code = "frag needed and DF set"
• So these pkts are dropped by either firewall or disabled on the router

Wireshark Tips (1)

Quick refresh:

• Where to capture有讲究
• Capture on host machine不好，不是真正packets on the wire
• Wireshark setup: 
• 必须有time delta, 
• TCP: seq + tcpLen = nextSeq, ACK
• TCP: winSize，必须有，scaleNum in 3-way handshake
• TCP: byteOnTheFlight

Sources:

• A youtube video - Using Wireshark to Solve Real Problems for Real People (by Kary Rogers)
• packetbomb.com 
• packetbomb.com's "Fundamental Course"

Takeaways:

• Packet capture:
• Where matters
• Close to the source of complaint, capture all traffic w/o filter. If doing this in the network, have to figure out a smart way because of too much traffic. 
• Near the servers, if out of the domain, on the edge where the traffic leave
• Both if possible
• How to capture:
• On the client machine: NOT a good idea
• wireshark, tcpdump, windump
• easy but not the real traffic on the wire, why?
• application -- kernel's tcp/ip stack -- pcap (where is wireshark) -- NIC -- network. 
• for example, the packet before segmentation
• SPAN/Mirror
• TAP
• Hub??
• Wireshark setup (on my MAC wireshark 2.14)
• Wireshark -> Preference
• Layout
• Name Resolution: MAC, protoName, IP addr. 
• Protocol -> TCP
• Column:
• [] means data not in packet but calculated 
• 1. Add [time delta from previous displayed frame] to column
• right-click == apply as column
• 2. Add TCP's seq, Len, nextSeq, ACK to column. 
• Easy to see if losing a packet
• 3. Add TCP's windowSize
• Scale # is ONLY in 3-way handshake, x 256
• 4. Add "Bytes in flight"

EOS - CPU monitor session

monitor session 1 source Ethernet3/1
monitor session 1 destination Cpu

wa463.vtepB2(config)#show monitor sess

Session 1
------------------------
Source Ports:
  Both:        Et3/1
Destination Ports:
    Cpu :  active (mirror0) <<< use this interface to tcpdump


wa462.vtepB1#bash tcpdump -nvvi mirror0 udp
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:56:32.121346 44:4c:a8:a5:11:40 > 44:4c:a8:97:8c:51, ethertype IPv4 (0x0800), length 164: (tos 0x0, ttl 63, id 47642, offset 0, flags [DF], proto UDP (17), length 150)
    200.255.200.34.62729 > 200.255.200.56.4789: VXLAN, flags [I] (0x08), vni 2000
00:1c:73:ff:3c:30 > 44:4c:a8:97:72:b7, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 32537, offset 0, flags [none], proto ICMP (1), length 100)

Arista EOS Tips - alias

1. Multiple line alias
alias ccc
10 clear arp
20 clear mac add dyn
30 bash clear

2. Alias argument

alias da bash tcpdump -nvvvi %1 arp

Arista EOS: VARP - GARP

If you have the following VARP configuration:

ip virtual-router mac-address 00:00:33:33:44:44
ip virtual-router mac-address advertisement-interval 1
interface Vlan2000
   ip address 20.0.134.14/16
   ip virtual-router address 20.0.134.1

You will see the incoming GARP every 1 sec (collecting by "bash tcpdump -nvvi vlan2000 -w arp.cap")



Arista EOS

• Use opcode=2 (reply) as GARP 
• SrcMAC = vMAC, DstMAC = all FF
• Send/Target MAC/IP = vMAC/vIP
• You see 2 GARP in 1 sec because of 2 mlag peers 

FB: A Billion user load balancer

https://www.usenix.org/conference/lisa16/conference-program/presentation/shuff

• A 2016 presentation by PE - Patrick Shuff
• Tb of egress traffic leaving FB routers, 85% from outside US

FB Req Flow vs Architecture

• Client - DNS AAAA req for facebook.com
• Client - Router
• Router - ECMP to L4LB(ipvs)
• L4LB(ipvs) - Http GET to L7LB(proxygen)
• L7LB(proxygen) - Http GET to HHVM (websever)
• Router + L4LB + L7LB + HHVM = a cluster/DC
• Data flow:
• router/ECMP --> L4LB (ipvs) --> L7LB(proxygen) --> HHVM
• L4LB = 10+             
• L7LB = 100+
• HHVM = 1000+
• And L4LB/L7LB/HHVM are NOT dedicted servers or devices. All x86 servers dynamically allocated. 

L4/L7 LB:

• L4LB (ipvs) + xBGP, a python BGP dameon to talk to TOR to announce 
• ipv4:/32, 
• ipv6:/64
• Router to L4LB, just ECMP hash
• L4LB to L7LB, hash + state table
• 要考虑两种failures
• 1) L4LB down, 新的L4LB用同样的hash，到原来的HHVM;
• 2) L7LB down, TCP breaks, L4LB hash到不同的L7LB, 如果L7LB back up, 因为有state table，不会影响TCP
• DSR = direct server access
• return traffic bypass the L4LB, L7LB to router. 
• L4LB ==ipInip== L7LB
• 然后L7LB直接 original ipv4 packet back to client. 和微软很像

Speed up 

• 用PoP early terminates客户TCP/SSL handshake，这样加快链接建立的速度，而不是到DC
• Pop has direct SSL connections to DC. 
• 需要看TCP (3-way), SSL (4-way)

DNS

• Real time monitor

Arista DCS-7050QX-32S-F enable eth1-4

On the DCS-7050QX-32S-F, there are 4x10G SFP on the front panel. This 4 ports share the hardware with Eth5/1-4. So to switch the hardware of eth5/1 to eth1-4 of 10G, use this command

ck475.vtepX4(config)#hardware port-group 1 select ?
  Et1-4    Activate ports Et1-4
  Et5/1-4  Activate ports Et5/1-4

Arista EOS MLAG - domain name must be matched

snp262.vtepA1#sh run | sec mlag config
mlag configuration
   domain-id vtepA
   local-interface Vlan4094
   peer-address 200.255.134.2
   peer-link Port-Channel2000

snp263.vtepA2(config-mlag)#sh run sec mlag config
mlag configuration
   domain-id vtepA2  <<<<< a typo in mlag domain-name
   local-interface Vlan4094
   peer-address 200.255.134.1
   peer-link Port-Channel2000
   reload-delay mode lacp standby

snp263.vtepA2#sh mlag det | grep State
State                           :            inactive
Peer State                      :             unknown
State changes                   :                   3

snp263.vtepA2(config)#mlag config
snp263.vtepA2(config-mlag)#domain-id vtepA
snp263.vtepA2(config-mlag)#end

After correcting it, the mlag is up

snp263.vtepA2#sh mlag det | grep State
State                           :             primary
Peer State                      :           secondary
State changes                   :                   4

Arista EOS BGP wait-for-install and wait-for-convergence

"update wait-for-install"

Quite straightforward feature, update the prefixes until they are installed in hardware

• Obviously, this prevents packet loss. Without this feature, routers start to advertise prefixes when hardware forwarding not ready yet. When its peers start to flood traffic to it, the only thing it can do is to drop them. 
• Very useful when working with MLAG, which requires the routers to put Mlag downstream interface to hold state during initialization. 

"update wait-for-convergence"

• https://aristaeos.blogspot.com/2018/03/arista-eos-bgp-update-wait-for.html
• This is a feature for spine routers. The key point behind it is, don't advertise prefixes until fully convergence (which means you have full routing info). 

Where to use them:

• On MLAG or L2/L3 boundary,  ONLY use wait-for-install
• On spine, use both

FB: Eng Egress with Edge Fabric

https://research.fb.com/wp-content/uploads/2017/08/sigcomm17-final177-2billion.pdf?

PR's BGP connection types:

• Transit: private link with dedicated b/w
• Peers:
• private peer: dedicated PN ?~= transit ?
• public: via public fabric
• route server: prefix redirected by RS and traffic via public fabric

How prefixes are preferred:

• Prefer peer routes over transit (via local_pref), as_path tiebreaker
• In/egress traffic over same path
• If still tie, private peer > public > route server peer, using MED
• to avoid cross-congestion over fabric

BGP multi-path vs. ECMP

• Cisco BGP multipath doc
• Requirement of path characteristics to be multipath
• Weight
• local_pref
• as_path length
• origin
• MED
• one of these:
• neighbor AS or sub-as
• as_path

BGP limitation

• Not capacity-aware + ECMP
• unbalances links get equal load
• Static bgp policy likely optimizes traffic, but
• as-path != performance

Avoid congestion

• Input:
• Prefix via BMP, BGP only has 1 best
• controller does best selection
• sFLOW, IPFIX, traffic info
• SNMP, interface info
• Output, via BGP update by using higher local_pref

Performance-based routing

• servers set DSCP
• PBR on PRs, 1 DSCP - 1 route (table?)
• PR ISIS-SR/MPLS to ASW
• eBPF - extended berkley packet filter
• change pkt egress this server

to be continued.....

How to check # of BGP attributes

Check BGP export/shadown table

First enable "sflow extension bgp" then run command - "show bgp export-table path-attribute" or "show bgp shadown-table bgp-attribute" if older releases

bn302.17:01:15#show bgp export-table path-attribute
Bgp Attribute Info Table
key             nextHop  origin  pathFlags  originatorId  aspType  med  localPref  asPathAttrInfoId  commListId  extCommListId
 64        210.100.22.1       0         16             0        2    0          0                 3  4294967295     4294967295
 80  2000:210:255:254::       0          0    3539992321        1    0        100        4294967295  4294967295     4294967295
 88        210.100.11.1       0         24    3539992321        1    0        100                 2  4294967295     4294967295
 56        210.100.22.1       2         16             0        2    0          0                 2  4294967295     4294967295
 20  2000:210:100:32::1       0         48             0        2    0          0                 1  4294967295     4294967295

A new command:
bn302.17:01:08#bash smash -p ar/Smash/routing/bgp/export/pathAttrTableInfo/default/pathAttrEntry | egrep '^bucket' | wc -l
23

MAC 11:11:xx:.... is not valid MAC address

On IXIA if you configured MAC/Eth address to be 11:11:00:02:01:01, the switch doesn't take them.

Thanks Paul Lavelle for the why :-) 

The least significant bit of the first byte is known as the IG bit. An IG bit of 0 indicates that this is a unicast MAC address, an IG bit of 1 indicates a multicast or broadcast address. It is illegal to use a multicast source MAC address in ARP replies.

References:

https://en.wikipedia.org/wiki/Organizationally_unique_identifier#Format

https://blog.ipspace.net/2012/05/arp-reply-with-multicast-source-mac-is.html

I use a simple trick: If you are manually setting a MAC address on Ixia, make sure the first byte of the address is an even number :) 

Arista EOS - customized CLI

https://www.arista.com/assets/data/pdf/Whitepapers/Arista_EOS_parser.pdf

Arista EOS CLI is implemented in Python, so you can customize the CLI yourself. Of course it is limited to simple output instead of complicated contents.  

Step 1: Modify the CLI script

bn303.17:40:17#bash

Arista Networks EOS shell

[admin@bn303 ~]$ cd /usr/lib/python2.7/site-packages/CliPlugin/
[admin@bn303 CliPlugin]$ vi RoutingBgpShowCli.py
[admin@bn303 CliPlugin]$ sudo vi RoutingBgpCli.py

Step 2: save the new CLI script and load up during boot

copy the modified to /mnt/flash
the /usr/lib is file system in memory, unsustainable after reboot. 
vi /mnt/flash/rc.eos, so copy the save script to the location

Arista EOS, SSO support starts from 4.20.5F

On Sand platform, 7500R/E series. And 4.20.5F was released around Arp 2018. (bl)

We have 2 commands to check if system is sso-ready:

• wait-for-warmup checks for any agent to be not ready
• show redundancy stat is to ensure sso stage participating agents to be warm

And better to run these 2 commands on both sup:

bn302.10:49:50#wait-for-warmup

bn302.10:51:11#show redundancy states

  my state = ACTIVE

peer state = STANDBY HOT

      Unit = Primary

   Unit ID = 1

Redundancy Protocol (Operational) = Stateful Switchover

Redundancy Protocol (Configured) = Stateful Switchover

Communications = Up

switchover completion timeout = 120.0 seconds (default)

Not ready for switchover (Agents not ready in standby supervisor)

Agents not ready =

   Sand

  Last switchover time = 10:00:13 ago

Last switchover reason = Supervisor has control of the active supervisor lock

bn302.10:51:16#sess peer-supervisor wait-for-warmup

bn302.10:51:29#sess peer-supervisor show redu stat

  my state = STANDBY HOT

peer state = ACTIVE

      Unit = Secondary

   Unit ID = 2

Redundancy Protocol (Operational) = Stateful Switchover

Redundancy Protocol (Configured) = Stateful Switchover

Communications = Up

switchover completion timeout = 120.0 seconds (default)

Not ready for switchover (Agents not ready in standby supervisor)

Agents not ready =

   Sand

Arista Linux Essential (2)

Useful Utilities

• Date/time
• date +%Y%m%d-%H%M%S
• Epoch - Linux born time
• Sort
• du -s /var/* | sort -n
• -n: numeric order
• Cut
• cut -d: -f:1,6 /etc/password
• -c: cut columns
• Diff:
• Cli -p15 -c "show run" | diff -y -w -B --suppress-common-lines - /mnt/flash/startup-config
• -B --ingore-blank-lines
• -y --side-by-side
• -w --ignore-all-space
• regexp:
• greedy vs lazy, ? at the end of pattern
• grep = global reg exp print
• -i: ignore case
• -v: inverse, not-matching
• -A: print # lines AFTER matching
• -B: print # lines BEFORE matching
• sed = powerful stream editor
• remove quotes: sed -e 's/"//g'
• reverse ip address:
• echo "10.20.30.40" | sed -e 's/\([0-9]+\)\.\([0-9]+\)\.\([0-9]+\)\.\([0-9]+\)/\4.\3.\2.\1/'
• reverse A-record to fwd record
• awk
• awk -F: '{print $1, $6}
• alias shmc show int | awk '/^[A-Z]/ { intf=$1 } 
• tar
• tar czvf config.1.gz config.1

[admin@bn303 etc]$ Cli -p15 -c "show run" | diff -y -w -B --suppress-common-lines - /mnt/flash/startup-config

! Command: show running-config        | ! Startup-config last modified at  Wed Nov 28 17:01:44 2018 b

! device: bn303 (DCS-7512N, EOS-4.20.1F)       | ! device: bn303 (DCS-7512N, EOS-4.20.10M)

Arista Linux Essentials (1)

From the course - "Arista Linux Essentials"

Linux Flavors

• Linux Distribution = Distro
• Generally includes:
• Kernel
• Package manager
• GNU tools and libraries
• Documentation
• GUI
• Debian
• All Ubuntu are Debian distro
• Gentoo
• For power users
• ChromeOS is based on Gentoo
• Android:
• Uses a Linux Kernel
• NO GNU tools and libraries, like glibc
• SUSE
• Fedora:
• Red Hat, CentOS
• Oracle's Unix OS
• Not Linux anymore because changing kernel 
• Arista EOS is Fedora Core
• No change in kernel
• ver 18 or 21 now

Bootup

• Boot Loaders:
• Same on all computers
• Multi-stage
• First: BIOS/POST
• Stored in ROM or NVRAM
• Initialize system hw
• MBR (Master Boot Record, Boot Loader) 
• Not OS-specific
• Examples: Coreboot, LILO, GRUB
• @arista switch, Aboot = a mini linux
• Point to VBR
• VBR (Volume Boot Record) .... OS Boot .... Kernel
• OS Boot
• dmesg display kernel message buffer

Arista Boot:

• Power On:
• BIOS
• Active partition
• Aboot -> init
• init -> boot-config
• ctrl - C to stop Aboot calls kexec
• EOS Kernel:
• Aboot calls kexec

EOS Boot Stages

• EOS Stage 1:
• /mnt/flash/persist
• boot hooks - patch, bug fix here
• not change kernel, change boot
• Hw Init:
• FRU initialize
• Cell type config - module/fixed, supervisor
• hw device tree
• EOS Stage 2:
• Kernel modules
• ProcMgr
• SysDB
• Launcher

Init/runLevel:

• scripts in /etc/init.d
• init <runlevel>

Package Management:

• wget - a web client to download files
• in aboot
• curl - more protocol support, http/ftp/imap/scp/....
• not in aboot
• Package Managers:
• different distros use different manager
• Debian/Ubuntu
• uses dpkg
• format is .deb
• advanced tool apt, manage dependencies
• apt-get, manage installation
• Fedors (EOS) 
• uses rpm
• format is .rpm
• yum manage dependencies
• as Fedora 18, yum is replaced by dnf
• sudo dnf install sysstat

Python Notes (3) - Beautiful Python Code by Raymond Hettinger

https://www.youtube.com/watch?v=OSGv2VnC0go

Faster and prettier code:

1. Looping backwards
for color in reversed(colors):
    print color

2. Looping over collection and indices
for i, color in enumerate(colors):
    print i, ":", color

3. Zip of 2 lists
for name, color in zip(names, colors):
    print name, ":", color

! zip has higher memory req, prone to cache miss;
! in python 3.x, using izip instead of zip

4. Sorted list
for color in sorted(colors):
for color in sorted(colors, reverse=True):

def compare_length(c1, c2):
    if len(c1) &lt; len(c2): return -1
    if len(c1) &gt; len(c2): return 1
    return 0

for color in sorted(colors, cmp=compare_length):
for color in sorted(colors, key=len):

Looping over a dict with keys and values:
for k,v in d.items():     # req. memory to store list
for k,v in d.iteritems(): # use iterator instead of mem

Counting with dict
d ={}
for color in colors:
    d[color] = d.get(color, 0) + 1

d = defaultdict(int)
for color in colors:
    d[color] += 1

Grouping with dictionaries:
# group the list by length
names = ['Raymond', 'Rachel', 'Matthew', 'Roger', 'Betty']

#old
d = {}
for name in names:
    key = len(name)
    if key not in d:
        d[key] = []
    d[key].append(name)

#1
d = {}
for name in names:
    key = len(name)
    d.setdefault(key, []).append(name)

#2
d = defaultdict(list)
for name in names:
    key = len(name)
    d[key].append(name)

Function calls with keyword arguments
twitter_search('@obama', False, 20, True)

twitter_search('@obama', retweets=False, numtweets=20, 
               popular=True)

Packing/Unpacking = simultaneous state updates

x, y, dx, dy = ( x + dx *t,

                 y + dy *t,

                 influence(m,x,y),

                 influence(m,x,y))

Concatenating strings

', '.join(names)

Updating sequences

names = ['Raymond', 'Rachel', 'Matthew', 'Roger', 'Betty']

del names[0]

names.pop(0)

names.insert(0, 'mark')

#==&gt;

names = deque(['Raymond', 'Rachel', 'Matthew', 'Roger', 'Betty'])

del names[0]

names.popleft()

names.appendleft('mark')

Python Notes (2)

1. Sort: 

• sorted() vs list.sort()
• Sorted() returns a new list vs list.sort() does in place. 
• list.sort() only for list. But sorted() for all iterables. 
• key function and (reverse=True)
• Sorted by multiple elements, like by first item then 2nd. 

>>> p = [[1,2,3],[3,2,1],[2,4,6],[1,2,5],[1,3,7],[3,1,0],[1,7,2]]
>>> p.sort(key=lambda x:(x[0], x[2]))
>>> p
[[1, 7, 2], [1, 2, 3], [1, 2, 5], [1, 3, 7], [2, 4, 6], [3, 1, 0], [3, 2, 1]]

• Sorted by reversed plus asc

>>> p = [[1,2,3],[3,2,1],[2,4,6],[1,2,5],[1,3,7],[13,1,0],[-1,7,2]]
>>> p.sort(key=lambda x:(-x[0], x[2]))
>>> p
[[13, 1, 0], [3, 2, 1], [2, 4, 6], [1, 2, 3], [1, 2, 5], [1, 3, 7], [-1, 7, 2]]

2. SET:

1) No duplicate
2) No order
3) Sets contain only hashable items, for __contain__ method
4) Sets have operator like "-"

_set = set([1,2,3,3]) # (1,2,3)

s1 = set([1,2,3,4])

s2 = set([2,3,5])

l1 = [1,2,3,4]

l2 = [2,3,5]

s1 - s2

set([1, 4])

l1 - l2

Error

3. Variable length arguments in function call

positional argument with unknown number of arguments
def __func__(*args):
    for count, item in enumerate(args):
        print '{0}. {1}'.format(count, thing)

named arguments
def __func__(**kwargs):
    for name, value in kwargs.items():
        print '{0} = {1}' % (name, value)

TCL:
proc __proc__ {first args} {}

4. How to re-import python module in interactive mode
>>> from myrange import *
>>> for i in myrange(1,3,1):
...     print i
...
1 <<<<< extra print out due to debug print
1
2

$ more myrange.py
class myrange:
    def __init__(self, start, end, step):
        self.start = start
        self.end = end
        self.step = step
        print self.step   <<<< need to remove this

Change the py file, and do 2 steps

>>> import myrange   <<< need to import module for reload()

>>> reload(myrange)

<module 'myrange' from 'myrange.py'>

>>> from myrange import *

>>> for i in myrange(1,3,1):

...     print i

...

1  <<<< corrected

2

5. Python class class/instance/internal variables

啥叫class/instance/internal variables? 
- classVar 是Object之间share，一个改，大家都改，也叫Static variable
- instanceVar，所以叫Self.instanceVar，就是object自己本地
- internal，就不说了

class Obj(object):
    classVar = [10,20,30]
    def __init__(self):
        self.instanceVar = [1,2,3]
        internalVar = [100,200,300]

o1 = Obj()
o2 = Obj()

o1.instanceVar.append(11)
o1.instanceVar.append(12)
o2.instanceVar.append(21)
o2.instanceVar.append(22)

print o1.instanceVar
print o2.instanceVar

o1.classVar.append(111)
o2.classVar.append(222)
print o1.classVar
print o2.classVar