Arista EOS/Networking Tips by Solomon Yang: SR

Showing posts with label SR. Show all posts

11/05/2019

Arista EOS Segment Routing (3) - SR Routing #1, Static Route + NHG

The Arista MPLS Segment Routing White Paper(Page 8) lists 4 SR routing solutions:
1) Static label push via NHG
2) Controller via EOS SDK
3) BGP LU with SR
4) SR-TE

This blog gives an example of solution #1.

The configuration is straightforward:

Instead of running a routing protocol between PE1 and PE2, a static route of remote destination (100.255.4.1/32 on PE1) is configured and pointing to NHG.
The NHG defines:

MPLS encapsulation
Push a label of 900004
The NH is 10.1.2.2 for out-interface and destination MAC.
Obviously, 2 NHGs are needed to 2 way traffic.

Not like the regular NHG setup, decap group is NOT needed.
P1 and P2 have no idea of destination ip - 100.255.4.1, but use label to forward traffic

PE1#sh ip route 100.255.4.1

VRF: default

Codes: C - connected, S - static, K - kernel,

O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,

E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,

N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,

R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,

O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,

NG - Nexthop Group Static Route, V - VXLAN Control Service,

DH - DHCP client installed default route, M - Martian,

DP - Dynamic Policy Route, L - VRF Leaked

NG 100.255.4.1/32 [1/0] via sr-1-push-label-900004, Nexthop Group ID 1

The output of tcpdump on P2's et42/4 (the interface facing to P1) shows MPLS with the correct label in both directions.

23:10:17.713412 44:4c:a8:97:84:5f > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900001, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 6381, offset 0, flags [none], proto ICMP (1), length 100)

100.255.4.1 > 100.255.1.1: ICMP echo request, id 9258, seq 5, length 80

23:10:17.713473 44:4c:a8:97:8c:51 > 44:4c:a8:97:84:5f, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900004, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 2480, offset 0, flags [none], proto ICMP (1), length 100)

100.255.1.1 > 100.255.4.1: ICMP echo reply, id 9258, seq 5, length 80

Now you have the simplest, but completed ISIS segment-routing solution, control plane + data plane :-) In the real network, you can complete the puzzle by adding,

Have a controller participate in the ISIS domain via the ISIS over GRE tunnel.
As an ISIS neighbor, it can fetch the full ISIS LSDB to have a global view of the network including the segment.
Then it can program the edge router vi CLI/capi to steer the traffic. Or use the EOS SDK for faster program speed (this is the solution #2)

Arista EOS Segment Routing (2) - MPLS Ping to verify SR data plane

With the same topology, to verify the SR data plane, we can use the command - "ping mpls" to have PE1 send a UDP packet with the correct label.

PE1#ping mpls segment-routing ip 4.4.4.4/32 repeat 1
LSP ping to Segment-Routing route 4.4.4.4/32
timeout is 5000ms, interval is 1000ms
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
Reply from 10.3.4.4: seq=1, time=0.53ms, success: egress ok

--- Segment-Routing target fec 4.4.4.4/32 : lspping statistics ---
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
1 packets transmitted, 1 received, 0% packet loss, time 150ms
1 received from 10.3.4.4, rtt min/max/avg 0.530/0.530/0.530 ms

Turning on tcpdump on P1, it shows the following packets:

P1#sh run | grep moni
monitor session sr source Ethernet54/4
monitor session sr destination Cpu

P1#bash tcpdump -nvvvi mirror1 udp or mpls
tcpdump: listening on mirror1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:48:15.934459 44:4c:a8:97:77:21 > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 98: MPLS (label 900004, exp 0, [S], ttl 255)
(tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 80, options (RA))
10.1.2.1.36260 > 127.0.0.1.lsp-ping:
LSP-PINGv1, msg-type: MPLS Echo Request (1), length: 48
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: No return code or return code contained in the Error Code TLV (0)
Return Subcode: (0)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: no timestamp
Target FEC Stack TLV (1), length: 12
Unknown subTLV (31744), length: 6
0x0000: 0404 0404 2002
0x0000: 7c00 0006 0404 0404 2002 0000
13:48:15.934737 44:4c:a8:97:8c:51 > 44:4c:a8:97:77:21, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 36448, offset 0, flags [DF], proto UDP (17), length 60)
10.3.4.4.lsp-ping > 10.1.2.1.36260:
LSP-PINGv1, msg-type: MPLS Echo Reply (2), length: 32
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: Replying router is an egress for the FEC at stack depth 1 (3)
Return Subcode: (1)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: -16:-40:-1.401478080

From the above packet capture,

The icmp echo request is encap'ed in MPLS packet with label 90004 as expected.
The dest address of inner ip is 127.0.0.1. With destination address on network 127, the packet can't be routed if LSP is broken before the final destination and the packet is decap'ed prematurely.
The icmp echo reply is just a native ipv4 packet.

Arista EOS Segment Routing (1) - Simplest Setup/Configuration

Here is the simplest SR setup,
1) Only IPv4, only ipv4 node-segment;
2) No IPv6, no ISIS multi-topology

From the above configuration, we can observe:

The basic ISIS-SR configuration is quite simple:

enable "segment-routing mpls" under router isis to attach SR info to ISIS LSA;
Specify the loopback0 interface with a globally unique node-segment index.
ISIS-SR calculates the node-segment label = mpls label range base + index, so the routers in whole domain use the same label. This is an important characteristic of SR, for anycast.

To verify the SR setup, you can use the following CLI commands:

show mpls label ranges
show isis segment-routing
show isis database PE2.00-00 detail !PE2.00-00 is id
show mpls lfib route
show mpls segment-routing bindings
show platform jericho mpls route

Detailed output of PE1:

PE1#show mpls label ranges

Start End Size Usage

------------------------------------------------

0 15 16 reserved

16 99999 99984 static mpls

100000 116383 16384 ldp (dynamic)

116384 132767 16384 pseudowire (dynamic)

132768 149151 16384 bgp (dynamic)

149152 165535 16384 isis (dynamic)

165536 362143 196608 free (dynamic)

362144 899999 537856 unassigned

900000 965535 65536 isis-sr <<< default ISIS-SR range

900000 965535 65536 bgp-sr

965536 1031071 65536 srlb

1031072 1036287 5216 unassigned

1036288 1048575 12288 l2evpn

PE1#show isis segment-routing

System ID: PE1 Instance: sr

SR supported Data-plane: MPLS SR Router ID: 1.1.1.1

SR Global Block( SRGB ): Base: 900000 Size: 65536

Adj-SID allocation mode: SR-adjacencies

Adj-SID allocation pool: Base: 149152 Size: 16384

All Prefix Segments have : P:0 E:0 V:0 L:0

IS-IS Reachability Algorithm : SPF (0)

Number of IS-IS segment routing capable peers: 3

Self-Originated Segment Statistics:

Node-Segments : 1

Prefix-Segments : 0

Proxy-Node-Segments : 0

Adjacency Segments : 1

PE1#show isis database PE2.00-00 detail

IS-IS Instance: sr VRF: default

IS-IS Level 2 Link State Database

LSPID Seq Num Cksum Life IS Flags

PE2.00-00 12 57553 1049 L2 <>

NLPID: 0xCC(IPv4)

Hostname: PE2

Area address: 49.0001

Interface address: 4.4.4.4

Interface address: 10.3.4.4

IS Neighbor : P2.11 Metric: 10

LAN-Adj-sid: 100000 flags: [ L V ] weight: 0 system ID: 0000.0000.3333

Reachability : 4.4.4.4/32 Metric: 10 Type: 1 Up

SR Prefix-SID: 4 Flags: [ N ] Algorithm: 0

Reachability : 10.3.4.0/24 Metric: 10 Type: 1 Up

Router Capabilities: Router Id: 4.4.4.4 Flags: [ ]

SR Local Block:

SRLB Base: 965536 Range: 65536

Area leader priority: 250 algorithm: 0

SR Capability: Flags: [ I ]

SRGB Base: 900000 Range: 65536

PE1#show mpls lfib route

MPLS forwarding table (Label [metric] Vias) - 4 routes

MPLS next-hop resolution allow default route: False

Via Type Codes:

M - MPLS via, P - Pseudowire via,

I - IP lookup via, V - VLAN via,

VA - EVPN VLAN aware via, ES - EVPN ethernet segment via,

VF - EVPN VLAN flood via, AF - EVPN VLAN aware flood via,

NG - Nexthop group via

Source Codes:

G - gRIBI, S - Static MPLS route,

B2 - BGP L2 EVPN, B3 - BGP L3 VPN,

R - RSVP, LP - LDP pseudowire,

L - LDP, M - MLDP,

IP - IS-IS SR prefix segment, IA - IS-IS SR adjacency segment,

IL - IS-IS SR segment to LDP, LI - LDP to IS-IS SR segment,

BL - BGP LU, ST - SR TE policy,

DE - Debug LFIB

IA 149152 [1]

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900002 [1], 2.2.2.2/32

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900003 [1], 3.3.3.3/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900004 [1], 4.4.4.4/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

PE1#show mpls segment-routing bindings

1.1.1.1/32

Local binding: Label: imp-null

Remote binding: Peer ID: 0000.0000.2222, Label: 900001

2.2.2.2/32

Local binding: Label: 900002

Remote binding: Peer ID: 0000.0000.2222, Label: imp-null

3.3.3.3/32

Local binding: Label: 900003

Remote binding: Peer ID: 0000.0000.2222, Label: 900003

4.4.4.4/32

Local binding: Label: 900004

Remote binding: Peer ID: 0000.0000.2222, Label: 900004

PE1#show platform jericho mpls route

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------

| Mpls Table |

|---------------------------------------------------------------------------------------------|

|---------------------------------------------------------------------------------------------|

|149152 |Et2/4 |1027 | 44:4c:a8:97:8c:51 | PopE pipe auto|32773 |8200 |8 | 0 |

Egress Action Codes:

M - Mpls Tunnel, G - GRE Tunnel, MoG - Mpls-over-GRE Tunnel

ECMP Codes:

D - ECMP is divergent across switching chips

-----------------------------------------------------------------------------------------------

| Mpls Table

|-----------------------------------------------------------------------------------------------

|-----------------------------------------------------------------------------------------------

|900002 |Pop | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900003 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900004 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

2/13/2019

Arista BGP-LU/SR

In this blog, talking a bit about BGP-LU/SR and its use cases. Here are some References:

Arista BGP-LU

By the RFC 3107, the equivalent Cisco feature is called labeled BGP. The concept is quite simple to advertise an IPv4 prefix with a label.
Use case 1) - replace RSVP-TE, cloud provider

Central controllers push routes with multiple label via BGP-LU
Replace RSVP-TE

Use case 2) - CSC

ce1--pe1---asbr1--[--cPE1-----cPE2--]--asbr2---pe2--ce2
between asbr and cPE, eBGP-LU, so cPEs know the label to reach PEs
cPEs switch packets based on labels of PEs' loopback

Configuration:

router bgp 100
bgp labeled-unitcast rib tunnel|ip
neighbor 77.255.11.11 remote-as 100
neighbor 77.255.11.11 update-source lo77
address-family ipv4 labeled-unicast
neighbor 77.255.11.11 act
show platform jericho mpls route

Tunnel RIB vs IP RIB

Mutli-agent vs GateD

Arista MPLS Segment Routing WP

3 segment routing approaches
1) Static mpls push + NHG

prefix based - 100.1.1.0/24 -> NHG1 3.3.3.3 {2, 3, 5, 8}
class based - 100.1.1.0/24 EF -> NHG1 3.3.3.3 {1,2,3}

2) EOS SDK by controller

Listen: controller leans ISIS topology SIDs (node, adj, prefix) via ISISoGRE or BGP-LS (not supported by Arista EOS yet as of Feb 2019)
Write: program SR multi-label stack tunnel using NHG

3) BGP-LU

Listen: same as 2)
Write: use BGP-LU

11/16/2018

SR Policy Architecture

https://datatracker.ietf.org/doc/html/draft-filsfils-spring-segment-routing-policy-06.txt

SR Policy包括什么？

ID of SR Policy = <headend, color, endpoint>
1个Policy可以有多个Candidate Path，一个CP可以有多个SID-lists, 可以有weight for LB
Protocol origin of CP

10: PCEP;
20: BGP SR;
30: Local, CLI/Yang...

Originator of CP, 160b = 20B

= 4B AS# + 16B Addr (128b)
如果Addr是v4，就放在最后4B

Discriminator of CP, default 0, 这个是啥？一个就是seq#，作为tie-breaker
ID of CP = <protoOrig, originator, discrimator>
pref of CP = 100
Valid of CP, 就是SID是valid
Active CP

higher origin id
lower originator ip
higher discriminator

SR policy还可以有priority，就是有topology change, 先算那个

SR Policy 例子：

1个policy, <headend, color, endpoint>
2个CP，CP1 is active 因为preference 200
CP1有两个SID-List, 都installed in HW, 而且ECMP

验证 CP

An explicit CP with SID-list = 应该是指静态CP

为空
Weight = 0
1st SID不能resolve
non-1st SID of type 3~11 into MPLS Label or SRv6 SID? 什么意思
挺多的，还有最后一个不是prefix SID

Dynamic CP

Binding SID

非常关键的一个概念
= CP，SR的无缝衔接？
可以代表任何的interface, tunnel.

Steering:

这个是很关键的概念
Headend可以steer traffic，以下方式：

local BSID
Per-dest, 需要BGP
Per-flow?
PBR

如果SR Policy失效，就fall back to 普通的routing
SR Policy可以是Drop

BSID Steering

<B, L2, L3> 变成 <S1, S2, S3, L2, L3> 如果S1不是PHP

Per-Dest Steering:

这个一个是最常见的
BGP routes <prefix, N, ext-color-C, VPN-Label-V>
Valid SR Policy - <endPt = N, color = C> of SID-list <S1, S2, S3> and BSID B
如果都met，NH != N, 而是=SR policy P of BSID B.
收到prefix pkt, push <S1, S2, S3, V> label
一个BGP update里面可以有多个color, 如果有对应的policy with colors, 最终只有一个FIB，因为higher color prefered

10/15/2018

FB's back bone network - EBB

https://code.fb.com/networking-traffic/building-express-backbone-facebook-s-new-long-haul-network/

Why need EBB? Inter-DC traffic growth is much faster than internet egress traffic and WAN technology evolution.

What needed for the new EBB?

Incremental deployment of software features. 不是一步到位，而且可以roll back.
NO RSVP-TE, because of inefficiency and complexity. 嗯，锤！
Use MPLS SR so network state lean. 这倒是真的，SR分离CP和Local segment management.

How

Use DC network ideas

Small routing table - on the order of K routes
4 planes.

3 Steps:

IGP + full-mesh iBGP for packet forwarding
TE + controller
OPEN/R to replace IGP

Components:

3 server-side:

SFlow - read
BGP route injector - write
Controller - think

2 device-side:

Open/R = IGP
SR = forwarding

还有clean fault domain splitting, 这是咋做的？

Software Design:

Hybrid approach: centralized controller + SR, best way

path computation is at central level with global view
network failure is handled at distributed level.

Open/R

how to open/R on arista devices?

Traffic estimator:

SFlow, need src/dst pair
what is the sample rate? 1 of 1000? how accurate

LSP agent:

With thrift-based API (eapi on eos) to program hw
In future? need failure detection and fails over within hundreds of msec.

Next step:

per-service model
scheduler for bulk transfer

9/11/2018

ISIS Segment Routing

CCIE SR discussion

说起来不难，就是用IGP(ISIS)来signal labels，而不是LDP或者RSVP，更简单
但是还是用MPLS，所以软件升级就可以了
有Node Segment和Adj Segment，Ingress Router构建Label Stack就可以控制LSP

Arista MPLS SR

网络的难点是：

如何Classify traffic 和 Engineer path.
SDN如何control traffic path/flow，特别是native solution for ipv6
MPLS解决了一些，但是太复杂，例如TE，而且还没有ipv6 native support.
Segment是Arista认为的solution

SR operation:

网络分成Segment, 给一个SID - segment id. 利用BGP/ISIS/OSPF extension来distribute. 而不在需要Label protocols - LDP/RSVP
SID有Global Unique或者Local significant, Base + Index
3种 global SID:

1) prefix;
2) node; loopback of node
3) anycast; loopback shared by a set of routers, ecmp

Prefix SID: 全网都是same value, 非常关键，reducing DP state;

图上的例子是, 所有的router SRGB(SR Global Block)都是900,000-965,535, Rtr5's 5.5.5.5/32, prefix-SID是10，所有的router全部assign label 900,000+10 = 900,010

Adj Sid是locally significant, 只给neighbor，只installed at neighbor
最简单应用，用到ECMP，ingress LER就push一个SR label. 或者是push a set of label

SR vs LDP:

相同的地方：

easy configured, "plug and play"
Both form stateless Mp-to-pt LSP

不同：

LDP全是local signficant label, SR是global unique labels, 减少DP state; scale 好；
SR有TE，LDP TE没有流行，v6也没有

SR vs RSVP-TE

RSVP-TE的特点：

constraints routing like b/w, shared link risk group and explicit paths，可以不按照IGP shortest path;
有b/w
FRR，有pre-computed backup paths

不好的地方：

full-mesh p2p TE tunnels, 没有ECMP
failure后有churn，需要re-signaling.
scalability issue，所以不那么流行

SR利用SDN，在head加入stack of mpls label，而不需要中间router纪录state。还有scale
什么都好，没有Multicast?

Arista SR:

必须是R-series with FlexRoute?

三种SR Solutions:

Static MPLS push + NHG

在ingress LER上，configure a route pointing to a label stack via CLI;
ECMP = multiple tunnels
Class-based service policy
Easy start

Controller based using Eos SDK
BGP-LU (labeled unicast/RFC 3107)

应用：Cloud-WAN, CDN, NFV

Arista ISIS SR (from 4-17-0f)

9/10/2018

Tutorial: Segment Routing

https://www.youtube.com/watch?v=BEo5MdB3o3Y

Take-away points:

SR中最重要的是TE
Segment (1-hop or n-hop) vs SID
SR是hybrid TE approaches: path info in packet and network
CSPF - attribute(b/w, color), requirement and shortest path
CP = controller

What's SR?

A tunneling tech - nothing fancy
A TE! - steer packet instead of routing path. ~= RSVP TE

Terminology:

Domain
SR path
Segment = SID, can be multiple hops

Traditional TE Approaches

path info in packet

like IPv4 strict routing option

path info in the network

RSVP-signaled MPLS

SR is hybrid TE approach

Segment types:

1 router hop
multiple router hops
Types:

Adjacency (IGP adjacency, 1-hop)
Prefix = IGP least cost path to a prefix

SR Encapsulation

MPLS

Label = segment

IPv6:

Segment Routing Extension Header (SRH)
A list of ipv6 addr
each ipv6 addr = a segment

Local Labels

Some SIDs have node-local significance
Adjancency
Why important

Stack can be too big, ASIC cannot handle
MTU

Global Labels

Some have domain-wide significance
Each node reserves a block of labels.
SRGB base

IPv6 forwarding

SRH, segment routing hdr
A list of ipv6 address as SID

Binding SIDs

Pointing to another SR paths or tunnels
Reasons:

Label stack
MTU size too big

Control Plane

Path computation: on SR ingress or central controller.

CSPF, 这个讲的清楚！

Attributes to segment: color, b/w, SRLG
Req to each path
Shortest path meeting req.
LSDB, TED = extension to carry info.
Alternative path to protect
All info in LSDB, no need for RSVP or LDP

SR convergence after failures

Fast recovery by IGP reconvergence.
TI-FLA, speed up convergence if not fast enuf
Use anycast SID (ecmp)

SR benefits:

If using SR to reserve b/w, MUST go for controller
Central controller has global view

Controller Protocol options:

pull LSDB:

controller: a passive mbr in IGP
BGP-LS

push segment list to ingress SR

PCEP
BGP

push policy

binding what traffic to which path
PCEP or BGP

Conclusion:

SR moves state from network to packet - simplified
Some open issues: OAM, Fast Reroute
Need experience