FB: Eng Egress with Edge Fabric

https://research.fb.com/wp-content/uploads/2017/08/sigcomm17-final177-2billion.pdf?

PR's BGP connection types:

• Transit: private link with dedicated b/w
• Peers:
• private peer: dedicated PN ?~= transit ?
• public: via public fabric
• route server: prefix redirected by RS and traffic via public fabric

How prefixes are preferred:

• Prefer peer routes over transit (via local_pref), as_path tiebreaker
• In/egress traffic over same path
• If still tie, private peer > public > route server peer, using MED
• to avoid cross-congestion over fabric

BGP multi-path vs. ECMP

• Cisco BGP multipath doc
• Requirement of path characteristics to be multipath
• Weight
• local_pref
• as_path length
• origin
• MED
• one of these:
• neighbor AS or sub-as
• as_path

BGP limitation

• Not capacity-aware + ECMP
• unbalances links get equal load
• Static bgp policy likely optimizes traffic, but
• as-path != performance

Avoid congestion

• Input:
• Prefix via BMP, BGP only has 1 best
• controller does best selection
• sFLOW, IPFIX, traffic info
• SNMP, interface info
• Output, via BGP update by using higher local_pref

Performance-based routing

• servers set DSCP
• PBR on PRs, 1 DSCP - 1 route (table?)
• PR ISIS-SR/MPLS to ASW
• eBPF - extended berkley packet filter
• change pkt egress this server

to be continued.....

FB's back bone network - EBB

https://code.fb.com/networking-traffic/building-express-backbone-facebook-s-new-long-haul-network/

Why need EBB? Inter-DC traffic growth is much faster than internet egress traffic and WAN technology evolution.

What needed for the new EBB?

• Incremental deployment of software features. 不是一步到位，而且可以roll back. 
• NO RSVP-TE, because of inefficiency and complexity. 嗯，锤！
• Use MPLS SR so network state lean. 这倒是真的，SR分离CP和Local segment management.

How

• Use DC network ideas
• Small routing table - on the order of K routes
• 4 planes. 
• 3 Steps:
• IGP + full-mesh iBGP for packet forwarding
• TE + controller
• OPEN/R to replace IGP
• Components:
• 3 server-side:
• SFlow - read
• BGP route injector - write
• Controller - think
• 2 device-side:
• Open/R = IGP
• SR = forwarding
• 还有clean fault domain splitting, 这是咋做的？

Software Design:

• Hybrid approach: centralized controller + SR, best way
• path computation is at central level with global view
• network failure is handled at distributed level. 
• Open/R
• how to open/R on arista devices?
• Traffic estimator:
• SFlow, need src/dst pair
• what is the sample rate? 1 of 1000? how accurate
• LSP agent:
• With thrift-based API (eapi on eos) to program hw
• In future? need failure detection and fails over within hundreds of msec. 

Next step:

• per-service model
• scheduler for bulk transfer