TNO064: The Realities of Running SONiC at Scale

Host Scott interviews Brett Likens, who has nearly 20 years of networking experience spanning ISPs, healthcare, Rackspace, Network to Code, OpsMill, and now Amazon's retail infrastructure team. Brett's current role involves deploying SONiC-based disaggregated networking in an enterprise context — distinct from AWS's hyperscale operations — focused on edge/access switching across Amazon's large retail organization.

Brett explains that SONiC originated at Microsoft roughly a decade ago for hyperscale data center use and has since evolved into a viable enterprise option. Two primary business drivers are pushing enterprise adoption: cost control (reducing recurring vendor fees and per-port costs at scale) and vendor diversity (gaining negotiating leverage and controlling one's own destiny by decoupling the OS from proprietary hardware).

The hardware ecosystem now includes white-box vendors like Edgecore, Wistron, and Celestica offering enterprise-friendly form factors. Feature parity across hardware largely depends on the underlying ASIC — primarily Broadcom (Tomahawk, Trident, Jericho families) and Marvell — and whether the SONiC community or a third party has implemented the corresponding Switch Abstraction Interface (SAI) drivers. Brett notes that companies like Avis and PL Vision have emerged to fill this development gap, particularly for organizations lacking in-house software expertise or Broadcom SDK access.

From an operational standpoint, Brett describes SONiC's management interfaces — CLI, REST API, and gNMI — as flexible enough to integrate with existing automation stacks without requiring a complete overhaul. The OS's containerized architecture enables nonstop forwarding capabilities during upgrades (warm/soft reboot), reducing planned maintenance impact. CI/CD pipelines are used for image builds, with approximately 70-80% of testing achievable in software via virtual SONiC, reserving physical lab testing for ASIC-specific behaviors.

Brett's team at Amazon effectively acts as an internal TAC, with direct access to source code enabling faster root cause analysis compared to waiting on vendor support. He acknowledges that SONiC is being used in enterprise-adjacent features like spanning tree and PoE that haven't had the same battle-testing as data center features, meaning some rough edges remain. The containerized architecture also enables edge compute use cases — collecting, aggregating, and shipping custom observability data directly from the switch.

On the organizational change management side, Brett stresses that transitioning to SONiC is harder than simply adding a new vendor; it represents a paradigm shift requiring software development capability, vendor relationship maturity, team training, and process re-engineering. He recommends this path primarily for organizations with 10,000+ switches, though he expects the viability threshold to decrease over time. His practical advice: start by running virtual SONiC in a lab, and acquire a compatible used switch (e.g., an old Arista) to build a real business case before committing at scale.