TNO067: The Impact of AI on Internet Traffic
In this episode of Total Network Operations, Russ White (Nokia) and Lenny Giuliano (HPE) discuss how generative AI may fundamentally change internet traffic patterns, moving from today's cacheable CDN model to distributed inference computing at the edge. They explore the tension between centralized cloud compute and edge deployment, considering factors like latency, video processing, privacy, power constraints, and the need for deterministic networking similar to training infrastructure.
Summary
The episode begins with introductions of the guests and their backgrounds across major networking companies. Russ explains that CDNs currently serve 80-90% of internet traffic by pre-positioning cacheable content at the edge, with eyeball time (page load latency) being a critical metric for engagement. The discussion then shifts to how generative AI breaks this CDN model because AI inference results are not cacheable in the same way—each user query generates unique output.
A key concept introduced is 'time to first token,' the latency before an AI system begins generating its response. While human-facing applications tolerate higher latency (users can read output slower than generation), agent-to-agent AI interactions require low latency because machines expect responses immediately. This distinction drives the need for inference models to be positioned closer to end users.
The hosts discuss two stages of AI inference: preload (memory-bound) and decode (compute-bound), suggesting these might eventually be disaggregated and run on different hardware or locations for efficiency. They explore whether edge compute, long marketed but rarely justified, finally has a genuine use case in distributed AI inference, particularly for video processing, AR/VR applications, and real-time decision-making (e.g., autonomous vehicles, security cameras).
A crucial economic tension emerges: content providers must trade off transportation costs (sending traffic across long-haul networks) against storage costs (replicating compute and models at the edge). Privacy and data sovereignty concerns are also discussed, particularly for enterprises unwilling to send sensitive workloads to cloud providers, potentially creating a market for local inference through ISPs or private data centers.
The hosts note that infrastructure will likely follow demand rather than precede it—similar to how roads are built after people establish travel patterns. They discuss potential impacts on network operations: more complexity, new performance requirements, and the possible return of deterministic networking concepts like traffic engineering, admission control, and bandwidth reservation that were previously abandoned in favor of best-effort IP networking.
They reference historical parallels, particularly ATM's over-engineered feature set that IP ultimately displaced, and 5G's unfulfilled promises of revolutionary applications. This leads to speculation that like before, networks will adapt through simple, general-purpose mechanisms rather than specialized complexity. Finally, they discuss how outages during training runs might have severe consequences, joke about robots and driverless cars as marketing hype indicators, and invite operations professionals to discuss real AI data center challenges.
About this episode
Scott Robohn sits down with guests Lenny Giuliano and Russ White to examine whether generative AI will affect internet traffic patterns. Together they explore the role of CDNs, the importance of latency, and the potential for a shift toward edge-based inference. They also discuss the impact of data sovereignty on infrastructure and the potential return<a class="excerpt-read-more" href="https://packetpushers.net/podcasts/total-network-operations/tno067-the-impact-of-ai-on-internet-traffic/" title="ReadTNO067: The Impact of AI on Internet Traffic">... Read more »</a>
Key Insights
- CDNs currently handle 80-90% of internet traffic by pre-positioning cacheable content, but AI inference results are fundamentally non-cacheable because each query generates unique outputs requiring different network approaches
- Time to first token latency matters critically for agent-to-agent AI communication but less for human-facing applications, since humans read output slower than machines expect responses
- Video already comprises 60-98% of internet traffic and will likely remain the dominant use case for edge-based AI inference, particularly for real-time processing like autonomous vehicles and security monitoring
- AI model training has created strict coupling between compute, network, and scheduling requirements that may force adoption of deterministic networking technologies like admission control and traffic engineering that IP networks largely abandoned
- Power and space constraints in centralized data centers are creating necessity, not choice, to distribute compute across multiple smaller locations, potentially leading to federated training and inference fabrics across WANs
- The economic trade-off between transporting traffic across networks versus storing compute at the edge will vary by use case, but personalized content generation (custom video, podcasts) favors distributed models
- Enterprises concerned about data sovereignty may prefer local inference through ISPs or private data centers rather than cloud providers, creating a potential new business model for access providers to compete with hyperscalers
- Disaggregated inference separating memory-intensive preload stages from compute-intensive decode stages could improve hardware utilization, but optimal distribution (same rack, across city, centralized vs. edge) remains undetermined
- Historically, over-engineered network technologies like ATM lost to simpler best-effort approaches like IP, suggesting AI requirements may evolve to reduce dependencies on perfect network behavior rather than requiring permanent infrastructure changes
- Network operator impacts will likely appear as increased complexity, new performance requirements, and return of technologies like anycast, multicast, and traffic steering for directing inference workloads to optimal edge locations based on GPU availability
- CDN providers with existing edge infrastructure are natural candidates to add GPU-accelerated inference capability, but this intensifies competition with hyperscalers who already operate distributed data centers and understand both compute and delivery
- Industry pattern shows product development typically prioritizes functionality first, then management, with security coming late (step 79), suggesting current AI deployments may lack robust security frameworks despite rapid production adoption
Topics
Transcript
. Welcome to Total Network Operations. You know the gig. We're here to talk about great ideas and network operations, and we have a really fun episode today. We're here at Kerasoft in Reston, Virginia recording right before the Virginia NUG, the Virginia Networking User Group meeting in May 2026. And couldn't have two more fun guests to have for this episode today, Lenny Giuliano and Russ White. And let me have them do the honors and introduce themselves. Lenny, why don't you go ahead? Thanks, Scott. Great to be here. I work at HPE now, formerly Juniper, it's HPE networking. And I am on their architecture team, senior distinguished technologist on the service provider architecture team. Great. Welcome, Russ.…
Full transcript available for MurmurCast members
Sign Up to AccessMore from The Everything Feed - All Packet Pushers Pods
HN836: Optimize AI Traffic on the WAN with Ciena’s Integrated IP Networking (Sponsored)
This sponsored podcast episode explores how AI inferencing is changing WAN infrastructure demands, with Ciena executives discussing the need for latency optimization, dynamic traffic engineering, and integrated IP-optical networks to support distributed AI workloads across multiple cloud providers.
LIU019: Jason Gooley: The Godfather of Programmability
Jason Gooley, known as the Godfather of Programmability, discusses his journey from poverty in Chicago to becoming a Cisco engineer and author, explaining how financial necessity and early passion for technology drove him to obtain certifications like the CCNA and CCIE, and how he's now focused on building community and making certification paths more accessible through initiatives like the CCST exam.
HW083: Inside the WLAN Pros Toolbox – A Free, Multipurpose App
Keith Parsons introduces the WLAN Pros Toolbox, a free cross-platform app containing over 100 Wi-Fi tools, calculators, and references available on iPhone, iPad, Mac, Android, and web browsers. Built using Flutter and AI-assisted development, the app is intentionally free with no ads, subscriptions, or data collection because Parsons believes essential professional tools should be accessible to all engineers worldwide.
NB582: Infoblox Adds Network Observability with Kentik Buy; Satellite Data Centers vs. the Environment
Network Break covers major tech news including Infoblox's acquisition of Kentik for network observability, alarming electricity consumption by data centers (especially in Ireland), security advances in AI agent detection, and developments in space infrastructure including Rocket Lab's acquisition of Iridium and environmental concerns about orbital data centers.
TCG079: Why Your State File is Actually a Distributed Systems Problem
Malcolm Matalka argues that Terraform's value lies not in its HCL syntax but in its state management, which is fundamentally a distributed systems problem inadequately solved by file-based locking. He discusses how StateGraph reimagines infrastructure state as a database rather than a JSON file, enabling concurrent operations, better queryability, and solving the scalability issues that plague teams as they grow.