TNO067: The Impact of AI on Internet Traffic
In this episode of Total Network Operations, Russ White (Nokia) and Lenny Giuliano (HPE) discuss how generative AI may fundamentally change internet traffic patterns, moving from today's cacheable CDN model to distributed inference computing at the edge. They explore the tension between centralized cloud compute and edge deployment, considering factors like latency, video processing, privacy, power constraints, and the need for deterministic networking similar to training infrastructure.
Summary
The episode begins with introductions of the guests and their backgrounds across major networking companies. Russ explains that CDNs currently serve 80-90% of internet traffic by pre-positioning cacheable content at the edge, with eyeball time (page load latency) being a critical metric for engagement. The discussion then shifts to how generative AI breaks this CDN model because AI inference results are not cacheable in the same way—each user query generates unique output.
A key concept introduced is 'time to first token,' the latency before an AI system begins generating its response. While human-facing applications tolerate higher latency (users can read output slower than generation), agent-to-agent AI interactions require low latency because machines expect responses immediately. This distinction drives the need for inference models to be positioned closer to end users.
The hosts discuss two stages of AI inference: preload (memory-bound) and decode (compute-bound), suggesting these might eventually be disaggregated and run on different hardware or locations for efficiency. They explore whether edge compute, long marketed but rarely justified, finally has a genuine use case in distributed AI inference, particularly for video processing, AR/VR applications, and real-time decision-making (e.g., autonomous vehicles, security cameras).
A crucial economic tension emerges: content providers must trade off transportation costs (sending traffic across long-haul networks) against storage costs (replicating compute and models at the edge). Privacy and data sovereignty concerns are also discussed, particularly for enterprises unwilling to send sensitive workloads to cloud providers, potentially creating a market for local inference through ISPs or private data centers.
The hosts note that infrastructure will likely follow demand rather than precede it—similar to how roads are built after people establish travel patterns. They discuss potential impacts on network operations: more complexity, new performance requirements, and the possible return of deterministic networking concepts like traffic engineering, admission control, and bandwidth reservation that were previously abandoned in favor of best-effort IP networking.
They reference historical parallels, particularly ATM's over-engineered feature set that IP ultimately displaced, and 5G's unfulfilled promises of revolutionary applications. This leads to speculation that like before, networks will adapt through simple, general-purpose mechanisms rather than specialized complexity. Finally, they discuss how outages during training runs might have severe consequences, joke about robots and driverless cars as marketing hype indicators, and invite operations professionals to discuss real AI data center challenges.
About this episode
Scott Robohn sits down with guests Lenny Giuliano and Russ White to examine whether generative AI will affect internet traffic patterns. Together they explore the role of CDNs, the importance of latency, and the potential for a shift toward edge-based inference. They also discuss the impact of data sovereignty on infrastructure and the potential return<a class="excerpt-read-more" href="https://packetpushers.net/podcasts/total-network-operations/tno067-the-impact-of-ai-on-internet-traffic/" title="ReadTNO067: The Impact of AI on Internet Traffic">... Read more »</a>
Key Insights
- CDNs currently handle 80-90% of internet traffic by pre-positioning cacheable content, but AI inference results are fundamentally non-cacheable because each query generates unique outputs requiring different network approaches
- Time to first token latency matters critically for agent-to-agent AI communication but less for human-facing applications, since humans read output slower than machines expect responses
- Video already comprises 60-98% of internet traffic and will likely remain the dominant use case for edge-based AI inference, particularly for real-time processing like autonomous vehicles and security monitoring
- AI model training has created strict coupling between compute, network, and scheduling requirements that may force adoption of deterministic networking technologies like admission control and traffic engineering that IP networks largely abandoned
- Power and space constraints in centralized data centers are creating necessity, not choice, to distribute compute across multiple smaller locations, potentially leading to federated training and inference fabrics across WANs
- The economic trade-off between transporting traffic across networks versus storing compute at the edge will vary by use case, but personalized content generation (custom video, podcasts) favors distributed models
- Enterprises concerned about data sovereignty may prefer local inference through ISPs or private data centers rather than cloud providers, creating a potential new business model for access providers to compete with hyperscalers
- Disaggregated inference separating memory-intensive preload stages from compute-intensive decode stages could improve hardware utilization, but optimal distribution (same rack, across city, centralized vs. edge) remains undetermined
- Historically, over-engineered network technologies like ATM lost to simpler best-effort approaches like IP, suggesting AI requirements may evolve to reduce dependencies on perfect network behavior rather than requiring permanent infrastructure changes
- Network operator impacts will likely appear as increased complexity, new performance requirements, and return of technologies like anycast, multicast, and traffic steering for directing inference workloads to optimal edge locations based on GPU availability
- CDN providers with existing edge infrastructure are natural candidates to add GPU-accelerated inference capability, but this intensifies competition with hyperscalers who already operate distributed data centers and understand both compute and delivery
- Industry pattern shows product development typically prioritizes functionality first, then management, with security coming late (step 79), suggesting current AI deployments may lack robust security frameworks despite rapid production adoption
Topics
Transcript
. Welcome to Total Network Operations. You know the gig. We're here to talk about great ideas and network operations, and we have a really fun episode today. We're here at Kerasoft in Reston, Virginia recording right before the Virginia NUG, the Virginia Networking User Group meeting in May 2026. And couldn't have two more fun guests to have for this episode today, Lenny Giuliano and Russ White. And let me have them do the honors and introduce themselves. Lenny, why don't you go ahead? Thanks, Scott. Great to be here. I work at HPE now, formerly Juniper, it's HPE networking. And I am on their architecture team, senior distinguished technologist on the service provider architecture team. Great. Welcome, Russ.…
Full transcript available for MurmurCast members
Sign Up to AccessMore from The Everything Feed - All Packet Pushers Pods
TNO071: The Network Team Is Drowning. Is AI the Life Raft? (Sponsored)
Rekha Shenoy and Irfan Kimji from Backbox discuss how the exponential growth of vulnerabilities (49,000 CVEs annually) has made manual network operations unsustainable, and how AI-powered automation can help network teams manage patches and security updates at scale while maintaining human control and oversight.
HN840: How to Make a Technology Buying Decision
Sean Morgan, a research director at Deloro Group, discusses how technology buying decisions should extend beyond engineering specifications to include business alignment, ROI calculations, and understanding total cost of ownership. Engineers must shift from viewing IT as a cost center to positioning it as a business enabler by connecting technical decisions to revenue impact and organizational objectives.
IPB207: Flying Blind: Monitoring Might Not See IPv6
The IPv6 Buzz hosts discuss critical gaps in IPv6 monitoring across enterprise networks, highlighting that many monitoring platforms lack IPv6 awareness, vendor parity, and advanced analytical capabilities. They emphasize that while basic IPv6 data ingestion has improved, sophisticated features like cross-protocol event correlation, extension header analysis, and device identity tracking remain significant industry challenges.
N4N063: Link Layer Discovery Protocol
Link Layer Discovery Protocol (LLDP) is a standardized Layer 2 protocol that enables network devices to announce information about themselves to directly connected neighbors, facilitating network topology discovery and device identification in multi-vendor environments. The protocol uses Ethernet frames with special multicast destination MAC addresses to ensure frames don't propagate beyond immediate neighbors, and includes mandatory TLVs (Type-Length-Values) like chassis ID, port ID, and TTL alongside optional ones for extended information.
TCG083: Superintelligence for Everyone: Who Actually Holds the Power?
Three technology experts discuss Mark Zuckerberg's manifesto on distributed superintelligence, examining whether his promises of universal access and individual empowerment align with infrastructure realities. They conclude that while decentralized AI is theoretically safer than centralized control, the manifesto fails to account for human complexity, existing inequalities, and the enormous capital requirements that will likely concentrate power rather than distribute it.