The AI factory playbook for engineering teams
Zach Lloyd, CEO of Warp, explains how software factories—organized systems of AI agents integrated across development tools—enable teams to automate the full software development lifecycle while maintaining human oversight. The approach centralizes work in public spaces like Slack, measures efficiency through metrics like human interactions per PR, and uses continuous evaluation and self-improvement loops to optimize both cost and quality.
Summary
The transcript covers a detailed discussion of Warp's software factory concept, which represents a shift from individual local coding agents to organized, cloud-based systems that integrate with Slack, GitHub, Linear, and other tools. Lloyd demonstrates how a team member can kick off work in Slack by tagging their factory agent 'Wilson,' which then executes a complete workflow: triaging the task, creating a Linear issue, implementing the code, opening a PR, performing QA with computer use verification (including video documentation), and merging the result—all visible and collaborative in public channels.
A central innovation is the measurement and analysis infrastructure. Lloyd introduces 'human interactions per PR' as a key metric—counting reprompts in Slack, Linear comments, and code review iterations—as a proxy for how much human steering is needed. He notes that at Warp, kickoff to PR takes 35 minutes, but PR to first human review takes 3.5 hours, revealing humans as the current bottleneck. The team is moving toward risk-scored code review, where low-risk PRs get automated approval while higher-risk changes require human review.
The factory approach includes sophisticated evaluation infrastructure. Warp has implemented 'scoring' systems using LLM-as-judge evaluations across dimensions like redundant tests, code quality, and other failure modes. When agents write unnecessary tests, a scorer detects this pattern across all runs. Beyond measurement, Warp implements a 'self-improvement' loop where an observer agent analyzes failed runs (requiring 20-25 samples for significance) and proposes updates to the factory definition itself—all in code, allowing agents to improve the system autonomously.
Lloyd demonstrates CEO-level applications beyond engineering, using coding agents with MCPs (Model Context Protocols) to modify Figma slides semantically (describing desired layout relationships like 'host contains sandbox'), to analyze sales call transcripts via Granola MCP to extract the top 10 customer questions about Warp Factories, and to identify cold leads from past six months of emails and calendar events. He emphasizes maintaining perspective on the technology's capabilities and maintaining human-driven product development processes—design jams and user interviews—even while accelerating implementation.
About this episode
Zach Lloyd is the co-founder and CEO of Warp, an AI-powered terminal and software factory platform used by tens of thousands of engineers. Before Warp, he spent nearly a decade at Google, including time as a principal engineer on Google Sheets. He built Warp from the ground up as a modern, AI-native alternative to legacy terminals, and the team has since expanded into software factories: a full cloud-based system that takes an idea in Slack all the way through to a merged PR. *In this episode:* 1. Why a software factory is more than a coding agent 2. The public Slack → Linear → GitHub → QA workflow 3. Human interactions per PR as a signal of automation and throughput 4. Why human review is still the bottleneck 5. Scoring agent runs, finding failure modes, and self-improving agent workflows 6. Replaying real tasks to choose model cost and quality tradeoffs 7. CEO workflows with Figma MCP, Granola, and research agents *Brought to you by:* DX—Engineering intelligence for the AI era: https://getdx.com/howiai OpenArt—An all-in-one AI creation platform for images, videos, music, audio, and more: https://openart.ai/suite/chat?utm_source=online&utm_medium=influencer&utm_campaign=infl-howiai-ga-na-acq-web *In this episode, we cover:* (00:00) Intro (02:35) Warp’s AI software factory, Wilson (09:23) Automatic factory triggers (11:12) The engineering leader dashboard Zach wishes he’d had (15:18) How code review is changing in an AI factory (17:08) Tracking cost per PR across model configs (18:47) Using LLM-as-a-judge to score every agent run (20:02) Catching redundant tests (22:19) How the factory self-improves from failed runs (26:03) Quick recap (28:33) Building a cost-quality Pareto chart for model selection (31:35) How Zach uses AI for non-technical CEO work (32:10) Figma MCP demo (35:43) Granola MCP demo (36:41) GOG CLI demo (38:20) Thinking in parallel tasks instead of sequential ones (40:42) Zach’s prompting strategy for factory tasks (44:48) Where to find Zach *Blog and detailed workflow walkthroughs from this episode:* Inside Warp’s Software Factory: https://www.chatprd.ai/how-i-ai/inside-warps-software-factory ↳ How to Build an Automated Software Factory: https://www.chatprd.ai/how-i-ai/workflows/how-to-build-an-automated-software-factory ↳ How to Build an AI-Powered CEO Toolkit: https://www.chatprd.ai/how-i-ai/workflows/how-to-build-an-ai-powered-ceo-toolkit ↳ How to Measure and Self-Improve Your AI Software Development Factory: https://www.chatprd.ai/how-i-ai/workflows/how-to-measure-and-self-improve-your-ai-software-development-factory *Tools referenced:* • Warp (AI terminal and software factories): https://warp.dev • Warp Factories: https://warp.dev/factories • Linear (project and issue tracking): https://linear.app • GitHub (version control and PR management): https://github.com • Slack (team communication and factory input layer): https://slack.com • Sentry (crash reporting and automated issue triggers): https://sentry.io • Figma (design, used via Figma MCP): https://figma.com • Granola (AI meeting notes and MCP integration): https://granola.so • Grok Bot (fast inference, cost/quality trade-off): https://x.ai/bot/guides/grok-bot-101 *Where to find Zach:* X: https://x.com/ZachLloydTweets *Where to find Claire:* ChatPRD: https://www.chatprd.ai/ Website: https://clairevo.com/ LinkedIn: https://www.linkedin.com/in/clairevo/ X: https://x.com/clairevo _Production and marketing by https://penname.co/._ _For inquiries about sponsoring the podcast, email [email protected]._
Key Insights
- Humans are currently the bottleneck in software development, taking 3.5 hours from PR to first human review compared to 35 minutes from kickoff to PR, making code review and human capacity management the critical constraint
- A software factory consists of interconnected repos, MCP servers, configuration, and agents—all defined in code—rather than ad-hoc local setups, enabling reproducibility, testing, and agent-driven improvements
- Work performed in public Slack channels rather than private local environments creates visibility for teams, enables knowledge transfer from power users to average users, and allows anyone to observe and contribute to tasks
- Human interactions per PR—counting reprompts, comments across tools, and corrections—serves as a metric for throughput limitations, reflecting how much steering and intervention is required to complete automated tasks
- Factory agents can perform computer use verification by creating video documentation of completed features, providing evidence of correct implementation beyond code review alone
- Scoring systems using LLM-as-judge enable retroactive evaluation of all agent runs across custom dimensions, identifying failure modes like redundant tests or code quality issues at scale rather than individually
- Observer agents can analyze aggregated failed runs (requiring 20-25 samples minimum) and propose code-level updates to the factory definition itself, creating a self-improving system that prevents recurring failure modes
- Teams can replay past factory tasks with different model configurations to measure real cost and quality tradeoffs on their own data rather than relying on generic benchmarks, informing model routing decisions
Topics
Transcript
[0:00] Can I give you a hard time that humans really are the bottleneck? Because if you look at kickoff to PR time, it's 35 minutes, but if you look at PR to first human review, it's 3 and 1/2 [music] hours. And when you're doing I think it was like over 2,000 PRs in the last month. Like how do you keep things in the team from going as I say like chaos reigns? >> What is a software factory? For us at least, it's an actual noun. It's like a product concept where it consists of a bunch of repos, a bunch of like MCP servers, like a bunch of configuration and then a bunch of agents essentially.…
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
How I Happy Hour - Lenny's Summit 2026 - Tiny Mic
At Lenny's Summit 2026, speakers discuss practical AI automation tools and workflows they use daily, including Cloud Code for project setup, customer feedback ingestion systems, and AI-powered content creation for employee training at scale.
Muse made me a 6-min AI podcast
The speaker demonstrates Muse, an AI tool that generates podcasts, by having it create a 6-minute podcast about AI news from a weekend in September 2026. The tool quickly produced a conversational two-host format discussing contrasting perspectives on AI safety concerns, and impressively offers direct publishing options to Spotify and Apple Podcasts.
Grok Bot sends this design into Figma from my phone
A designer demonstrates how an AI bot can convert voice descriptions and screenshots into interactive Figma prototypes, generating multiple design iterations from natural language input. The tool interprets conceptual descriptions and produces working prototypes without requiring manual prototyping work in Figma.
Muse beat OpenClaw and Codex on design
The speaker demonstrates Muse, Meta's AI agent, successfully generating a one-page PDF newsletter for family schedules and news that surpasses designs from competing AI tools like OpenClaw and Codex. After connecting to email and allowing the AI to ingest personal family information, Muse produced superior design and content quality for the requested morning brief.
Muse gets AI agent UX right
Claire reviews Meta's Muse personal AI agent, praising its consumer-friendly design, intuitive UX patterns, and thoughtful features like identity customization, activity feeds, and goal tracking. While impressed by the overall experience and artifact generation quality, she identifies browser-based shopping as an area where Muse underperforms compared to competitors like Claude.