D2DO301: Actually Implementing AI

Enrico Teodi, an independent consultant with 25 years of software experience spanning engineering and product management, joined the Day 2 DevOps podcast to share concrete, real-world AI implementation stories from a 13-14 month engagement at a software company. He was originally brought in to address software quality issues, including poor test coverage and brittle unit tests that gave false confidence without genuine integration testing.

His first major AI discovery was connecting tools like Windsurf to the full application codebase rather than just the database schema. This allowed the AI agent to reason about business logic encoded in enums and application code, not just table definitions—dramatically expanding the quality of questions it could answer. He then escalated this by connecting the agent to a read-only production database replica via Kubernetes port forwarding, enabling root cause analysis on real production data rather than local test data.

Enrico described several concrete use cases: debugging a record ordering bug where the agent initially concluded the ordering was correct before being redirected to find a missing ORDER BY clause in a polymorphic table; and diagnosing a slow page load by combining MCP-connected tools including Sentry for performance monitoring and PostHog for analytics, only to discover that the slow feature had been clicked by just three people in four months, two of whom were likely developers testing it. This multi-tool agentic approach compressed what would have been hours of manual investigation into approximately five minutes.

He raised strong concerns about AI permissions and trust, explicitly stating he does not trust AI with write access to production systems. He advocated for read-only database access, isolated VMs for tools like Open Claude, and tightly scoped permissions that expand incrementally. He warned that system prompts are ineffective guardrails and that AI makes poor decisions when given admin-level permissions.

On the human side, Enrico argued that the developers who succeeded were not the fastest coders but those who understood the product, asked precise questions, and recognized when the AI was wrong. He expressed concern about 'AI slop'—code generated without clear acceptance criteria, proper testing, or human oversight. He recounted an analyst who burned $800 in hours by bypassing the company's AI gateway with a direct API key, and another who ran Cloud Code against a disconnected local environment for hours using CI failures as feedback, ultimately failing to deploy a working feature.

He framed the current moment as requiring tighter product-engineering collaboration, stronger definitions of 'done,' and test-driven development as a non-negotiable foundation. He compared the current AI shift to the rollout of Microsoft Office—a moment that democratized powerful tools but required new literacy to prevent catastrophic misuse. He also expressed optimism about local models eventually reducing dependence on expensive third-party frontier model services, viewing AI as simply the next abstraction layer above previous programming paradigms.