Leaderboards, Not Cameras [89]
The episode examines why AI adoption dashboards and leaderboards often fail to drive behavior change, and proposes an alternative approach: publicly sharing AI workflows that actually shipped on real deals, ranked by peer reuse rather than usage metrics.
Summary
Tim and the SE Rockstars team present a case study on measuring AI adoption in solution engineering teams. The episode begins with a real scenario where a leader discovered that despite all 12 team members claiming daily AI usage in a meeting, actual adoption metrics showed only 2 heavily using the tools and 4 barely touching them—a classic over-reporting problem.
The hosts reference a study about surgeons' hand-washing compliance, where a camera in the scrub room produced no behavior change, but making results public via leaderboard significantly improved compliance. The mechanism: peer visibility matters more than monitoring.
However, the speakers caution against naive leaderboard implementations. One example illustrates a director celebrating high token usage ($7,000/month per SE) that turned out to be artificial—prompts fired just to stay on top of the metrics. They also cite research showing employees consistently over-report AI adoption to their managers.
The solution presented is a "Shipped with AI" channel where SEs post real artifacts from live deals (demo scripts, discovery briefs, security questionnaires, etc.) alongside which deal benefited and rough time saved. Critically, the ranking metric isn't self-reported time saved or tokens consumed—it's reuse rate: whether peer SEs adopt the workflow on their own deals. This creates peer accountability that's harder to fake than personal metrics.
A concrete example shows a hesitant SE posting a competitive battle card workflow in week two, which three colleagues reused on live deals, converting her from reluctant to proactive.
For larger organizations (200+ SEs across regions), the hosts recommend ranking teams rather than individuals, with managers handling individual performance privately. This preserves the peer effect at team level while avoiding individual shaming and potential legal concerns in regulated environments.
The episode concludes by noting that adoption holdouts often have legitimate reasons (data privacy concerns, unclear policies) rather than skill gaps, and recommends investigating root causes through one-on-ones rather than assuming resistance indicates laziness.
About this episode
Ava found that quietly monitoring her team's AI usage changed nothing, while a public board of reused AI workflows did. Nate pushes back on what happens to individual leaderboards at 200 SEs.
Key Insights
- The speaker found that direct monitoring via private dashboards produced no behavior change over three months, but discovered through research that making results publicly visible via leaderboard (like surgeons' hand-washing compliance) significantly changed behavior—suggesting peer visibility works better than surveillance.
- Token usage leaderboards can incentivize theater rather than value: one SE burned $7,000/month simply firing prompts to appear active, revealing that metrics divorced from actual business output reward gaming rather than adoption.
- The speaker argues that ranking workflows by peer reuse rather than self-reported metrics creates harder-to-fake accountability because colleagues choosing to adopt another person's prompt is harder to artificially inflate than personal usage statistics.
Topics
Transcript
Hey there and welcome to Leading Pre-Sales, the show for solution engineering leaders who want to build teams that drive revenue and not just demos. My name is Tim and I'm the co-founder of SE Rockstars and together with Jan, we've coached over 350 solution engineers and their leaders across several dozens of companies. Every conversation you hear on this show is based on real coaching situations, real challenges, real problems that SE leaders like you are dealing with right now. None of this is made up. We use AI to bring these stories to life through our two hosts, Nate and Ava, but the insights come straight from the trenches. Each episode gives you one actionable takeaway you can…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Leading PreSales | The Solution Engineering Leadership Show
Core vs. Context: When Your SEs Should Let a Build Deal Go [92]
The episode teaches solution engineering leaders to distinguish between core and context business functions when customers consider building custom solutions versus buying software. SEs should let lightweight, context-focused builds go while fighting harder for core capabilities, and leaders must publicly validate this approach to change team culture.
The Iceberg Under Every Build Deal [91]
Solution engineering leaders need a standardized five-year cost model to evaluate Build vs. Buy decisions in the AI era, grounded in hidden maintenance costs and opportunity expenses rather than vendor pitches. The key is creating an 'iceberg checklist' of often-overlooked costs and having customers co-create the financial analysis rather than presenting a pre-built case.
Your SEs Just Inherited the Customer's Bill [88]
Solution engineers must own usage estimation and sizing in consumption-based pricing models, even though they don't set prices. When SEs fail to accurately estimate how often workflows trigger, customers face billing surprises that damage relationships and renewals.
Rebrand or Real Role? The FDE Test [86]
The episode examines whether Forward Deployed Engineers (FDEs) represent a genuinely different role from Solution Engineers or merely a rebranding strategy. Through analysis of real job postings and market trends, the hosts present a three-question test to distinguish authentic FDE positions from SE roles with inflated titles.
No Quota, Still Commercial [85]
Forward Deployed Engineers (FDEs) should be measured on time-to-first-value and usage ramp rather than traditional sales quotas, with compensation tied to commercial outcomes that directly correlate to revenue under consumption pricing models. Success requires a disciplined 10x cost-recovery model, a defined pilot-to-contract path, and mandatory product feedback loops to prevent FDE teams from becoming expensive consulting operations.