ROI on Token: Stop Rewarding AI Usage [64]
AI usage metrics are vanity metrics that mask whether AI tools actually produce business value. Leaders must shift focus from measuring token consumption to defining specific outcomes and explicitly allocating reclaimed time, or efficiency gains will disappear into building more tools.
Summary
Tim introduces a coaching story from a working session where 12 solution engineers listed AI tools they'd built (demo data generators, call recording bots) but couldn't account for how they spent the recovered time. This reveals a fundamental leadership gap: celebrating AI adoption without tracking outcomes.
Nate and Ava discuss how usage metrics create false progress signals. Token consumption leaderboards and dashboards measure activity, not impact. Some companies even tied AI usage to compensation, replicating the "booked meetings" metric mistake where activity was rewarded regardless of actual value. The pendulum has already swung—vendors introduced token budgets, then cut them when costs were overestimated, suggesting neither extreme solves the core problem.
Ava pushes back that early-stage usage signals matter for adoption (knowing if a third of the team hasn't opened a tool is real data), but caps this at 90 days before usage becomes the goal itself rather than a leading indicator.
Nate shares a personal example: an hour of AI tool-building consumed his Saturday, but the quality output was superior to what he'd produce manually. This reveals that the payoff isn't always speed—it can be quality—and measuring only acceleration would have incorrectly logged a success as failure.
They identify a permanent hidden cost: quality control doesn't disappear with AI. A director's team had AI draft bid sections with duplicated clauses and contradictory commitments because the model lost context across documents. Human review remains non-negotiable.
The critical insight: without explicit direction, reclaimed capacity vanishes into more building because building is intrinsically enjoyable while follow-up work on stalled deals is not. Tim provides a concrete example: his team spends roughly half their week on internal work (prep, docs, handovers, decks). He set a target of reducing that to a quarter of the week so each SE gains a full customer-facing day—a measurable, held commitment rather than vague efficiency gains.
They close with the core framework: before adopting any AI workflow, write two lines—what outcome it produces and where reclaimed hours go. If the second line is empty, you have a hobby, not a business case. If the answer is quality rather than speed, state that explicitly to prevent clock-gaming.
About this episode
Ava ran an AI session with her twelve SEs and discovered they'd built plenty — but nobody could say what they did with the time they saved. She and Nate dig into why usage metrics feel like progress, and how to make outcomes the thing you actually measure.
Key Insights
- Tim observed that 12 SEs collectively built multiple AI tools but when asked what they did with the recovered time, not a single person could account for it—all the efficiency gains disappeared into building more tools, revealing a leadership gap in directing reclaimed capacity.
- Nate argues that measuring AI value as speed-to-completion can actually hide genuine wins: an AI tool that produced higher quality work took him longer (a full Saturday), yet would be incorrectly logged as a failure if acceleration alone was the metric.
- Token consumption leaderboards and usage-based compensation replicate the failed 'booked meetings' model by rewarding activity divorced from outcomes—some vendors even created internal rankings that shout out top token consumers regardless of business impact.
Topics
Transcript
Hey there and welcome to Leading Pre-Sales, the show for solution engineering leaders who want to build teams that drive revenue and not just demos. My name is Tim and I'm the co-founder of SE Rockstars and together with Jan, we've coached over 350 solution engineers and their leaders across several dozens of companies. Every conversation you hear on this show is based on real coaching situations, real challenges, real problems that SE leaders like you are dealing with right now. None of this is made up. We use AI to bring these stories to life through our two hosts, Nate and Ava, but the insights come straight from the trenches. Each episode gives you one actionable takeaway you can…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Leading PreSales | The Solution Engineering Leadership Show
Leaderboards, Not Cameras [89]
The episode examines why AI adoption dashboards and leaderboards often fail to drive behavior change, and proposes an alternative approach: publicly sharing AI workflows that actually shipped on real deals, ranked by peer reuse rather than usage metrics.
Your SEs Just Inherited the Customer's Bill [88]
Solution engineers must own usage estimation and sizing in consumption-based pricing models, even though they don't set prices. When SEs fail to accurately estimate how often workflows trigger, customers face billing surprises that damage relationships and renewals.
Rebrand or Real Role? The FDE Test [86]
The episode examines whether Forward Deployed Engineers (FDEs) represent a genuinely different role from Solution Engineers or merely a rebranding strategy. Through analysis of real job postings and market trends, the hosts present a three-question test to distinguish authentic FDE positions from SE roles with inflated titles.
No Quota, Still Commercial [85]
Forward Deployed Engineers (FDEs) should be measured on time-to-first-value and usage ramp rather than traditional sales quotas, with compensation tied to commercial outcomes that directly correlate to revenue under consumption pricing models. Success requires a disciplined 10x cost-recovery model, a defined pilot-to-contract path, and mandatory product feedback loops to prevent FDE teams from becoming expensive consulting operations.
Engineering Is the Entry Ticket [84]
Forward-deployed engineering engagements often fail due to poor discovery, scoping, and stakeholder management rather than technical deficiencies. Engineering skills are necessary to enter the room, but success depends on identifying success criteria, mapping stakeholders (especially those who lose if the project succeeds), and having leadership cover to push back on misguided requests.