ROI on Token: Stop Rewarding AI Usage [64]

Tim introduces a coaching story from a working session where 12 solution engineers listed AI tools they'd built (demo data generators, call recording bots) but couldn't account for how they spent the recovered time. This reveals a fundamental leadership gap: celebrating AI adoption without tracking outcomes.

Nate and Ava discuss how usage metrics create false progress signals. Token consumption leaderboards and dashboards measure activity, not impact. Some companies even tied AI usage to compensation, replicating the "booked meetings" metric mistake where activity was rewarded regardless of actual value. The pendulum has already swung—vendors introduced token budgets, then cut them when costs were overestimated, suggesting neither extreme solves the core problem.

Ava pushes back that early-stage usage signals matter for adoption (knowing if a third of the team hasn't opened a tool is real data), but caps this at 90 days before usage becomes the goal itself rather than a leading indicator.

Nate shares a personal example: an hour of AI tool-building consumed his Saturday, but the quality output was superior to what he'd produce manually. This reveals that the payoff isn't always speed—it can be quality—and measuring only acceleration would have incorrectly logged a success as failure.

They identify a permanent hidden cost: quality control doesn't disappear with AI. A director's team had AI draft bid sections with duplicated clauses and contradictory commitments because the model lost context across documents. Human review remains non-negotiable.

The critical insight: without explicit direction, reclaimed capacity vanishes into more building because building is intrinsically enjoyable while follow-up work on stalled deals is not. Tim provides a concrete example: his team spends roughly half their week on internal work (prep, docs, handovers, decks). He set a target of reducing that to a quarter of the week so each SE gains a full customer-facing day—a measurable, held commitment rather than vague efficiency gains.

They close with the core framework: before adopting any AI workflow, write two lines—what outcome it produces and where reclaimed hours go. If the second line is empty, you have a hobby, not a business case. If the answer is quality rather than speed, state that explicitly to prevent clock-gaming.