MurmurCast
TechnicalInsightful

You don't have to review every AI-generated PR

How I AI

The speaker describes an AI agent framework that automatically reviews pull requests by scoring them across six risk factors using markdown-based instructions and skills. PRs are classified as low (0-24 points), medium (25-64), or high risk (65+), with only medium and high risk requiring human approval.

Summary

The speaker explains how to build AI review agents using a framework that relies on writing instructions and skills in markdown format. Once an agent reads the code in a pull request, it evaluates risk across six dimensions: the size of the change surface and blast radius, whether changes are easily reversible, whether the PR touches data security and if it addresses security concerns, whether it impacts operations, and the verification gap (whether tests are complete and CI has passed). The agent runs a script to calculate a composite risk score. The scoring system uses three tiers: low risk (0-24 points), medium risk (25-64 points), and high risk (65+ points). This tiered approach allows teams to skip human review for genuinely low-risk changes while ensuring that medium and high-risk PRs still receive human attention, making the review process more efficient.

Key Insights

  • Creating AI review agents requires only writing instructions and skills, with markdown being particularly suitable for this purpose
  • The agent evaluates risk across six specific dimensions including change surface size, reversibility, data security impact, operational changes, and verification completeness
  • The framework uses a numerical scoring system where PRs below 24 points are considered low risk and automatically approved without human review
  • Medium and high-risk PRs (25+ points) require human approval, creating a hybrid approach that eliminates unnecessary human review while maintaining safety gates
  • Verification gaps are assessed by checking whether tests are complete and CI has passed, factoring into the overall risk calculation

Topics

AI agents for code reviewRisk scoring frameworkPull request evaluation criteriaAutomated vs. human reviewMarkdown-based instructions

Transcript

[0:00] Writing these agents is not that hard. It is literally just writing instructions and skills. That is what it takes to create an agent like this and that's why I like this e-framework cuz you can do a lot of it in markdown. And then once it reads the code, it scores the risk. And so it looks at six things. How big is the change surface and blast radius? Is it easily reversible? Does it touch data security and does it cover data security? Does it change anything about our operations? And verification gap is like are our tests complete? Did CI complete? Can we actually test a [0:31] couple ways that this is true? And then it…

Full transcript available for MurmurCast members

Sign Up to Access
View original source →

More from How I AI

Get AI summaries like this delivered to your inbox daily

Get AI summaries delivered to your inbox

MurmurCast summarizes your YouTube channels, podcasts, and newsletters into one daily email digest.