PP112: When You Look But Don’t Find: The Art of Knowing When to Stop

Sydney Maroney, a detection engineer and threat hunter with experience at utilities, Nordstrom, telecom, and Splunk, joins hosts Jennifer Jabosh and Drew Conrey-Murray to discuss the art and science of threat hunting. The conversation opens with a foundational explanation of threat hunting: it begins with threat intelligence, forms a hypothesis about attacker behavior in an environment, and then searches logs to prove or disprove that hypothesis. Critically, Sydney emphasizes that a negative result — finding nothing — is still a successful hunt, as it builds knowledge, reveals process gaps, and can surface new detection ideas.

Sydney introduces the PEAK Threat Hunting Framework, which she co-created with David Bianco and Dr. Ryan Fetterman at Splunk. PEAK stands for Prepare, Execute, and Act with Knowledge, where Knowledge is woven throughout. She stresses that most hunters skip directly to Execute, but the Prepare phase — including scoping data sources, researching attacker behaviors, and forming a precise hypothesis — is where the most time should be spent. The Act phase includes documentation, which Sydney argues is critical: if it's not documented, it didn't happen. She notes that threat hunt outputs can include new detections, process improvements, risk findings, and expanded logging coverage.

The discussion then turns to the central theme of her blog post: knowing when to stop hunting. Sydney outlines several criteria, including coverage (do you have visibility into the relevant data sources?), diminishing returns (are you finding the same things repeatedly?), and time-boxing (setting hour-or-two increments and not exceeding them significantly without new findings). She describes a confidence spectrum — low, medium, high — that should be communicated transparently to leadership, especially when low confidence stems from visibility gaps like missing network logs. She advocates for raising these gaps as findings rather than hiding them.

Sydney also describes red flags for stopping too early or going too long, with scope creep being a primary concern. When hunters encounter interesting but off-topic threads, she recommends parking those in a backlog rather than chasing them, preserving curiosity while maintaining focus. She notes that hypothesis scope almost always needs to be narrowed once execution begins, as initial scopes frequently yield millions of events.

The conversation shifts to Sydney's newer Agentic Threat Hunting Framework, which layers AI on top of PEAK to solve the memory and documentation problem. Built around Markdown files stored in a repository, the framework enables both humans and AI agents to query past hunts semantically, answer questions like 'what have I hunted in the past six months,' and compound institutional knowledge over time. It includes a maturity model and a CLI with built-in semantic search, and is designed to help teams move beyond basic ChatGPT interactions into more agentic AI workflows.

Finally, Sydney discusses the Thor Collective, a group she co-founded with former colleagues Lauren Prail and John Gregeta. They publish a newsletter called Dispatch, run a podcast, and maintain a GitHub repository called Hearth, which contains nearly 200 threat hunting hypothesis ideas for practitioners looking for starting points.