Goodfire Cuts AI Reasoning Debug Costs by 100x

AI safety startup Goodfire has introduced Forking Fast, a technique designed to slash the computational expense of Forking Paths Analysis (FPA) by approximately 100-fold. FPA is a diagnostic method that identifies exactly which tokens in a long chain of thought cause an AI model to change its final answer. While highly effective for tracing hallucinations and safety failures, the process historically required resampling millions of tokens per reasoning chain, making it too expensive for practical deployment.

Forking Fast overcomes this financial barrier by treating low-sample uncertainty dynamics as multinomial sampling noise, where reconstruction error decays in proportion to the square root of the sample count. Instead of paying for expensive high-sample runs, the method fits a statistical smoothing model to cheap, low-sample estimates to reconstruct a clean distribution. The researchers evaluated the technique on Llama-3-8B-Instruct and DeepSeek-R1-Distill-Llama-8B using tinyMMLU, a 100-item subset of the Massive Multitask Language Understanding benchmark. In total, the evaluation processed nearly two billion tokens of reasoning data.

For practitioners, this dramatic cost reduction transforms FPA from an academic exercise into a viable tool for active agent pipelines. Developers can now pinpoint the exact tokens where an agent locks in an incorrect or unsafe decision, allowing them to apply targeted interventions like steering vectors or activation edits rather than scanning entire transcripts. Goodfire has released the code and an interactive dashboard on GitHub at github.com/ericb-goodfire/forking-fast to let developers run the smoothing model on their own rollouts.

Notably, the underlying research paper and its initial draft were generated under human guidance by Silico, Goodfire's autonomous interpretability research agent. Silico is commercially available to individual researchers for $1,000 per month, with discounts offered for life sciences and AI safety work. The project's human lead compared the agent's contribution to that of a first- or second-year doctoral student, marking a significant milestone in autonomous AI research.