{"id":743,"date":"2026-04-18T14:00:41","date_gmt":"2026-04-18T06:00:41","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=743"},"modified":"2026-04-18T14:00:41","modified_gmt":"2026-04-18T06:00:41","slug":"google-ai-releases-auto-diagnose-an-large-language-model-llm-based-system-to-diagnose-integration-test-failures-at-scale","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=743","title":{"rendered":"Google AI Releases Auto-Diagnose: An Large Language Model LLM-Based System to Diagnose Integration Test Failures at Scale"},"content":{"rendered":"

If you have ever stared at thousands of lines of integration test logs wondering which of the sixteen log files actually contains your bug, you are not alone \u2014 and Google now has data to prove it.<\/p>\n

A team of Google researchers introduced Auto-Diagnose<\/strong>, an LLM-powered tool that automatically reads the failure logs from a broken integration test, finds the root cause, and posts a concise diagnosis directly into the code review where the failure showed up. On a manual evaluation of 71 real-world failures spanning 39 distinct teams<\/strong>, the tool correctly identified the root cause 90.14% of the time<\/strong>. It has run on 52,635 distinct failing tests<\/strong> across 224,782 executions<\/strong> on 91,130 code changes<\/strong> authored by 22,962 distinct developers<\/strong>, with a \u2018Not helpful\u2019 rate of just 5.8% on the feedback received.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.12108<\/figcaption><\/figure>\n<\/div>\n

The problem: integration tests are a debugging tax<\/strong><\/h3>\n

Integration tests verify that multiple components of a distributed system actually communicate to each other correctly. The tests Auto-Diagnose targets are hermetic functional integration tests<\/em>: tests where an entire system under test (SUT) \u2014 typically a graph of communicating servers \u2014 is brought up inside an isolated environment by a test driver, and exercised against business logic. A separate Google survey of 239 respondents found that 78% of integration tests at Google are functional<\/strong>, which is what motivated the scope.<\/p>\n

Diagnosing integration test failures showed up as one of the top five complaints in EngSat<\/em>, a Google-wide survey of 6,059 developers. A follow-up survey of 116 developers found that 38.4% of integration test failures take more than an hour to diagnose, and 8.9% take more than a day<\/strong> \u2014 versus 2.7% and 0% for unit tests.<\/p>\n

The root cause is structural. Test driver logs usually surface only a generic symptom (a timeout, an assertion). The actual error lives somewhere inside one of the SUT component logs, often buried under recoverable warnings and ERROR-level lines that are not actually the cause.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.12108<\/figcaption><\/figure>\n<\/div>\n

How Auto-Diagnose works<\/strong><\/h3>\n

When an integration test fails, a pub\/sub event triggers Auto-Diagnose. The system collects all test driver and SUT component logs at level INFO and above \u2014 across data centers, processes, and threads \u2014 then joins and sorts them by timestamp into a single log stream<\/strong>. That stream is dropped into a prompt template along with component metadata.<\/p>\n

The model is Gemini 2.5 Flash<\/strong>, called with temperature = 0.1<\/code> (for near-deterministic, debuggable outputs) and topp<\/sub> = 0.8<\/code>. Gemini was not fine-tuned on Google\u2019s integration test data; this is pure prompt engineering on a general-purpose model.<\/p>\n

The prompt itself is the most instructive part of this research. It walks the model through an explicit step-by-step protocol: scan log sections, read component context, locate the failure, summarize errors, and only then attempt a conclusion. Critically, it includes hard negative constraints \u2014 for example: if the logs do not contain lines from the component that failed, do not draw any conclusion.<\/em> <\/p>\n

The model\u2019s response is post-processed into a markdown finding with ==Conclusion==<\/code>, ==Investigation Steps==<\/code>, and ==Most Relevant Log Lines==<\/code> sections, then posted as a comment in Critique<\/strong>, Google\u2019s internal code review system. Each cited log line is rendered as a clickable link.<\/p>\n

Numbers from production<\/strong><\/h3>\n

Auto-Diagnose averages 110,617 input tokens and 5,962 output tokens per execution<\/strong>, and posts findings with a p50 latency of 56 seconds and p90 of 346 seconds<\/strong> \u2014 fast enough that developers see the diagnosis before they have switched contexts.<\/p>\n

Critique exposes three feedback buttons on a finding: Please fix<\/em> (used by reviewers), Helpful<\/em>, and Not helpful<\/em> (both used by authors). Across 517 total feedback reports from 437 distinct developers, 436 (84.3%) were \u201cPlease fix\u201d<\/strong> from 370 reviewers \u2014 by far the dominant interaction, and a sign that reviewers are actively asking authors to act on the diagnoses. Among dev-side feedback, the helpfulness ratio (H \/ (H + N)<\/code>) is 62.96%, and the \u201cNot helpful\u201d rate (N \/ (PF + H + N)<\/code>) is 5.8% \u2014 well under Google\u2019s 10% threshold for keeping a tool live. Across 370 tools that post findings to Critique<\/strong>, Auto-Diagnose ranks #14 in helpfulness, putting it in the top 3.78%<\/strong>.<\/p>\n

The manual evaluation also surfaced a useful side effect. Of the seven cases where Auto-Diagnose failed, four were because test driver logs were not properly saved on crash, and three were because SUT component logs were not saved when the component crashed \u2014 both real infrastructure bugs, reported back to the relevant teams. In production, around 20 \u2018more information is needed<\/em>\u2018 diagnoses have similarly helped surface infrastructure issues.<\/p>\n

Key Takeaways<\/strong><\/h3>\n