{"id":686,"date":"2026-04-09T11:25:24","date_gmt":"2026-04-09T03:25:24","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=686"},"modified":"2026-04-09T11:25:24","modified_gmt":"2026-04-09T03:25:24","slug":"google-ai-research-introduces-paperorchestra-a-multi-agent-framework-for-automated-ai-research-paper-writing","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=686","title":{"rendered":"Google AI Research Introduces PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing"},"content":{"rendered":"

Writing a research paper is brutal. Even after the experiments are done, a researcher still faces weeks of translating messy lab notes, scattered results tables, and half-formed ideas into a polished, logically coherent manuscript formatted precisely to a conference\u2019s specifications. For many fresh researchers, that translation work is where papers go to die.<\/p>\n

A team at Google Cloud AI Research propose \u2018PaperOrchestra<\/strong>\u2018, a multi-agent system that autonomously converts unstructured pre-writing materials \u2014 a rough idea summary and raw experimental logs \u2014 into a submission-ready LaTeX manuscript, complete with a literature review, generated figures, and API-verified citations. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.05018<\/figcaption><\/figure>\n<\/div>\n

The Core Problem It\u2019s Solving<\/strong><\/h3>\n

Earlier automated writing systems, like PaperRobot, could generate incremental text sequences but couldn\u2019t handle the full complexity of a data-driven scientific narrative. More recent end-to-end autonomous research frameworks like AI Scientist-v1<\/strong> (which introduced automated experimentation and drafting via code templates) and its successor AI Scientist-v2<\/strong> (which increases autonomy using agentic tree-search) automate the entire research loop \u2014 but their writing modules are tightly coupled to their own internal experimental pipelines. You can\u2019t just hand them your data and expect a paper. They\u2019re not standalone writers.<\/p>\n

Meanwhile, systems specialized in literature reviews, such as AutoSurvey2<\/strong> and LiRA<\/strong>, produce comprehensive surveys but lack the contextual awareness to write a targeted Related Work<\/em> section that clearly positions a specific new method against prior art. CycleResearcher requires a pre-existing structured BibTeX reference list as input \u2014 an artifact rarely available at the start of writing \u2014 and fails entirely on unstructured inputs.<\/p>\n

The result is a gap: no existing tool could take unconstrained human-provided materials \u2014 the kind of thing a real researcher might actually have after finishing experiments \u2014 and produce a complete, rigorous manuscript on its own. PaperOrchestra is built specifically to fill that gap.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.05018<\/figcaption><\/figure>\n<\/div>\n

How the Pipeline Works<\/strong><\/h3>\n

PaperOrchestra orchestrates five specialized agents that work in sequence, with two running in parallel:<\/strong><\/p>\n

Step 1 \u2014 Outline Agent:<\/strong> This agent reads the idea summary, experimental log, LaTeX conference template, and conference guidelines, then produces a structured JSON outline. This outline includes a visualization plan (specifying what plots and diagrams to generate), a targeted literature search strategy separating macro-level context for the Introduction from micro-level methodology clusters for the Related Work, and a section-level writing plan with citation hints for every dataset, optimizer, metric, and baseline method mentioned in the materials.<\/p>\n

Steps 2 & 3 \u2014 Plotting Agent and Literature Review Agent (parallel):<\/strong> The Plotting Agent executes the visualization plan using PaperBanana<\/strong>, an academic illustration tool that uses a Vision-Language Model (VLM) critic to evaluate generated images against design objectives and iteratively revise them. Simultaneously, the Literature Review Agent conducts a two-phase citation pipeline: it uses an LLM equipped with web search to identify candidate papers, then verifies each one through the Semantic Scholar API<\/strong>, checking for a valid fuzzy title match using Levenshtein distance, retrieving the abstract and metadata, and enforcing a temporal cutoff tied to the conference\u2019s submission deadline. Hallucinated or unverifiable references are discarded. The verified citations are compiled into a BibTeX file, and the agent uses them to draft the Introduction and Related Work sections \u2014 with a hard constraint that at least 90% of the gathered literature pool must be actively cited.<\/p>\n

Step 4 \u2014 Section Writing Agent:<\/strong> This agent takes everything generated so far \u2014 the outline, the verified citations, the generated figures \u2014 and authors the remaining sections: abstract, methodology, experiments, and conclusion. It extracts numeric values directly from the experimental log to construct tables and integrates the generated figures into the LaTeX source.<\/p>\n

Step 5 \u2014 Content Refinement Agent:<\/strong> Using AgentReview<\/strong>, a simulated peer-review system, this agent iteratively optimizes the manuscript. After each revision, the manuscript is accepted only if the overall AgentReview score increases, or ties with net non-negative sub-axis gains. Any overall score decrease triggers an immediate revert and halt. Ablation results show this step is critical: refined manuscripts dominate unrefined drafts with 79%\u201381% win rates<\/strong> in automated side-by-side comparisons, and deliver absolute acceptance rate gains of +19% on CVPR<\/strong> and +22% on ICLR<\/strong> in AgentReview simulations.<\/p>\n

The full pipeline makes approximately 60\u201370 LLM API calls and completes in a mean of 39.6 minutes<\/strong> per paper \u2014 only about 4.5 minutes more than AI Scientist-v2\u2019s 35.1 minutes, despite running significantly more LLM calls (40\u201345 for AI Scientist-v2 vs. 60\u201370 for PaperOrchestra).<\/p>\n

The Benchmark: PaperWritingBench<\/strong><\/h3>\n

The research team also introduce PaperWritingBench<\/strong>, described as the first standardized benchmark specifically for AI research paper writing. It contains 200 accepted papers from CVPR 2025 and ICLR 2025 (100 from each venue), selected to test adaptation to different conference formats \u2014 double-column for CVPR versus single-column for ICLR.<\/p>\n

For each paper, an LLM was used to reverse-engineer two inputs from the published PDF: a Sparse Idea Summary<\/strong> (high-level conceptual description, no math or LaTeX) and a Dense Idea Summary<\/strong> (retaining formal definitions, loss functions, and LaTeX equations), alongside an Experimental Log<\/strong> derived by extracting all numeric data and converting figure insights into standalone factual observations. All materials were fully anonymized, stripping author names, titles, citations, and figure references.<\/p>\n

This design isolates the writing task from any specific experimental pipeline, using real accepted papers as ground truth \u2014 and it reveals something important. For Overall Paper Quality<\/strong>, the Dense idea setting substantially outperforms Sparse (43%\u201356% win rates vs. 18%\u201324%), since more precise methodology descriptions enable more rigorous section writing. But for Literature Review Quality<\/strong>, the two settings are nearly equal (Sparse: 32%\u201340%, Dense: 28%\u201339%), meaning the Literature Review Agent can autonomously identify research gaps and relevant citations without relying on detail-heavy human inputs.<\/p>\n

The Results<\/strong><\/h3>\n

In automated side-by-side (SxS) evaluations using both Gemini-3.1-Pro and GPT-5 as judge models, PaperOrchestra dominated on literature review quality, achieving absolute win margins of 88%\u201399%<\/strong> over AI baselines. For overall paper quality, it outperformed AI Scientist-v2 by 39%\u201386%<\/strong> and the Single Agent by 52%\u201388%<\/strong> across all settings.<\/p>\n

Human evaluation \u2014 conducted with 11 AI researchers across 180 paired manuscript comparisons \u2014 confirmed the automated results. PaperOrchestra achieved absolute win rate margins of 50%\u201368%<\/strong> over AI baselines in literature review quality, and 14%\u201338%<\/strong> in overall manuscript quality. It also achieved a 43% tie\/win rate against the human-written ground truth in literature synthesis \u2014 a notable result for a fully automated system.<\/p>\n

The citation coverage numbers tell a particularly clear story. AI baselines averaged only 9.75\u201314.18 citations per paper, inflating their F1 scores on the must-cite (P0) reference category while leaving \u201cgood-to-cite\u201d (P1) recall near zero. PaperOrchestra generated an average of 45.73\u201347.98 citations<\/strong>, closely mirroring the ~59 citations found in human-written papers, and improved P1 Recall by 12.59%\u201313.75%<\/strong> over the strongest baselines.<\/p>\n

Under the ScholarPeer evaluation framework, PaperOrchestra achieved simulated acceptance rates of 84% on CVPR<\/strong> and 81% on ICLR<\/strong>, compared to human-authored ground truth rates of 86% and 94% respectively. It outperformed the strongest autonomous baseline by absolute acceptance gains of 13% on CVPR and 9% on ICLR.<\/p>\n

Notably, even when PaperOrchestra generates its own figures autonomously from scratch (PlotOn mode) rather than using human-authored figures (PlotOff mode), it achieves ties or wins in 51%\u201366%<\/strong> of side-by-side matchups \u2014 despite PlotOff having an inherent information advantage since human-authored figures often embed supplementary results not present in the raw experimental logs.<\/p>\n

Key Takeaways<\/strong><\/h3>\n