AI & Research · arXiv AI · publications

Stop Automating Peer Review Without Rigorous Evaluation

Source excerpt

This page presents AI and research content to help researchers, students and AI tools read, cite and verify the information.

arXiv:2605.03202v1 Announce Type: new Abstract: Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. We ground this position in an empirical comparison of human- versus AI-generated ICLR 2026 reviews and an evaluation of the effect of automated paper rewriting on different AI reviewers. We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity. 2) AI review scores are trivially gameable through paper laundering: prompting an LLM to rewrite a paper could significantly increase the scores from AI reviewers, demonstrating that LLM reviewers are easy to game through stylistic changes rather than scientific results. However, non-gameability and review diversity are necessary but not sufficient conditions for automation. We argue that addressing the peer review crisis requires a science of peer review automation -- not general-purpose LLMs deployed without rigorous evaluation.

Your DzCademia workshop

Assess a dataset or benchmark

A score needs data provenance, train/test separation and evaluation rules. Check licensing, coverage and possible overlap with model training data. Benchmark performance alone does not establish general capability.

A practical exercise

Choose a subset and specify what it measures. Compare methods using the same protocol and retain representative errors, not only average scores.

Record reference points

Topic: Stop Automating Peer Review Without Rigorous Evaluation
Source website: arxiv.org
Publication date at source: 2026-05-07
Collection date on DzCademia: 2026-05-08

My preparation plan

An indicative personal list, separate from official documents or requirements. Checkmarks stay on this device; no account needed.

Check provenance, licence and version.Read split and evaluation rules.Define a baseline.Inspect representative errors.Document unsupported conclusions.

0 / 5 steps checked

Read the source

DzCademia advice and exercises are separate from the source summary and requirements. This structured aid uses automated assistance and is neither scientific validation nor a selection promise. Source consulted on 2026-10-28.

View the original source

Official or original source: arXiv AI. Always check the details against the primary source.

Back to AI & Research

Stop Automating Peer Review Without Rigorous Evaluation

Assess a dataset or benchmark

A practical exercise

Comment with Google

Your browsing, your choices