Harvard Professors Find AI Peer Reviews Beat Human Ones
Prepared with partner assistance
Two Harvard professors set out to defend human peer review and ended up showing that AI wrote better reviews. Their finding puts the future of their own review service in doubt.
The service and the test
Gary King directs Harvard's Institute for Quantitative Social Science. Steven Worthington is its director of data science. They run rapidPeer, a double-blind review service for university faculty, started in 2019 and now used at several other institutions. It gives quality reviews of paper drafts before they go to a journal. Reviewers are paid, which is unusual in academia, and the pair say editors from major journals have called the reviews among the best they have seen.
The pair took 134 human reviews of 124 papers submitted between 2019 and 2026. For each, they produced an equivalent review with Anthropic's Claude Opus. The papers ranged from statistical analyses of empirical data to purely qualitative studies.
A different model, OpenAI's GPT-5.5, acted as AI judge. Three former editors of leading journals also ranked 10 randomly chosen pairs. All judges used rapidPeer's criteria.
What the results showed
The AI review won in all pairs when another AI judged. It won in 87% when the editors judged. The paper, which has not itself been peer reviewed, says: "No matter how we look at the data, AI wins." The AI also had a higher mean rating in all seven areas assessed, including methods, sources and figures.
King said he expected the opposite. His team wrote the paper hoping to justify human review, and he said rapidPeer will have to pivot.
Gains, risks and policy ideas
King sees enormous room to speed up the process. "Imagine a review process that takes a week and a half," he said.
AI also removes what he calls sycophancy, since human reviewers and editors try not to offend each other. He and Worthington also name greater accuracy and less bias as benefits.
The risks are real, King said. He asked whether OpenAI and Anthropic should decide what counts as good science. The paper warns that AI may make science more homogenous, since models lean toward conventional and majority ideas.
The most realistic policy change they cite is for some journals to skip routine human reviews. An editor would instead weigh a dossier from the author with the paper, an AI review from the journal's official prompt and the author's reply. A new AI review from private prompts could help. A human review would come only if needed.
The authors end on a hopeful note for editors. With the right policies, they write, the editor's job could improve and draw more interest from the high-quality people the scientific community attracts.
Source: Inside Higher Ed