🔍 Read the full analysis: Can An LLM Tell If Your Research Idea Is New? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models gave inconsistent judgments about whether scientific ideas were novel, with a few prompt-word changes shifting accuracy by more than 50 percentage points in one test. Retrieval and additional reasoning did not consistently improve results. The team released a benchmark, data, code and an evaluation pipeline, while cautioning that some benchmark labels are imperfect.
As detailed in the original analysis, Ai2 researchers report that six large language models gave unstable assessments of scientific idea novelty, with small prompt changes shifting accuracy by more than 50 percentage points in one pairwise comparison test. The study questions whether a model’s score alone can reliably screen research proposals, and the team has released a benchmark, data, code and an automated evaluation pipeline for further testing.
The researchers tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. They used both absolute ratings, in which a model judges an idea on its own, and pairwise comparisons, in which it chooses between two ideas. According to the study, changing a few words in a prompt produced a greater-than-50-percentage-point accuracy shift in one pairwise experiment. The source material does not give the exact prompt variants or a complete breakdown of results for every model and setup.
For one benchmark set, the team used ICLR 2026 peer reviews to identify ideas with explicit reviewer comments about originality. The high-novelty group includes 154 ideas from accepted papers in the top-rated 10% of their research areas, selected when reviewers praised novelty and no reviewer disputed it. The low-novelty group includes 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised novelty. An annotator agreed with the direction of the extracted novelty signal in 47 of 50 checked cases; the authors say the three disagreements did not change the review classifications.
A second test setting paired those 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated generated proposals as less novel, but describe those labels as weak because each idea was not individually verified. In a separate expert check of 30 pairs—selected heavily from examples the strongest judge misclassified—the expert favored the human-authored idea in every pair and judged none of the generated ideas novel. The study also reports that literature retrieval and additional reasoning did not consistently improve judgments, while two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Prompt Sensitivity Matters
Research groups may use language models to filter candidate ideas before committing time to literature searches, experiments or expert review. If a judgment changes substantially when a prompt is lightly edited, a score may depend partly on the evaluation setup rather than solely on the idea’s originality. That makes scores harder to compare across projects and could cause teams to prioritize or discard proposals based on an unstable measure.
The findings also bear on efforts to automate scientific discovery. Systems that generate hypotheses or summarize research need a credible way to assess whether a proposal adds something new. The study does not establish that every LLM novelty assessment is unreliable, but it indicates that a good result on one selected prompt or benchmark is not by itself evidence of dependable judgment. Researchers building such tools may need to test several prompts, formats and models and report how much their results vary.
Products Worth Considering
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Sonos Era 100 Wireless Smart Speaker, Black
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Portable Waterproof Speaker, Black
As an affiliate, we earn on qualifying purchases.
How Ai2 Tested Novelty Judgments
A novelty judge is a model asked to assess whether an idea is new relative to existing research, either by rating it directly or comparing it with another proposal. Ai2 says there is no standard evaluation recipe: researchers can vary the model, prompt, comparison format, reasoning budget and access to literature. The study was designed to examine whether conclusions hold up when these choices change.
The authors argue that evaluations often use human-written papers even when a tool is intended to assess AI-generated ideas. Their benchmark therefore includes both reviewer-backed human ideas and a separate set of generated proposals. They caution against treating acceptance or rejection as a direct measure of novelty, since papers can succeed or fail for reasons other than originality. The benchmark’s review-based selection criteria provide evidence of how reviewers viewed the work, not a definitive measure of whether an idea was genuinely new.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels have limits. Reviewer comments and paper rankings act as signals of relative novelty, not ground-truth measures of originality. The generated-idea set has a more direct caveat: researchers assumed those proposals were less novel than the selected human ideas, but did not verify each label individually. The 30-pair expert check is also a limited sample and was weighted toward cases the strongest judge had got wrong; it cannot establish performance across the full benchmark.
The supplied study description does not provide full results for every model, prompt, reasoning budget or evaluation format. It also does not show whether the reported instability carries over to other academic fields, different idea generators or systems with broader access to research literature. The authors say some conclusions rely on disagreement between evaluation setups rather than benchmark labels, but the material does not detail how widely that result holds. No specific follow-up study or release timeline is given.
Further Tests Across Research Fields
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help other researchers build and test evaluation sets. The next step is independent testing across fields, sources of research ideas and judge designs, including whether any combination of prompts, retrieval and model methods can make judgments more consistent.
The release description does not announce a scheduled follow-up or claim that the benchmark settles how novelty should be measured. Until more tests are available, the study offers researchers a way to compare evaluation choices while leaving open whether LLMs can provide dependable novelty screening under a carefully specified setup.
Key Questions
What did the Ai2 study find?
Ai2 researchers report that six language models’ novelty judgments varied across evaluation setups. In one pairwise experiment, small prompt-word changes shifted accuracy by more than 50 percentage points.
Did adding literature retrieval make the models more reliable?
The authors report that literature retrieval did not consistently improve judgments. The supplied material does not give a complete model-by-model account of retrieval results.
How did the researchers label ideas as novel or not novel?
One set was selected using ICLR 2026 peer-review comments and paper rankings. A second set compared human-authored ideas with generated proposals, which the researchers treated as less novel but described as weakly labeled because each proposal was not individually verified.
Does the study show that LLMs cannot judge research novelty?
No. It reports instability across tested prompts and setups, not that every novelty judgment fails. The authors’ results do not establish how the findings transfer to all fields or evaluation systems.
What has Ai2 made available?
Ai2 released the study, benchmark data, code and an automated pipeline for building additional evaluation sets. The supplied material does not specify a date for the release or a timetable for further work.
Primary source: Hugging Face · via ThorstenMeyerAI.com
