Diagnosing Inconsistency When Reviewers Contradict Each Other

Every editor and program officer has seen it. Two reviewers read the same manuscript against the same criteria and reached conclusions that cannot both be true. Reviewer 1 describes the methodology as "rigorous and appropriate for the research questions." Reviewer 3 describes it as "fundamentally flawed in its approach to sampling." These are not two points on a continuum. They are contradictory factual claims about the same text, and averaging their scores does not produce a compromise. It produces a number that corresponds to no one's actual assessment.

The instinct in most editorial systems is to treat disagreement as noise and averaging as the correction. That instinct has theoretical justification when disagreements are random and uncorrelated, in which case averaging does cancel the noise and converge on a more accurate central estimate. But much of the disagreement we see in review panels is not random. It is systematic. Reviewers disagree because they read the rubric differently, because they brought different expertise to the table, because they read the paper at different levels of attention, or because one of them made a factual error the other did not. Averaging these sources of disagreement does not cancel them. It hides them.

What follows treats cross-reviewer inconsistency as a measurable quality signal: something to be diagnosed and classified by severity, then resolved before it contaminates the editorial or funding decision.

The Taxonomy of Disagreement

Not all disagreements are equal, and your response should differ by type. Three classes cover almost everything that arrives in a review packet:

• Threshold disagreements. Reviewers agree on what the manuscript says and disagree on whether it is good enough.
• Interpretive disagreements. Reviewers read the same text and extract different meanings from it, so they are grading different papers.
• Factual contradictions. Reviewers make incompatible claims about what the manuscript contains, which means at least one of them read it wrong.

Severity runs in that order, and so does the amount of work each one costs you.

Threshold Disagreements

Threshold disagreements arise when reviewers agree on what they observed but disagree on how to evaluate it. Reviewer A and Reviewer B both note that the study uses a sample of 150 participants. Reviewer A considers this adequate for the stated research design. Reviewer B considers it underpowered. Both have read the same text correctly. They simply apply different standards for what counts as "enough."

Threshold disagreements are the least troublesome type, because they are transparent in the review text and resolvable by your own judgment. You can read both assessments, weigh the methodological context, and decide which threshold fits the submission. No further investigation is required and no error needs correcting. The disagreement reflects genuine uncertainty about evaluative standards, which is exactly the uncertainty editorial discretion exists to resolve.

One caution: threshold disagreements sometimes wear the costume of factual ones. "The sample is too small" and "the sample is inadequate for the analysis presented" look similar and mean different things. The first is a standard applied to a number both reviewers agree on. The second may be a claim that the analysis requires more power than the design supplies, which is checkable against the text. Read the sentence closely enough to know which one you have before you file it under "difference of opinion."

Interpretive Disagreements

Interpretive disagreements arise when reviewers read the same text and extract different meanings. Reviewer A interprets the authors' theoretical framework as grounded in institutional theory. Reviewer B interprets it as a resource-based view argument. Both interpretations may be defensible, but they lead to different evaluative conclusions, because the standards for a good institutional theory paper differ from the standards for a good resource-based view paper. The disagreement is not about the quality of the work but about what the work is, and until that interpretive question is settled, the evaluations built on top of it are incommensurable.

Interpretive disagreements are harder to detect than threshold ones. Each reviewer writes as though their interpretation is obviously correct, and you may not be a specialist in the paper's specific theoretical tradition, so nothing in either review announces that the two reports are evaluating different versions of the same manuscript. The remedy is to surface the divergence explicitly and locate where in the manuscript the ambiguity starts. Then either settle it editorially or ask the authors to clarify.

Factual Contradictions

Factual contradictions are the most severe type and the most consequential for decision quality. They arise when reviewers make incompatible claims about what the manuscript contains. One reviewer states that the authors ran a sensitivity analysis, the other states that no sensitivity analysis was reported. One reviewer describes the sample as longitudinal, the other describes it as cross-sectional. These cannot both be true. At least one reviewer has misread the text or confused it with another manuscript.

Factual contradictions are not differences of opinion. They are diagnostic evidence that at least one reviewer failed to read accurately. The editorial response is not to split the difference but to determine which reading is correct.

Factual contradictions are dangerous precisely because they are usually invisible in the normal editorial workflow. You read each review in sequence and form a cumulative impression, and the clash between Review 1's claim about the sensitivity analysis and Review 3's denial of it may never register unless you are actively comparing the reviews criterion by criterion. In a high-volume editorial context, that comparison is the exception and not the rule. The same blind spot lets whole criteria go unaddressed, which is the failure mode covered in our post on reviews that miss what the rubric requires.

Why Score-Level Agreement Masks Criterion-Level Contradiction

The standard metric for inter-reviewer agreement in peer review research is the correlation (or kappa statistic) between reviewers' overall scores or recommendations. That metric captures the big picture, which is whether reviewers agree on accept or reject. It is insensitive to the criterion-level dynamics that actually drive the decision.

Two reviewers can assign the same overall score for entirely different reasons. Reviewer A gives a 3/5 because the paper has a strong contribution but weak methodology. Reviewer B gives a 3/5 because the methodology is strong but the contribution is incremental. At the score level they agree perfectly. At the criterion level they disagree on every substantive dimension. Editors who see matching scores and read consensus into them are working from a misleading signal.

This is not a hypothetical edge case. It is the routine operating condition of multi-reviewer evaluation. The criteria interact differently inside each reviewer's judgment, and the summary score is a lossy compression that throws away exactly what you need, which is why the reviewer landed on that number and whether the reasons hold up across the panel.

So consistency analysis has to operate at the criterion level. For each criterion in the framework, compare what each reviewer claimed, mark the agreements and the contradictions, then label each contradiction by type so you know which ones need investigation and which ones your own discretion can close.

Working Through One Contradiction

Take the sensitivity analysis case, since it is the most common shape a factual contradiction takes.

Reviewer 1 writes: "The authors report a sensitivity analysis excluding the outlier sites, and the main effect holds." Reviewer 3 writes: "No sensitivity analysis appears anywhere in the paper, which makes the headline effect impossible to evaluate." Those two sentences sat under scores far apart on the same criterion.

Open the manuscript and search the term. You will land in one of four states, and each one has a different response:

• The analysis is in the main text and clearly labeled. Reviewer 3 missed it. Discount Reviewer 3's assessment on that criterion, keep the rest of their review, and say so in your internal note. Their concerns about presentation may still be sound.
• The analysis is genuinely absent. Reviewer 1 asserted something that is not in the paper, which is the more worrying error, because a reviewer who supplies missing evidence from memory may be doing it elsewhere too. Reread the rest of that review with fresh suspicion.
• The analysis sits in the supplementary materials with no pointer from the main text. Neither reviewer is wrong. The manuscript is, and the revision request should require a signpost in the methods section.
• The analysis is described without ever using the term. Both reviewers read accurately and one of them searched for a label instead of a procedure. Flag it as an authorial clarity problem and note in your letter that the analysis exists but is unfindable.

Two of those four outcomes assign fault to the manuscript instead of to a reviewer, which is the part editors most often get wrong. The reflex is to decide which reviewer to believe. Often the honest answer is that the paper made the disagreement possible, and the revision request is where you fix that.

Whatever you conclude, write one line into the file: what the contradiction was, what the manuscript actually says, and how you weighted the two reviews afterward. That line is what protects the decision if the authors appeal.

Detecting Inconsistency at Scale

Manual consistency analysis, which means reading three to five reviews side by side and comparing their claims criterion by criterion, is feasible for individual manuscripts and does not scale. Editors handling 500 submissions per year, each with 2-4 reviews, would need over a thousand pairwise comparisons for comprehensive coverage. Funding agencies processing 2,000 proposals with three reviewers each face a comparison space of 6,000 reviews organized into 2,000 panels, with consistency analysis needed inside each one. That arithmetic is why the comparison quietly stops happening, and it is the same arithmetic behind the staffing problem in our post on scaling review oversight.

The computational approach mirrors the manual process and runs at a speed the manual process cannot reach. For each criterion, the system pulls the relevant claims out of each reviewer's text and aligns them, then flags the pairs that are in tension. Each flagged pair gets a severity label: minor for transparent threshold disagreements, moderate for interpretive divergence that may need clarification, major for factual contradictions that point to a misreading.

ReviewPanel.ai generates a cross-reviewer consistency report that surfaces these contradictions with severity ratings and direct textual evidence. The report does not resolve the contradictions, since that is your job or the panel chair's, but it makes them visible before the decision instead of leaving them buried in narratives nobody has time to compare. For program officers, the report doubles as documentation: if a funding decision is questioned later, the record shows that reviewer disagreements were identified and addressed instead of ignored. For the limits of what automated analysis can verify here, see what AI can and cannot do for peer review quality assurance.

What to Do With Inconsistency Once You Find It

Diagnosis without a response protocol is incomplete. Calibrate the response to the severity label.

Minor Inconsistencies (Threshold Disagreements)

Log the disagreement and note it in the editorial record, then resolve it by editorial judgment. No further action is required. Your decision letter should acknowledge the disagreement, which signals to the authors that it was identified and weighed instead of overlooked. Authors who see two contradictory reviews and no editorial guidance will guess at which one to obey, and they usually guess wrong.

Moderate Inconsistencies (Interpretive Disagreements)

Flag the interpretive divergence and identify the textual source of the ambiguity. Then decide whether the ambiguity can be resolved from the manuscript alone. If it can, resolve it yourself. If it cannot, either seek a tie-breaking review from a specialist in the relevant area or fold the interpretive question into the revision request and ask the authors to clarify the part of the manuscript that produced the divergent readings.

Major Inconsistencies (Factual Contradictions)

Verify which reviewer's factual claim is correct by checking the manuscript text. If one reviewer misread it, discount their assessment on the affected criterion and note the correction in the editorial record. If both claims turn out to be plausible, which happens with ambiguously written manuscripts, the ambiguity belongs to the authors and not to the reviewers, and it should be flagged in the revision request.

In panel contexts (funding agencies, conference review committees), major inconsistencies belong in the panel discussion with the specific contradiction and its textual evidence attached. That gives the panel chair a structured entry point for resolving the disagreement instead of hoping open discussion surfaces it organically.

Edge Cases That Break the Protocol

• The reviewer who misread is the senior expert. The protocol does not change. Seniority predicts nothing about whether someone read the supplementary materials. Record the correction in the same words you would use for anyone else.
• You have two reviewers and they contradict each other on a factual point. Resolve the fact yourself first. Recruit a third reviewer only if the contradiction survives your reading of the manuscript, since a third review commissioned to break a tie that the text already breaks wastes weeks.
• The contradiction is about something the manuscript never states. Both reviewers inferred, and they inferred differently. That is a manuscript defect and should be written into the revision request as a required clarification.
• One reviewer contradicts the other while citing their own work as the standard. Check that one before discounting anyone, because reviewer bias and honest disagreement can produce the same sentences.

The Institutional Case for Consistency Monitoring

Beyond individual decision quality, systematic consistency monitoring produces institutional benefits that compound. Patterns across a journal's reviewer pool identify reviewers who repeatedly misread manuscripts, who apply idiosyncratic standards, or who land on the wrong side of factual questions more often than their peers. No single instance is grounds for dropping someone from the pool. The pattern is still useful data for reviewer management, because editors who know that Reviewer X generates factual contradictions at twice the base rate of the pool can adjust assignment strategy accordingly.

For funding agencies, consistency data across cycles identifies evaluation criteria that different panelists systematically read differently, which is evidence that the rubric itself is ambiguous. If 30% of panels show major inconsistencies on the "broader impacts" criterion but only 5% show them on "intellectual merit," the problem is probably in the criterion definition and not in the reviewers, and the institutional response is rubric revision instead of reviewer training. That specific split is a live issue for anyone running panels under the NSF merit review criteria.

Consistency is not conformity. The goal is not to eliminate disagreement but to make disagreement visible and classified before it reaches the decision. Systems that surface contradictions and route them to the right resolution mechanism produce better decisions than systems that hide contradictions behind averaged scores and hope. Where consistency sits alongside completeness and bias in a single quality framework is laid out in our post on what makes a good peer review.

Frequently asked questions

Is reviewer disagreement a sign that peer review is broken?

No. Disagreement between qualified experts on a difficult manuscript is expected and often informative, since it tells you which parts of the work are genuinely contested. What signals a problem is disagreement that nobody diagnoses. If two reviews contradict each other on a factual point about the manuscript and the decision is made by averaging their scores, the review process has absorbed an error instead of catching one. The failure is in the handling and not in the disagreement.

How do I decide which reviewer to believe?

On factual contradictions you do not decide, you check. Open the manuscript, find the passage in question, and see which description matches. On interpretive and threshold disagreements there is nothing to check, so the question changes: which standard fits this submission, this venue, and this field's current practice? That judgment is yours to make and yours to write down. Editors get into trouble by treating checkable questions as matters of opinion.

Should I send contradictory reviews to the authors as they are?

Send them, and add a paragraph of guidance. Authors who receive two mutually exclusive reviews with no editorial steer will spend the revision guessing which reviewer holds the decision, and a wrong guess costs them a round. One or two sentences naming the contradiction and saying which line to follow removes the guesswork. You do not have to declare a reviewer wrong in public. Saying "the reviewers differ on sample adequacy, and I would like you to address the power question directly" is enough.

Does inter-rater reliability tell me whether my reviews are consistent?

Only partly. Correlation and kappa are computed on overall scores or recommendations, so they measure whether reviewers agree on the verdict. They say nothing about whether reviewers agree on the reasons. Two reviewers who reach the same score through opposite reasoning score as perfect agreement and are in fact in complete disagreement about the manuscript. Criterion-level comparison is the only method that catches that, and it is also the only method that catches factual contradictions.

What if a reviewer refuses to accept the correction?

You do not need their agreement. Reviewer input is advice and the decision is yours, so the correction goes into the editorial record whether or not the reviewer concedes. What is worth doing is telling them what the manuscript says, with the location, and doing it privately. Most reviewers who misread a paper under time pressure accept the correction quietly. Reviewers who repeatedly dispute checkable facts about manuscripts are giving you assignment information you should act on.