There is a particular kind of peer review that editors learn to dread without ever quite naming the problem. The review arrives on time. It is polite. It demonstrates familiarity with the literature. It identifies a genuine weakness in the manuscript. And it says absolutely nothing about three of the five criteria the reviewer was asked to evaluate. The review is not incompetent — it is incomplete, and the incompleteness is more damaging than any single error it might have contained, because the editor must now make a decision on partial information while the review's apparent professionalism obscures the fact that the information is partial.
This failure mode — substantive engagement on some criteria, silence on others — is the single most common quality deficiency in peer review, more prevalent than outright bias, more frequent than factual error, and far more likely to escape detection than either. It escapes detection because editors read reviews as narratives, not as rubric mappings. A well-written narrative that covers two criteria in depth feels thorough. It takes a deliberate, criterion-by-criterion comparison against the evaluation framework to reveal that the other three criteria received no substantive treatment at all.
The costs of this failure are not symmetric. A review that is complete but harsh gives the editor the information needed to push back or contextualize the harshness. A review that is complete but lenient gives the editor the information needed to probe where the leniency might be misplaced. But a review that is incomplete gives the editor nothing on the missing dimensions — not a wrong answer, but no answer at all. And no answer, in a decision system that aggregates across reviewers, silently becomes a non-objection, which silently becomes an implicit endorsement.
Why Incompleteness Is the Default
Understanding why reviews are so often incomplete requires acknowledging the conditions under which they are produced. Reviewers are unpaid volunteers (in most systems) operating under time pressure, with competing demands on their attention, reviewing work that may fall partially outside their expertise. The rational reviewer strategy under these constraints is to focus on the aspects of the manuscript they feel most qualified to evaluate and to address the remaining criteria with either generic language or silence.
This strategy is individually rational but institutionally corrosive. The reviewer has done what they felt they could do well, and from their perspective, the review is honest — they would rather say nothing about a criterion than say something uninformed. But the editor's decision calculus assumes that each reviewer has addressed the full rubric, and when three reviewers each address a different subset of the criteria with no overlap on the criteria they skipped, the editor is left with a patchwork assessment that covers the evaluative space only by accident.
The reviewer thinks they are being intellectually honest by staying silent on what they do not know. The editor interprets the silence as implicit assent. Neither party is acting in bad faith, and the decision is compromised anyway.
The structural incentives reinforce this pattern. Most review invitation letters ask the reviewer to "evaluate the manuscript" in general terms and provide a rubric, but they do not verify post hoc that each rubric criterion received substantive attention. The reviewer form may include fields for each criterion, but a single-sentence entry ("This aspect is satisfactory") satisfies the form without satisfying the evaluative requirement. There is no penalty for incompleteness, no feedback loop that alerts the reviewer to the gap, and no systematic mechanism for the editor to detect the gap before integrating the review into a decision.
Measuring Completeness: The Rubric-Mapping Approach
If the problem is that reviews address the rubric selectively, the measurement approach is to map review text against rubric criteria explicitly and score each mapping for depth. This is not a subjective quality judgment — it is a structural audit that asks, for each criterion in the evaluation framework, whether the review contains text that substantively addresses that criterion.
The mapping produces a per-reviewer, per-criterion depth rating. A rating of "thorough" means the reviewer identified specific elements of the manuscript relevant to that criterion, evaluated their quality, and connected the evaluation to the overall recommendation. "Adequate" means the reviewer noted something relevant to the criterion but without the full evaluative chain. "Superficial" means the reviewer made a generic or boilerplate comment that could apply to any manuscript. And "missing" means the criterion was not addressed at all.
The Depth Scale in Practice
The distinction between "adequate" and "superficial" is where the practical value of completeness scoring concentrates, because superficial comments are the mechanism by which reviewers satisfy formal requirements without providing evaluative substance. Consider a review of a proposal that requires assessment of "research design and methodology." A superficial response might read: "The research design is generally appropriate for the proposed study." This sentence occupies space in the review, fills the relevant field in the reviewer form, and contributes zero information to the editor's decision — it does not identify what the research design is, why it is appropriate, or what its limitations might be.
An adequate response might read: "The authors propose a mixed-methods design combining survey data with semi-structured interviews, which is reasonable for the stated research questions. However, the sampling strategy is not fully described." This provides enough information for the editor to understand what the reviewer evaluated and what concern remains, even though it does not fully develop the evaluative argument.
A thorough response would develop the adequate response further by evaluating the sampling concern in context, considering whether the limitation is fatal or addressable, and connecting the assessment to the overall merit of the proposal. The difference between these levels is not one of opinion but of information density — how much does the editor learn from each version?
The Aggregation Problem
Completeness matters most when it is unevenly distributed across a review panel. If all three reviewers are incomplete on the same criterion, the editor has no evaluative input on that dimension and is making a decision that is, in effect, uninformed on that axis. If each reviewer is incomplete on different criteria but thorough on others, the coverage is accidental rather than designed — and the editor has no way to know, without a completeness audit, whether the collective coverage is adequate.
This aggregation problem is particularly acute for funding agencies, where panel discussions are supposed to resolve exactly these kinds of gaps. In theory, the panel discussion allows panelists to surface concerns that individual reviews overlooked. In practice, panel discussions are time-constrained, dominated by the panelists who wrote the most detailed reviews, and tend to focus on the controversial proposals rather than on verifying whether each proposal received a complete evaluation. A proposal that received three superficially positive but individually incomplete reviews may sail through the discussion without anyone noticing that the collective assessment rests on a narrower evaluative base than the rubric requires.
The most dangerous aggregation failure is three incomplete reviews that, stacked together, appear to cover the rubric because each mentions different criteria — but none addresses any criterion at the depth required for a defensible decision.
For journal editors managing large submission volumes, the aggregation problem is compounded by the practical impossibility of performing this analysis manually. An editor receiving twenty manuscripts per month, each with three reviews, would need to perform sixty rubric-mapping exercises per month to verify completeness — a workload that is obviously unsustainable and that, in practice, simply does not happen. The result is that completeness is never systematically assessed, gaps are never detected, and decisions are made on incomplete evaluations without anyone involved being aware of the deficiency.
What Editors and Program Officers Can Do
The interventions available fall into two categories: preventive (reducing the frequency of incomplete reviews before they are submitted) and diagnostic (identifying incompleteness in submitted reviews before decisions are made).
Prevention: Setting Explicit Expectations
The simplest preventive intervention is to make completeness expectations explicit in the review invitation. Rather than asking the reviewer to "evaluate the manuscript" and providing a rubric as reference, the invitation should state clearly that each criterion must be addressed with specific observations from the manuscript, that generic or boilerplate comments do not satisfy the evaluative requirement, and that silence on a criterion will be interpreted as inability to evaluate (not as endorsement). These are not onerous requirements — they are a restatement of what the review is supposed to accomplish. But making them explicit shifts the default from "cover what you can" to "cover everything or flag what you cannot."
A complementary intervention is structured review forms that require minimum response lengths per criterion or that prompt the reviewer for specific types of observations. Asking "What is the specific methodology employed?" before asking "Is the methodology appropriate?" forces the reviewer to demonstrate engagement with the text rather than rendering a judgment from summary impression alone. These structured prompts do not guarantee thorough reviews, but they make superficial reviews harder to produce.
Diagnosis: Completeness Scoring After Submission
Preventive measures reduce the problem but do not eliminate it, which is why post-submission completeness scoring is necessary as a second line of defense. The workflow is straightforward: after reviews are submitted and before the editorial decision is made, each review is mapped against the rubric and scored for per-criterion depth. Reviews with "missing" or "superficial" ratings on criteria that are central to the decision are flagged, and the editor either solicits a revision from the reviewer, requests an additional review to cover the gap, or adjusts the weight given to that reviewer's recommendation.
ReviewPanel.ai performs this scoring automatically, accepting the submitted reviews and the applicable rubric (either freeform or a custom rubric with user-defined criteria and weights) and producing a per-reviewer completeness profile within minutes. The output identifies exactly which criteria each reviewer addressed at which depth, making it possible for the editor to see, at a glance, where the evaluative coverage has gaps — and to act on those gaps before the decision is locked in. For a broader treatment of how completeness scoring integrates with consistency analysis and bias detection into a unified review quality framework, see our post on evaluating peer review quality.
The Completeness Threshold
Not every review needs to be thorough on every criterion. A realistic completeness threshold depends on the decision context: for a journal manuscript with three reviewers and five criteria, the minimum acceptable coverage might be that each criterion is addressed at "adequate" or better by at least two of three reviewers. For a funding proposal where the stakes are higher and the evaluation criteria are fewer, the threshold might be that every criterion is addressed at "thorough" by every panelist.
The point of setting explicit thresholds is not to bureaucratize the review process but to make the decision-maker's informational requirements transparent and verifiable. Without a threshold, completeness is assessed impressionistically — which means it is usually not assessed at all. With a threshold, incompleteness becomes a structural deficiency that triggers a defined editorial response (request more information, solicit another review, flag for panel discussion) rather than an ambient quality problem that everyone acknowledges and nobody addresses.
The completeness problem in peer review is not a problem of reviewer quality in the individual sense. Most reviewers are competent experts who produce substantive evaluations of the criteria they choose to address. The problem is structural: the system does not verify that the criteria each reviewer chose to address are the criteria the decision requires, and by the time the gap is discovered (if it is discovered at all), the decision has already been made. Closing this gap does not require better reviewers. It requires better measurement of what the existing reviewers actually delivered.