What Makes a Good Peer Review? A Framework for the Other Side

Peer review is the load-bearing wall of scientific publishing and research funding, and nearly everyone involved agrees it is cracking. Editors complain about shallow reports that arrive three weeks late. Program officers flag panelists who clearly skimmed the proposal. Authors have catalogued the dysfunction from their side for decades, and an entire cottage industry of "how to respond to Reviewer 2" advice exists because of it. Yet the institutional actors who manage review processes rarely hold a systematic framework for answering the more fundamental question: what, precisely, distinguishes a high-quality review from a low-quality one, and how would you measure the difference at scale?

That gap is not merely academic. When a journal editor reads three reports on a manuscript and has to decide whether to send it back for major revision, the decision depends on what the reviewers said and also on whether the reviewers did their jobs competently. Recommendations to reject that rest on a misreading of the methods section are not valid signal. They are noise dressed in the language of expertise. The same logic applies to funding panels. Proposals scored in the bottom quartile because one panelist fixated on a tangential weakness while ignoring the stated evaluation criteria did not deserve to fail. They received a defective review.

The difficulty is that "defective" is easy to sense and hard to operationalize. What follows is a working framework, grounded in what the empirical literature on review quality actually supports, for evaluating reviewer output along three measurable dimensions: completeness against the rubric, consistency across reviewers, freedom from systematic bias.

Completeness: Did the Reviewer Actually Address the Rubric?

The most common failure mode in peer review is not hostility, and not incompetence in the narrow technical sense. It is incompleteness. Reviewers who write three paragraphs of general impressions and a vague recommendation have not reviewed the manuscript. They have reacted to it. The distinction matters because editorial decisions downstream treat the review as though it covered the full evaluative terrain, when in practice it may have covered only the terrain the reviewer found interesting or felt qualified to assess.

Completeness, in this framework, means the degree to which a review addresses every criterion the evaluation rubric requires. For an NSF proposal that rubric is explicit: intellectual merit and broader impacts, each with defined sub-criteria. For a journal manuscript the rubric may be implicit (originality, rigor, significance, clarity) or structured through a formal reviewer form. Either way, a review that devotes 800 words to the statistical analysis and zero words to the study's limitations has not completed the assignment, however incisive those 800 words happen to be.

Reviews that are brilliant on one criterion and silent on three others are, from a decision-making standpoint, partial reviews. Partial reviews produce partial decisions.

Measuring completeness means mapping review text against rubric criteria and scoring each mapping for depth. Reviewers can technically mention "broader impacts" in a single sentence ("The broader impacts are adequate") with no evaluative substance at all, which is a surface acknowledgment that checks a box without informing a decision. Genuine completeness demands engagement with each criterion at a depth sufficient to justify the recommendation. The reviewer names specific strengths and specific weaknesses, then explains why those observations lead to the overall assessment.

That mapping exercise is tedious when performed by hand across dozens or hundreds of reviews per cycle. It is also exactly the kind of structured text-to-rubric alignment that computational tools handle well. ReviewPanel.ai scores each reviewer on each criterion along a depth scale (thorough, adequate, superficial, or missing) and returns a per-reviewer completeness profile, so gaps are visible to the editor or program officer without a full re-read. The manual alternative does not scale, and it assumes the decision-maker holds the same domain expertise as the reviewer, which is frequently untrue. The structural version of this problem, and what it costs journals, is the subject of our piece on incomplete reviews.

What "Thorough" Actually Looks Like

One practical way to calibrate completeness expectations is to define the minimum deliverable per criterion. For a journal review evaluating methodological rigor, "thorough" means the reviewer identified the specific analytical approach, evaluated its appropriateness for the research question, noted limitations or threats to validity, and connected those observations to the recommendation. "Adequate" means they noted the approach and flagged a concern without the connective reasoning. "Superficial" means a generic comment ("The methodology could be stronger") with no specifics. "Missing" means the criterion was never addressed at all.

These depth categories are not arbitrary quality tiers imposed from outside the review process. They reflect the information an editor actually needs to make a defensible decision. Editors who receive three reviews scored "thorough" across all criteria can decide with confidence that the evaluative work was done. Editors who receive three reviews with "missing" on two of five criteria for two of the three reviewers are deciding on incomplete information, whether or not they realize it.

Rubrics with explicit, published criteria make this measurable in a way implicit rubrics never will. That is one reason funder criteria are worth studying even for journal work, and why the two NSF criteria remain a useful case study in what panels systematically underweight.

Consistency: Do the Reviewers Agree on What They Observed?

Completeness tells you whether a single reviewer did the job. Consistency tells you whether the panel as a whole is producing a coherent evaluative signal. Disagreement between reviewers is not inherently a problem, since reasonable experts can look at the same evidence and reach different conclusions about its implications. What is a problem is unresolved contradiction: one reviewer calling the experimental design "rigorous and well-controlled" while another calls it "fundamentally flawed" on the same points, with no mechanism for the editor to determine which assessment the text better supports.

The peer review literature treats inter-reviewer agreement as notoriously low. The standard reference is the 2010 meta-analysis of inter-rater reliability in journal peer review by Bornmann and colleagues (PLoS ONE), which found agreement between reviewers only modestly above chance, and agreement on specific evaluative dimensions such as novelty or methodological soundness is lower still. The response to this finding has been oddly passive. Journals and funding agencies acknowledge the disagreement and then average the scores. Averaging is defensible only if the disagreements are random noise, in which case they cancel out. Averaging buries the signal if the disagreements instead reflect systematic differences in how reviewers read the rubric or applied their domain knowledge.

When two reviewers contradict each other on a factual observation (a claim about what the paper does or does not do instead of a judgment call), at least one of them is wrong. Averaging their scores does not fix the error. It launders it.

Detecting cross-reviewer inconsistency means comparing reviewer statements at the criterion level instead of comparing summary scores. Two reviewers can both assign 3 out of 5 for methodological rigor while making completely incompatible observations about what the methodology entails. Score-level agreement masks criterion-level contradiction, and criterion-level contradiction is what corrupts the downstream decision. For a fuller treatment of how to diagnose and act on these contradictions, see our post on cross-reviewer disagreement and inconsistent reviews.

Severity Matters More Than Frequency

Not all inconsistencies carry equal weight. Minor inconsistencies, such as one reviewer praising the literature review as "comprehensive" while another notes a few missing references, are differences in threshold instead of differences in observation. Major inconsistencies are different in kind. One reviewer asserting that the study uses a randomized controlled design while another identifies it as observational reflects a factual misreading by at least one party.

The editorial response should differ accordingly. Minor inconsistencies can be noted and set aside. Major inconsistencies demand resolution before a decision, either by soliciting an additional review or by the editor making an independent judgment about which reading is correct. The cost of getting this wrong is asymmetric, which is why the sorting rule matters more than the counting rule.

Bias: Is the Review Evaluating the Work or Something Else?

The third dimension is the most politically sensitive and the hardest to measure. Bias in peer review is not limited to the dramatic cases that make headlines, meaning personal vendettas, conflicts of interest, or demographic prejudice, though all of those exist and matter. The more pervasive forms are cognitive biases operating below the reviewer's conscious awareness. Reviewers anchor on the prestige of the submitting institution. They apply a halo effect from one strong section to the whole manuscript. They show severity bias toward work outside their own subfield, and they overreach past their actual expertise to render judgments they are not qualified to make.

These biases are not theoretical constructs imported from behavioral economics to make peer review sound worse than it is. Anchoring has been reported in studies of grant review panel discussion, where the first scores voiced shape where the conversation settles. Halo and horn effects are well established in the cognitive science literature and have no reason to be absent from review contexts where a single expert forms impressions across several evaluative dimensions at once. Language bias has been documented across multiple journal systems and multiple disciplines. It penalizes authors whose first language is not English for stylistic shortcomings that are not substantive ones.

What makes bias particularly dangerous in review contexts is that it masquerades as legitimate evaluation. Reviewers exhibiting confirmation bias write reports that look, on the surface, like normal critical assessment. The problem is that the assessment was predetermined by prior expectations instead of derived from the evidence in the manuscript. Detecting it means comparing what the reviewer claims against what the text actually contains, then looking for patterns across criteria that suggest the evaluation was shaped by something other than the rubric. For a taxonomy of the seven bias types most commonly encountered in reviewer reports, see our post on reviewer bias detection.

The most dangerous bias is the one that produces a well-written, confident, criterion-by-criterion review that happens to be anchored on the wrong thing. It reads like competence. It functions as distortion.

Putting the Framework to Work

The three dimensions are not independent. Incomplete reviews are harder to evaluate for bias, because there is less text to analyze, and they contribute less to consistency analysis, because they may not have addressed the criteria on which other reviewers disagree. Complete reviews with no detectable bias still require editorial intervention when they contradict every other reviewer on a key criterion. The framework operates as a system and not as a checklist.

For program officers managing panel reviews, the practical application is to build completeness scoring into the post-review workflow. Before scores are finalized, map each review against the evaluation criteria and verify that every criterion received substantive attention. Flag inconsistencies between reviewers and route them to the panel chair for resolution, well before the funding decision is made instead of after. Treat bias screening as a quality filter and not an accusation. The point is to identify reviews that may need a second look, never to impugn individual reviewers.

For journal editors the application is similar, at higher volume and with fewer institutional mechanisms for resolution. Editors receiving 500 manuscripts a year, each with two to four reviews, cannot manually perform completeness mapping, consistency analysis, and bias screening on every review set. The choice is between doing the work selectively, which introduces its own inconsistency in editorial oversight, and automating it so the same evaluative framework reaches every review set that crosses the desk. What that looks like operationally at publisher scale is covered in our piece on scaling review oversight.

The Cost of Not Measuring

The argument against systematic review evaluation is usually pragmatic: no time, no tools, no budget. The argument for it is also pragmatic and runs on a different ledger. Every funding decision made on the basis of an incomplete review may not reflect the actual merit of the proposal. Every journal decision influenced by an undetected bias may not reflect the actual quality of the science. Those errors carry real costs, from misallocated funding to delayed careers, even where the costs are diffuse and hard to attribute to any single defective review.

Measuring review quality does not guarantee better decisions. It does guarantee that the decision-maker knows what they are working with. Program officers who can see that Reviewer 3 missed two of five evaluation criteria and contradicts Reviewer 1 on a third occupy a fundamentally different epistemic position from program officers who read three narratives and trust their gestalt impression. The first is making an informed decision. The second is making a guess that feels informed.

A Scoring Sheet You Can Run on the Next Batch

Frameworks that stay abstract do not change decisions. Here is the smallest version that produces usable numbers, and it fits on one page per manuscript or proposal.

1. List the rubric criteria in the first column. Use the journal's reviewer form or the funder's published criteria verbatim. Do not invent criteria the reviewers were never told about, because scoring people against a hidden rubric produces noise and resentment in equal measure.
2. Score each reviewer on each criterion: thorough, adequate, superficial, or missing. One pass, no deliberation. Where you hesitate between two levels, take the lower one, because the point is to find gaps and not to award credit.
3. Compute a completeness percentage per reviewer. Count thorough as 1, adequate as 0.5, superficial and missing as 0, and divide by the number of criteria. The weights and the cutoffs that follow are our working heuristic and not a validated instrument, so adjust them once you have a few rounds of your own data. Anything under 60 percent is a review that should not carry decisive weight on its own.
4. Mark every factual contradiction between reviewers. Only factual ones, meaning claims about what the submission contains. Judgment differences are not contradictions and belong in a separate note.
5. Flag bias candidates with a reason. Write the trigger in five words or fewer ("scores prestige over evidence" or "outside stated expertise"). Flags without a stated trigger are opinions.
6. Decide what the set supports. Three reviews averaging above 80 percent completeness with no factual contradictions support a decision. Anything else supports either a request for clarification or an additional review.

Two numbers make this worth the effort over a full cycle. The first is the share of reviews under 60 percent completeness, which tells you whether your reviewer pool is overloaded. The second is the share of manuscripts with at least one unresolved factual contradiction, which tells you how often your decisions rest on a disagreement nobody adjudicated. Both give you something to act on in a way that a general sense of declining review quality never is.

Run the sheet on twenty review sets from the last cycle before you change any policy. The distribution usually surprises people, and it is far easier to argue for a reviewer-training program or a structured review form with a number attached.

Frequently asked questions

What is the difference between a good peer review and a harsh one?

Tone and quality are independent. Blunt reviews that identify a specific methodological problem, locate it in the text, and explain why it matters are high quality regardless of how they read. Polite reviews that praise the paper for three paragraphs and never engage the rubric are low quality regardless of how they read. When you evaluate review quality, strip the adjectives out and ask what evaluative work the report actually performed against each criterion.

How do you measure peer review quality objectively?

Score each review against the same rubric the reviewers were given, criterion by criterion, on a fixed depth scale such as thorough, adequate, superficial, or missing. Then compare reviewers on the same criterion to find factual contradictions, and flag reviews whose reasoning tracks something other than the rubric. None of the three steps requires you to agree with the reviewer's conclusion. That independence is what makes the measurement usable across a whole cycle.

Is low agreement between reviewers a sign of bad reviewing?

Not by itself. Reasonable experts disagree about significance and novelty all the time, and a panel where everyone agrees may simply be selecting for similar backgrounds. What signals a problem is disagreement about facts, such as which design the study used or whether an analysis appears in the paper. Track factual contradictions separately from judgment differences, because the first calls for resolution before a decision and the second is ordinary scholarly variation.

How many criteria should a review rubric have?

Between four and seven works for most journals and panels. Fewer than four and the scoring collapses into an overall impression, which is what the rubric was meant to prevent. More than seven and reviewers start treating the later criteria as a checklist to be cleared, which shows up as one-sentence answers with no evaluative content. Whatever the number, publish the criteria to reviewers and to authors, since criteria that only the editor knows cannot be enforced.

Can AI evaluate the quality of a peer review?

Yes, for completeness mapping and contradiction detection, with one caveat: the evidence for running it at scale is still mostly practitioner experience and vendor evaluation, ours included. Those tasks are structured comparisons between two texts, which is what language models do well. Judging whether the reviewer's scientific conclusion is correct is a different task, and it stays with human editors. The distinction between the two, and where the boundary sits in practice, is the subject of our piece on AI for peer review quality assurance.