What Panels Actually Get Wrong About the NSF Merit Review Criteria

The National Science Foundation's merit review system rests on two criteria, Intellectual Merit and Broader Impacts, that are among the most clearly articulated evaluation standards in research funding. Every panelist receives explicit guidance on both. Every proposal is required to address both. Every review is supposed to evaluate both. Cycle after cycle, the same asymmetry shows up anyway: Intellectual Merit receives substantive, detailed evaluation from most panelists, while Broader Impacts receives a few sentences of generic commentary that check the procedural box without giving a program officer anything to differentiate proposals with.

This asymmetry is not a calibration issue. It is a structural failure that produces funding decisions based on one criterion instead of two. When three panelists each write 500 words on Intellectual Merit and 50 words on Broader Impacts, the panel discussion draws on those written reviews as its raw material and gravitates toward the criterion that was evaluated in depth. The program officer, holding detailed assessments of merit and thin assessments of impact, then has two bad options. Make the Broader Impacts judgment independently, which bypasses the panel process. Or treat the thin assessments as though they carried more information than they do.

Neither outcome is what the system intends, and both are avoidable if the completeness of panel reviews is measured before the evaluative conversation starts.

The Broader Impacts Problem Is a Completeness Problem

Undertreatment of Broader Impacts usually gets blamed on definition. Panelists do not know what "broader impacts" means, or the definition is too vague to evaluate seriously. That diagnosis is partly correct and mostly misleading. The NSF has refined the criterion repeatedly, provided examples, and published guidance documents. The problem is not access to a definition. The problem is that the evaluation infrastructure never enforces engagement with the criterion at a depth matching its stated importance.

Panelists who write "The broader impacts are adequate and include outreach to underrepresented groups" have technically addressed the criterion. The assessment carries no evaluative substance. It does not describe what the proposed outreach involves, whether the approach is realistic, whether the PI has a track record of successful outreach, or how the proposed activities connect to the scientific work. Program officers reading that comment cannot tell whether the reviewer spent thirty seconds or thirty minutes on broader impacts, and in most cases the answer is closer to the former.

The Broader Impacts criterion is not underspecified. It is underenforced. Panelists know what it asks. They simply know, from experience, that superficial treatment carries no penalty.

The remedy is measurement. Map every panel review against both criteria and score it for depth on the thorough/adequate/superficial/missing scale described in our post on the completeness problem, and the asymmetry becomes visible before the panel meets. Program officers who can see that Reviewer 2 scored "thorough" on Intellectual Merit and "superficial" on Broader Impacts have real options: request a revision to the review, shift weight to the other reviewers' Broader Impacts assessments, or put the gap on the agenda for discussion.

What the Difference Looks Like in Two Paragraphs

The superficial version, which is what most panels produce, reads "The broader impacts are adequate. The PI proposes outreach to underrepresented groups and integration of the research into undergraduate teaching. These activities are appropriate for a project of this scope."

The adequate version, which is what the criterion asks for, reads "The PI proposes a summer research program for undergraduates recruited through regional partner institutions, with recruitment handled by the partners and mentoring by the PI's graduate students. The partnership is already established and the letters of collaboration are specific about numbers and responsibilities. The undergraduate teaching component is weaker: the proposal states that findings will be integrated into an existing course, without naming the course, the material, or who would do the integration. Recommend that the teaching component be developed or dropped in favor of strengthening the summer program."

The second version takes a panelist a few extra minutes. It gives the program officer something to act on, it gives the PI something to fix, and it survives contact with an appeal. Nothing about the first version does any of that.

Usable Broader Impacts assessments name four things. What the PI actually proposes to do. Whether the plan is executable with the resources requested. Whether there is evidence the PI can do it. How the activity connects to the science being funded. Reviews missing any of the four should come back as incomplete before the panel convenes.

Intellectual Merit: Where the Problems Are Subtler

The Broader Impacts asymmetry is the most visible failure in NSF panel review. Intellectual Merit evaluation is not immune to quality problems either. They are harder to see, because the sheer volume of text devoted to merit creates an illusion of thoroughness.

The most common failure mode in Intellectual Merit evaluation is selective depth. The reviewer engages deeply with the aspect of merit closest to their own expertise and handles the rest with generic approval or silence. Reviewers who specialize in computational methods write three paragraphs evaluating the proposed algorithms and one sentence noting that "the theoretical motivation is sound," which is a judgment that may or may not be well founded and that gives the program officer no basis for testing it.

Selective depth is a variant of the expertise overreach problem. The reviewer's detailed treatment of one aspect is excellent, and the cursory treatment of everything else produces a false impression of comprehensive evaluation. The review looks thorough because it is long and technically detailed, even though the technical detail sits on a subset of the evaluative space.

Criteria conflation is the second failure mode. The NSF defines Intellectual Merit through several elements: the potential to advance knowledge, the qualifications of the investigators, the soundness of the approach, the adequacy of resources, and the integration of research and education. Those are distinct evaluative dimensions, and proposals can be strong on some and weak on others. Panelists frequently collapse them into a single global impression ("this is an excellent research program") that hides which dimensions were evaluated favorably and which were never evaluated at all.

What Program Officers Should Look For

Before the discussion, check three things in each review:

• Depth on both criteria. Not the score, but enough evaluative reasoning to understand why that score was assigned. If you cannot reconstruct the reasoning, you cannot defend the decision built on it.
• Coverage across the merit elements. Does the merit assessment span the full set of elements, or does it concentrate on the subset matching the reviewer's own expertise?
• Text-to-score coherence. Reviewers who write three paragraphs of criticism and then assign a score of "Excellent" may be showing leniency bias, or may have written the text and the score at different times and never reconciled them. Either way you need to know before the score enters the average.

None of these checks requires you to second-guess the science. They are structural questions about whether the review does what a review is for, which is the distinction our post on what makes a good peer review develops in full.

The Panel Discussion Gap

Even well-written individual reviews can fail to produce a well-functioning panel if the discussion never resolves the disagreements and gaps the reviews contain. Panel discussion is the mechanism that is supposed to catch incomplete reviews and surface contradictions, then produce a consensus assessment more reliable than any single review. In practice, panel discussions run under a clock, typically 15–30 minutes per proposal in large panels. They get dominated by the most vocal panelists and drift toward the proposals where disagreement is most visible, which are not the proposals where quality problems are most consequential.

Proposals with three uniformly positive reviews sail through discussion in minutes, regardless of how complete those reviews are. The discussion mechanism engages only when disagreement is apparent, and apparent disagreement is driven by score divergence and not by completeness divergence. Three incomplete reviews that happen to agree on the subset of criteria they evaluated will generate less discussion than two complete reviews with a genuine threshold disagreement on one criterion, even though the first case is the worse evaluative outcome. Sorting those two situations apart is exactly the diagnostic work described in our post on cross-reviewer disagreement.

Panel discussions are designed to resolve disagreement. They are not designed to detect absence. Proposals that received three enthusiastic but incomplete reviews will be discussed less than one with a genuine split, and the decision will rest on less information.

So completeness analysis has to precede the panel discussion instead of emerging from it. Panel chairs who can see, before discussion opens, that Proposal X has a Broader Impacts gap across all three reviewers can steer the conversation to that gap in the first minute. Without that information the discussion follows the path of least resistance. The panelists who wrote the most detailed reviews drive the conversation, the underevaluated criteria get even less attention than they got in writing, and the program officer makes a decision resting on a partial evaluation without knowing it.

Calibration Across Panels

Consistency across panels inside the same program is a less discussed quality dimension and an equally important one. Different panels evaluating proposals in the same competition under the same criteria can apply those criteria at different thresholds, which makes a proposal's fate depend partly on which panel it landed in instead of on its merit.

That variation is not random. Panels are composed of different individuals with different disciplinary norms and different baseline expectations about what a sound approach looks like. Panels drawn primarily from experimentalists apply different standards for "soundness of the approach" than panels drawn primarily from theorists, even inside one program. The variation is not a sign of incompetence. It is what happens when evaluation bodies are assembled from diverse experts whose norms differ in ways the rubric does not resolve.

Measuring the variation requires cross-panel consistency analysis. Compare the score distributions, the depth of criterion-level evaluation, and the rate of identified quality problems across panels in the same program. If Panel A consistently produces more thorough Broader Impacts evaluations than Panel B, the difference traces to panel composition and norms and not to differences in proposal quality, and the program officer should weight the two panels' assessments accordingly.

This kind of analysis is beyond what any individual program officer can do by hand. It is tractable as an automated workflow: run each panel's reviews through completeness scoring and consistency analysis, then compare the distributions and flag panels whose quality profiles diverge sharply from the program mean. The findings inform the current cycle by identifying decisions resting on lower-quality evaluations, and they inform future panel composition by showing which reviewer characteristics go with more complete evaluations. What such analysis can and cannot establish is covered in our post on AI for peer review quality assurance.

Toward Measurable Review Standards

The NSF merit review system is by design one of the most structured evaluation frameworks in research funding. The criteria are explicit, the guidance is detailed, and the process has built-in mechanisms for catching evaluation failures. What the system lacks is measurement. Nobody systematically assesses, criterion by criterion, whether the reviews panelists produce actually satisfy the evaluative requirements the criteria define.

Adding measurement changes none of that. It does not rewrite the criteria, replace the panel process, or shrink the role of expert judgment. It makes the quality of that judgment visible and checkable, so program officers can find evaluation gaps before they become funding errors, panel chairs can steer discussion toward the dimensions that need it, and the agency can track evaluation quality across panels and cycles as an institutional metric instead of a matter of individual impression.

The tools exist. Completeness scoring, cross-reviewer consistency analysis, plus bias screening apply to NSF panel reviews as readily as to any structured evaluation context. The rubric is well defined (Intellectual Merit and Broader Impacts, with specified sub-criteria). The reviews are text-based and criterion-referenced. The workflow has an obvious integration point between review submission and panel discussion, where quality findings can inform the process without disrupting it.

What remains is the institutional decision to measure. That panel reviews are frequently incomplete on Broader Impacts and selectively deep on Intellectual Merit is neither new nor contested inside the agency. What is new is the ability to detect the problem at the speed the system requires, in time to improve the decision instead of documenting the deficiency afterward.

Until agencies measure it, the cost lands on applicants, who receive a panel summary shaped by these gaps and get one resubmission to answer it. Our post on writing the response to a funding panel works the problem from that side.

Frequently asked questions

Why do NSF panels underrate Broader Impacts?

Because nothing penalizes a thin Broader Impacts review. Panelists are experts in the science, they are volunteering under time pressure, and they know from experience that a single sentence on impacts satisfies the form. The criterion is not vague. The NSF has published examples and guidance repeatedly. What is missing is any check that the written assessment contains enough substance to support a judgment. When the check does not exist, the effort goes where the reviewer's expertise and interest already are.

What does a complete Broader Impacts review contain?

Four things. What the PI actually proposes to do, stated concretely enough that someone else could picture it. Whether the plan can be executed with the resources requested. Whether there is evidence the PI or the team can do it, usually from prior activity. How the activity connects to the science being funded. Assessments that hit all four give the program officer a basis for a judgment and give the PI something specific to strengthen. Assessments that hit none are procedural noise.

Can a panel chair fix incomplete reviews during the discussion?

Partly, and only with advance notice. Panel discussion runs on a clock and engages where scores diverge, so a proposal with three positive but incomplete reviews attracts the least attention. Chairs who receive a completeness profile before the meeting can open discussion on the gap directly, which takes a minute and produces a record. Chairs who discover the gap live are choosing between spending discussion time they do not have and letting the decision go forward on partial information.

Do the same problems appear at NIH?

The failure mode transfers even though the criteria differ. Any review system with multiple criteria and unequal reviewer interest across them will concentrate depth where the reviewers are most comfortable. What changes is which criterion absorbs the attention and how the scoring structure amplifies the gap. Our post on NIH study sections after the 2025 simplification works through how a restructured criteria set redistributes that attention.

Is cross-panel score variation a problem worth correcting?

Some of it is signal and some is noise, and telling them apart requires measurement you probably do not have yet. Panels assembled from different disciplinary subcommunities will legitimately apply different standards for a sound approach. That becomes a fairness problem when it means a proposal's outcome depends on panel assignment. Comparing depth of criterion-level evaluation and rates of flagged quality problems across panels in one program separates a real difference in standards from a difference in how carefully the reviews were written.