Understanding NIH’s system of percentiling can be a challenge. My posts on how percentiles relate to paylines and success rates continue to get a lot of hits. A recent presentation by NIH’s Center for Scientific Review provided an example that I thought might be helpful to people trying to understand more about how a percentile is calculated for some grant applications.
A percentile, defined in its broadest sense, is a relative ranking of an application within a set of applications. In many cases a percentile rank is based on impact scores, and calculated against the set of all applications reviewed in the current and the preceding two review rounds. But this isn’t always the case. For example, applications reviewed by an ad hoc study section might use a different percentile base. (When percentiling is used, the summary statement you receive will identify the base that was used.)
NIH uses percentile calculations to improve our ability to compare applications across different application cycles and across different study sections. Percentiling allows NIH institutes to compare applications even when different study sections have different scoring behaviors. It’s natural for reviewers to judge an application in comparison to those immediately around it. Additionally, some study sections may have a relatively small sample of applications; combining review rounds in these cases can reduce noise and variability.
Let’s look at an example so you can see how this works. Say we have 15 applications, with impact scores ranging from a high of 10 to those that are Not Discussed (ND). These applications would be ranked in order, like so:
However, as you can see, this formula doesn’t work as is for applications with tied scores (see the highlighted cells above) so the tied application are all assigned their respective average percentile:
Not all applications are percentiled. Whether an application is percentiled depends on the grant mechanism, the institute, and the funding opportunity. For example, applications submitted in response to a request for applications (RFA) are never percentiled.