Critics challenge NIH finding that bigger labs aren’t necessarily better
A strident debate has erupted among biomedical researchers over a proposed National Institutes of Health (NIH) policy that would shift money from richer to poorer labs. The policy—which would limit investigators to the equivalent of three NIH grants—is based largely on an agency-led analysis of lab productivity. It found that once an NIH-funded lab grows to a certain size, each additional grant produces a smaller productivity boost. But NIH’s study, and one graph in particular, has drawn widespread criticism.
Michael Lauer, NIH’s deputy director for extramural research in Bethesda, Maryland, led the study, which gathered data on grants held by 71,500 individual investigators over nearly 20 years. Because some types of research cost more than others, the analysts didn’t simply tally grant dollars; instead, they developed a point system that gives a standard R01 grant seven points, and other grants more or fewer points depending on how much of an investigator’s time they require. NIH calls this the Research Commitment Index or Grant Support Index (GSI). They then plotted an individual’s GSI score against an NIH-developed productivity measure called the Relative Citation Ratio (RCR). Instead of assessing papers by journal impact, the RCR compares an article’s citations with those of other papers in the same field.
The resulting curve (see graph, above) indicates that investigators’ productivity rises steeply as they go from holding one R01 grant (or a GSI of seven) to two and then three. However, above a GSI of 21, or three R01s, the “rate of increase decreases,” NIH says—that is, the gains per grant taper, in what economists call “diminishing marginal returns.” NIH says their unpublished findings “mirror” those of similar studies done at individual NIH institutes, in Canada, and in the United Kingdom.
Based on this analysis, NIH concluded it could set a 21-point GSI cap, and then fund more early- and midcareer investigators, “without hurting overall productivity,” Lauer wrote in a January blog post. NIH initially estimated that 6% of its 33,000 lead investigators were over the cap, and that the freed money could fund 1600 additional grants. The agency has since removed training and infrastructure grants from the GSI and reduced the points for shared grants, lowering those numbers to 3% of all investigators and 900 new grants.
Critics, many of them well-funded investigators or leaders at powerhouse research institutions, have questioned NIH’s study and its use of the RCR—a relatively new measure they say might not adequately capture a scientist’s contributions. They have also argued that it’s unwise for NIH to make such a dramatic, rigid policy move based on a study that produced average values across all investigators, potentially obscuring the contributions of unusually productive scientists. And some have argued that the correlation between lab size and productivity gains does not indicate causality. “Because [summer] ice cream sales correlate with the number of drownings doesn’t mean we should have a national policy to stop eating ice cream,” said Jonathan Epstein, executive vice dean at the University of Pennsylvania’s Perelman School of Medicine, at a recent NIH advisory board meeting.
Some also question NIH’s claim that it can shift funds from rich to poor labs without lowering the overall productivity of NIH’s investigator pool. One high-profile critique was posted on Medium by Shane Crotty, a vaccine researcher at the La Jolla Institute for Allergy and Immunology in San Diego, California. He argues that NIH’s productivity curve actually shows that productivity per grant is higher in labs with four or five grants compared with labs with just a single R01. So the cap “would clearly worsen NIH investment for productivity by creating more scientists with 1 R01,” Crotty writes.
Lauer responds that the critics have been misled by “eyeballing” NIH’s hotly debated productivity curve—which is plotted on a log scale to make the data easier to visualize. (When plotted in raw form, Lauer says the data form a “gigantic cloud” with little apparent pattern.) A separate analysis, he notes, shows that four investigators who each hold a single R01 are collectively more productive than a single investigator with four R01s. The details are in a preprint posted at bioRxiv.
Lauer notes that even if overall productivity dropped as a result of the grant cap, there are other reasons for NIH to spread its funding more widely—such as trying to boost the proportion of early- and midcareer investigators in the agency’s grantee pool, which currently skews toward older scientists. He points to studies suggesting that breakthroughs can come at any stage of a scientist’s career, and that because advances are partly a result of luck, NIH should fund as many scientists as possible. “The question,” Lauer wrote in an email, “is whether funding more scientists, along with a more diverse group of scientists, presents a greater likelihood that the enterprise will make great discoveries. Many thought leaders think so.”
If NIH never funds that extra scientist, “or we kick him/her out prematurely, a great opportunity is lost,” he wrote. “Meanwhile, the well-funded investigator is still well-funded and can continue their work.”
NIH is still collecting feedback on the policy, which it hopes to institute this fall. And it is expected to announce additional changes, including more flexibility in how it will use the GSI to make funding decisions, at a meeting tomorrow of the NIH Advisory Committee to the Director.
Additional reading: Science Editor-in-Chief Jeremy Berg examines the Lauer analysis in a 6 June post on his blog Sciencehound. And several responses to the GSI policy have been posted online, including an open letter from a group of new investigators, a letter from the University of Pennsylvania, and comments from the American Society for Biochemistry and Molecular Biology and the American Association of Immunologists.