Blog Archives

NASEM Report Analyzes Factors for Reproducibility and Replicability in Scientific Research

In response to a Congressional directive to conduct a comprehensive study of issues related to reproducibility and replicability of scientific research, the National Academies of Sciences, Engineering, and Medicine has released a new report, Reproducibility and Replicability in Science. The consensus study report defines the terms “reproducibility” – or getting similar results using the same research methods – and “replicability” – getting similar results across multiple research methods – as they relate to research practices. The report sets forth several steps on how to improve the reproducibility and replicability of research, including identifying clear descriptions of how the reported result was reached, providing training for scientific institutions on proper statistical analysis, investing in tools and infrastructure that support reproducibility, encouraging journals to consider reproducibility factors in publications, and having the National Science Foundation (NSF) facilitate the sharing of data for NSF-funded research. The report also considers how the public’s confidence in scientific findings can be improved by two factors: first, that scientists avoid overstating the implications of their research to public-facing audiences, and second, that journalists report on scientific results with “as much context and nuance as the medium allows.” The full report can be found on the National Academies website.

Back to this issue’s table of contents.

Tagged with: , ,
Posted in Issue 10 (May 14), Update, Volume 38 (2019)

National Academies Launches Reproducibility and Replicability Study

The National Academies of Sciences, Engineering, and Medicine held the first meeting of the study committee on Reproducibility and Replicability in Science on December 12 and 13.  The study is sponsored by the National Science Foundation (NSF) and is statutorily required by a provision in the American Innovation and Competitiveness Act. The committee will work to identify factors that affect reproducibility and replication, highlight best practices, and ascertain the extent of issues affecting reproducibility and replication. More information about project can be found here.

Back to this issue’s table of contents.

Tagged with: , ,
Posted in Issue 1 (January 9), Update, Volume 37 (2018)

NIH Issues Notice on Enhancing Reproducibility

On June 9, the National Institutes of Health (NIH) Office of Extramural Research (OER) issued a Notice (NOT-OD-15-103) in an effort to clarify and revise application instructions and review criteria “to enhance reproducibility of research findings through increased scientific rigor and transparency.”

The release of the Notice is to raise awareness and begin “culture shifts in the scientific community.” In a June 9 blog post, Sally Rockey, NIH Deputy Director for Extramural Research, and Larry Tabak, Principal Deputy Director of NIH, explained, “These changes will prompt applicants and reviewers to consider issues, which—if ignored—may impede the transparency needed to reproduce key results and thereby slow scientific progress.”

The newly revised grant applications are intended to clarify NIH’s expectations regarding rigor and transparency and how it would like to see this described in applications. The instructions will highlight the need for applicants to describe details that may have been previously overlooked, the need for reviewers to consider such details in their reviews through revised review criteria, and minimize additional burden. The new instructions and review criteria will focus on four areas: (1) the scientific premise of the proposed research; (2) rigorous experimental design for robust and unbiased results; (3) consideration of relevant biological variables; and (4) authentication of key biological and/or chemical resources.

Pending approval by the Office of Management and Budget (OMB), NIH reports that the new revised grant application instructions will be incorporated into the application guide funding opportunity announcements in Fall 2015 for applications submitted for January 2016 due dates and beyond. Meanwhile, additional detailed information regarding the specifics of implementation is expected this fall. Additional information is available at here.

Back to this issue’s table of contents.

Tagged with: , ,
Posted in Issue 11 (June 16), Update, Volume 34 (2015)

NIGMS Advisory Council Approves New Grant Mechanism, Discusses Reproducibility

At the September meeting of the National Institute of General Medical Sciences (NIGMS) Advisory Council, director Jon Lorsch provided an update on a number of issues, including the National Institutes of Health’s (NIH) data reproducibility efforts, the NIGMS strategic planning process, and an overview of the impacts of the previous NIH budget-doubling period “on the biomedical research ecosystem.” In addition, the Council approved the Institute’s concept clearance to create the new Maximizing Investigators’ Research Award (MIRA), clearing the way for NIGMS to proceed.


Lorsch noted that reproducibility is not a single issue but an issue of reproducibility of data, generalizability of conclusions, and the correctness and strength of the conclusions. According to the director, NIGMS has been leading the NIH effort around “exportable” training in this area. That effort includes training modules that will be available online, widely accessible, and free for use by any program. The topics of the modules will be in a wide range of areas and include the spectrum of issues that might impact the general area of reproducibility. Noting that the funding opportunity announcement, Training Modules to Enhance Data ReproducibilityRFA-GM-15-006, had been released, Lorsch reported that nine of the 27 NIH institutes and centers signed on and are funding additional training modules (see Update, September 8, 2014). The additional funding will allow the NIH to support up to 21 awards.

NIH Budget at a Crossroads

Pointing out that the doubling of the NIH budget, which ended a decade ago, dramatically changed the NIH ecosystem, Lorsch emphasized that the system has not recalibrated. Prior to the doubling of the NIH’s budget, the paylines were better than they are today, he stated, and emphasized the resulting paradigm shift where there are more investigators that are fully supported by the NIH. The problem is the way that science has been funded and the changes that have occurred over the past couple of decades, he explained.

According to Lorsch, the NIH could continue on the path it has been on the last few years and hope that things begin to work again. Or it could make a major course correction and seek more optimal ways of doing things. To this end, NIGMS is beginning to seek “new and more effective paths” to advance its mission. Affecting the Institute’s ability to do this is the issue of how to fund biomedical research, Lorsch explained. He pointed out that currently the primary way that the NIH funds research is by funding specific projects where it asks investigators to predict four years into the future what they will do and “define it in very precise ways.” In his view, this leads to inefficiencies which affect the ability to make progress and does not reflect the way science really works. NIGMS would like to give researchers more freedom from those restraints, which Lorsch believes will lead to better science in the end. Currently, the system also allows people to submit as many grant applications as they want and consequently spend a significant amount of their time writing new grant applications. Several of the institutes are developing similar awards (see related story).

Maximizing Investigators’ Research Award (MIRA)

It is these two issues the NIGMS would like to address and which led it to propose MIRA. The program’s goal is to enhance the willingness to take on scientifically ambitious problems, which the current funding system with its project-based emphasis discourages, causing people to become more conservative. NIGMS, Lorsch explained, would like to increase the availability and flexibility of investigators to go in other directions as the opportunities may arise. In addition, NIGMS would like to reduce the writing of grant applications. The feedback from the Institute’s request for information (RFI) was robust and the Institute received “many good ideas” for improvement, he noted.

As proposed, MIRA would be a single award in support of all NIGMS-relevant research in an investigator’s lab and would preclude receipt of other NIGMS funding except for research resources, training, workforce development, diversity building, clinical trials, SBIR/STTRs, and conference grants. The award would be a five-year renewable award that is longer and larger than NIGMS’ current average investigator-initiated (R01) award. Investigators would be expected to commit at least 50 percent research effort, excluding teaching, clinical, and administrative duties. Revision applications, however, to allow new collaborative work would be allowed. The Institute expects that in most cases renewal applications will be funded at the same level or with an increase or decrease, rather than an abrupt termination. NIGMS also expects that success rates will be higher than for R01 applications because fewer applications will be submitted per investigator.

The MIRA concept clearance was approved by the Council. For established principal investigators, it would provide $40 million in direct costs and $60 million in total costs. NIGMS could fund up to 100 awards with an average cost of approximately $400,000 in direct costs each. This assumes that all eligible established PIs compete and all of their current funding is reprogrammed to MIRA. For Early Stage Investigators (ESIs), NIGMS support for MIRA would cost $17 million in direct costs and $26 million in total costs, funding up to 75 awards at an average cost each of $250,000 in direct costs. Similarly, this assumes that all R01 funds awarded to ESIs are reprogrammed as MIRA awards.

In implementing MIRA, the Institute’s goal is to generate a “moderate” number of applications to test the application and review concepts. The pilot phase, with limited eligibility, is expected to have a neutral impact on NIGMS’ research project grant budget. The Institute plans to have separate funding opportunity announcements, review panels, and review criteria for established PIs and ESIs. The applications will be reviewed by special emphasis panels organized by NIGMS’ Office of Scientific Review. The award of MIRA will coincide with relinquishment of current NIGMS support.

New NIGMS Strategic Plan

The other overarching area that the Institute is focusing on is its general strategic direction. Accordingly, NIGMS is undergoing a strategic planning process. While the RFI seeking the extramural community’s input closed on September 26, Lorsch announced that a stakeholders meeting for professional societies associated with NIGMS is being planned for October 15th. Additional information about the process is available on NIGMS’ website.

Tagged with: , , ,
Posted in Issue 18 (October 6), Update, Volume 33 (2014)


Click here to subscribe to the COSSA Washington Update, our biweekly newsletter.


Looking for something from a previous issue of the COSSA Washington Update? Try our archive.


Browse by Month