Rigor & Reproducibility: How the standards of health science research are changing

Hindsight is 20-20, right? Sure. But in the realm of health science research, where funding is highly competitive and human subject participation often has potential to influence the health of individual volunteers as well as implementation of findings in patients, the opportunity to work retrospectively simply is not there. Of course, all researcher strive to remove potential for confounding variables or outliers, but is there room for improvement? Always.

To aid researchers, the National Institutes of Health has created a new set of application standards that challenges researchers to consider with greater depth the intricacies of the research planning process. The goal is simple: to create scientific research that is both rigorous and transparent. To do so, the NIH has refined application instructions that will focus on four crucial areas of research study planning: (1) scientific premise of the proposed research; (2) rigor of experimental design; (3) consideration of relevant biological variables such as sex; and (4) authentication of biological and chemical resources.

NIH plans to enhance reproducibility in multiple ways, by promoting greater scientific rigor and transparency in funding applications and publications, encouraging robust peer review, providing adequate training, removing perverse incentives, and emphasizing overlooked areas such as the consideration of both sexes in research and the use of authenticated cell lines.

Implementation of Rigor & Reproducibility into your Research Grants

These changes will take effect for most research grant applications submitted with due dates on or after January 25, 2016. For research contracts, this policy will be effective for proposals received on/after January 25, 2016 and expected to result in contract awards in Fiscal Year 2017 and beyond. The notice can be found here. Updates include:

Research Strategy Guidance

For the Significance and Approach sections of the Research Strategy, in addition to the existing instructions:

Significance: Describe the scientific premise for the proposed project, including consideration of the strengths and weaknesses of published research or preliminary data crucial to the support of your application. 

Approach: Describe the experimental design and methods proposed and how they will achieve robust and unbiased results. Explain how relevant biological variables, such as sex, are factored into research designs and analyses for studies in vertebrate animals and humans. For example, strong justification from the scientific literature, preliminary data, or other relevant considerations, must be provided for applications proposing to study only one sex. Refer to NOT-OD-15-102 for further consideration of NIH expectations about sex as a biological variable.

New Authentication of Key Biological and/or Chemical Resources Attachment

Grant applications for the activity codes covered by the policy must include a new PDF attachment related to the authentication of key biological and/or chemical resources. Briefly describe methods to ensure the identity and validity of key biological and/or chemical resources used in the proposed studies. Key biological and/or chemical resources may or may not be generated with NIH funds and: 

1) may differ from laboratory to laboratory or over time;
2) may have qualities and/or qualifications that could influence the research data; and 
3) are integral to the proposed research.

These include, but are not limited to, cell lines, specialty chemicals, antibodies, and other biologics. Standard laboratory reagents that are not expected to vary do not need to be included in the plan. Examples are buffers and other common biologicals or chemicals.

Reviewers will assess the information provided in this Section. Any reviewer questions associated with key biological and/or chemical resource authentication will need to be addressed prior to award. Information in this section must focus only on authentication and/or validation of key resources to be used in the study; all other methods and preliminary data must be included within the page limits of the research strategy. Applications identified as non-compliant with this limitation will be withdrawn from the review process (see NOT-OD-15-095). 

Review Criteria

Unless stated otherwise in the Funding Opportunity Announcement, reviewers will be asked to consider additional review questions in order to assess rigor and transparency in research grant applications. 

Significance: Is there a strong scientific premise for the project?

Approach: Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?  Have the investigators presented adequate plans to address relevant biological variables, such as sex, for studies in vertebrate animals or human subjects? 

Additional Review Considerations

Authentication of Key Biological and/or Chemical Resources
For projects involving key biological and/or chemical resources, reviewers will comment on the brief plans proposed for identifying and ensuring the validity of those resources.

For more information on implementation of these policies, visit: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-011.html

Impact on RPPRs

Research Performance Progress Reports (RPPR) submitted January 25, 2016 or later will be expected to emphasize rigorous approaches taken to ensure robust and unbiased results. Rigor should be addressed in the RPPR for any grant that funds research or training in research; grants that support other activities do not need to address rigor.  This includes non-competing continuation reports (Type 5) for grants reviewed and awarded before implementation of the policy. The RPPR instructions will be updated by January 25, 2016. Reporting on rigor in RPPR will help NIH implement and evaluate the policy for both current and new awards, as well as prepare non-competing renewals for the next competitive renewal.

These RPPR updates for rigor and transparency:

  • clarify long-standing expectations to ensure that NIH is funding the best and most rigorous science,
  • highlight the need for awardees to describe details that may have been previously overlooked,
  • prepare non-competing renewals for the next competitive renewal, and
  • will help NIH implement and evaluate the policy for both current and new awards.


Updates to Section B - Accomplishments. By January 25, 2016, the Research Performance Progress Report (RPPR) instructions will be updated to include the following additional guidance for 6.2 Section B - Accomplishments, in addition to the existing instructions.  Progress reports submitted on or after January 25, 2016 that are initiated prior to the instruction updates may use the current forms while following these additional instructions.  The instructions that will address rigor are listed below for your convenience.

B.2 What was accomplished under these goals? Include the approaches taken to ensure robust and unbiased results.

B.6 What do you plan to do for the next reporting period to accomplish these goals? Discuss efforts to ensure that the approach is scientifically rigorous and results are robust and unbiased.

For more information on how to implement any of these updates or for more information, visit the NIH website: http://grants.nih.gov/reproducibility/index.htmOr contact the University of Arizona Health Sciences Office of Research Administration at ahscgrants@list.arizona.edu

Originally Posted: 
Thursday, January 28, 2016 - 2:15pm