What is a Quality IRB Review?

Metrics in general are one of the ways we can gauge how well a program or service delivers. In the world of institutional review boards (IRBs), some measurements can be easily ascertained, such as review turnaround times and submission volumes, which help the human research protection program (HRPP) assess the IRB’s activities.

Measurements such as review timelines and volumes are typically easy to obtain and can inform the speed at which the IRB assesses proposed research and provides a determination back to the researcher. Measuring and seeking to improve upon such metrics can clearly benefit the research enterprise by facilitating scientific progress, offering faster study startup and subsequent participant enrollment.

However, there are other aspects of IRB quality and effectiveness that are more difficult to quantify but could provide equally useful insights. In this blog, we explore the challenges of defining and measuring “quality” IRB review and certain key efforts by research community stakeholders to overcome such obstacles.

Quality IRB Review Assessments

When attempting to identify the factors that define a quality IRB review, most stakeholders agree the goal is to determine whether the IRB’s actions truly improved protections for research participants.

The IRB’s primary mission is to protect the rights and welfare of participants in research. Thus, any measurement of IRB quality should assess how the IRB’s actions furthered (or distracted from) that primary responsibility. For example, questions that may be asked include, but are not limited to, the following:

• Did the IRB’s changes mitigate or lessen the study’s potential risks for participants?
• Did the IRB focus on what will truly make study participation safer?
• Did the review focus on “need to have” regulatory issues vs “nice to have” preferences?
• Does the review process help advance scientific progress with timely, informed ethical inputs?

Such measurements are not necessarily easy, but they can provide real insights into the IRB’s effectiveness in meeting its regulatory responsibilities. While other measurements may also be valuable, the primary focus in establishing IRB quality metrics should be on how the IRB is achieving its mission.

With that in mind, factors that may be indicative of a quality IRB review might include evaluating the types of changes an IRB may request or require prior to approving a research protocol. These changes can then be further categorized into “must have” versus “nice to have” as one way to measure if the changes led to improvement of participant protections.

It could also involve tracking how many times the IRB and submitting entity (e.g., study sponsor or principal investigator [PI]) go back and forth with questions or changes during the pre- and post-review process. Extending this back-and-forth may suggest the IRB’s instructions are unclear or incomplete. Alternatively, it may indicate the initial questions posed by the IRB reviewers led to protocol improvements, allowing the IRB to fully evaluate the criteria for approval – suggesting the extra time was well spent.

Once the metrics are defined, you can measure the data over time to drive improvement.

Current Initiatives to Measure IRB Quality

The Association for the Accreditation of Human Research Protection Programs (AAHRPP), the accrediting body for HRPPs, has recently provided accredited organizations with metrics to help “identify and support high-performing practices for HRPPs.” Their metrics focus on comparing information reported by academic organizations, hospitals, and a combination of all AAHRPP-accredited organizations.

The AAHRPP data includes elements like IRB review times, IRB staffing and budgets, the number of active studies the HRPP oversees, and applicable regulatory frameworks. HRPPs can potentially use the information from AAHRPP to, for example, help ensure appropriate staffing and budgets to support the review of research or compare review timelines with high-performing peers (assuming the metrics provide a fair comparison).

However, it’s unclear whether these quantitative metrics equate to the ultimate goal of protecting research subjects; rather, they help evaluate the quality of IRB operations. Evaluating quality IRB review is more difficult – consider:

• How do you measure and compare ethical considerations?
• How do you evaluate the consistent resolution of controverted issues?
• How did the IRB ensure the protection of research subjects?

The Consortium to Advance Effective Research Ethics Oversight (AEREO) has studied and thought about this challenge since 2018. AEREO considers the outcomes of interest for measuring IRB and HRPP effectiveness to collectively be:

• Protection of research participants
• Promotion of justice
• Creation of ethical research cultures
• Public trust in the research enterprise
• Socially valuable, scientifically valid, and ethical research

AEREO concedes these outcomes can be difficult to measure because they are in part not easily operationalized.

AEREO’s ongoing projects have led to a number of recommendations for IRBs and HRPPs assessment, such as:

• IRBs can set up a process to evaluate the scope of meaningful discussion and debate about key ethical and regulatory issues during board meetings
• HRPPs can implement survey assessments to gather research stakeholder feedback to help answer the question of what impact the IRB’s review had on the research conduct

Continuing research into how best to measure IRB quality or effectiveness will help move away from anecdotal beliefs. For example, if an IRB frequently uses outside experts, it might lead one to believe the membership composition was not appropriate. However, this could alternatively be an indication of a high-performing IRB. Recognizing the need for added expertise can help ensure knowledgeable review.

Projects such AEREO’s Institutional review board use of outside experts: What do we know? found the use of outside experts have important implications for IRB quality. A subsequent survey of IRB chairs confirmed appropriate expertise is critical to assess essential study elements.

Moving Forward

There are currently more questions than answers. But it’s important we continue to press on this to ensure the best possible participant protections and move the needle toward the development and adoption of specific, measurable goals.

Advarra believes in quality in everything we do, and we are committed to helping the research community develop IRB quality standards. As part of this commitment, we have collaborated with other stakeholders to review and to make sense of existing guidance and other research. We have also launched a working group focused on defining what makes a “quality IRB.”

We look forward to continuing to collaborate with our IRB colleagues and other stakeholders to better understand and apply what “quality” means for IRB review.

Back to Resources