Johansson, Anna C. PhD; Durning, Steven J. MD; Gruppen, Larry D. PhD; Olson, Marianne E. RN, PhD; Schwartzstein, Richard M. MD; Higgins, Patricia A. RN, PhD
A hallmark of academic medicine is its emphasis on sharing knowledge. In that spirit, new educational programs are often designed and implemented using methodologies to increase validity and reliability of outcomes, and the probability that the program may be useful to others. As the call for competency-based training and assessment has increased, medical school and residency training curricula have undergone extensive revisions, which require assessment of new programs designed to improve the quality of clinical care and education.1–4 In some instances, practitioners undertake an evaluation as part of a quality improvement (QI) activity intended to identify shortcomings and improve local processes; however, the activities may be research and involve human participants, which make them subject to federal human research regulations. Thus, the dilemma for educators and institutional review boards (IRBs) is to make what are often subtle distinctions between QI work and medical education (ME) research.
In 2007, the Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center, in conjunction with the Association of American Medical Colleges, hosted its biannual Millennium Conference. Representative teams of educational researchers from medical schools across the country gathered to explore pressing challenges for ME research.5,6 Teams suggested that medical educators face unique challenges in the IRB review process compared with traditional clinician researchers, especially when ME research is reviewed by IRBs managing predominantly biomedical research portfolios.
Following the conference, we formed a subgroup to reexamine the IRB process for ME research. After numerous discussions and a review of the literature, we became aware that the IRB community and the ME research community do not share a common understanding of the key concepts and definitions that underlie the regulatory framework that governs protection of human research subjects. This impacts ME inquiries in two important ways. First, because it is often difficult to determine when an inquiry, such as an analysis of an institution's residency match program, is considered human subject research, many institutions have inappropriately relied on the IRB to review and oversee activities that are legitimately classified as something other than human subject research. Institutions may lack sufficient guidance and procedures to assist IRB members and educators in making an appropriate determination about the regulatory status of a proposed project. Likewise, for inquiries that are legitimately determined to be human subject research, institutions may lack guidance needed to address the nuances of the levels of review allowed by the regulations to ensure that projects are handled in a consistent, efficient manner.7
We therefore explored the following questions:
* When does an ME inquiry meet the regulatory definitions of research and human subjects and therefore become subject to the human research regulations? How do we differentiate routine ME program updates from research?
* If an ME inquiry meets the regulatory definition of research and human subjects, what criteria should ME researchers and IRBs use to evaluate the risk to participants?
* If an ME inquiry is determined to be human subject research, what are the distinctions between the exempt and expedited provisions allowed by the federal regulations?
* How can IRBs maximize the latitude allowed by the federal human research regulations when reviewing minimal-risk ME research to provide protections for learners and educators, and support for the research?
We pose questions that require both conceptual consideration and practical guidelines. We hope that this article will engage IRBs and ME researchers in this important discussion and bring them closer to a shared understanding of key concepts underlying the regulatory framework and foster consensus on the application of these requirements.
When Is an ME Inquiry Human Subjects Research?
Key concepts and critical distinctions
We found the current discourse on QI useful for framing the role of education in academic health care and the features that distinguish routine education and evaluation from research.7,8 Like QI, ME is integral to the fabric of the academic health care environment; therefore, clinicians and educators have an ethical responsibility to address context-specific needs by implementing new knowledge or teaching strategies that have been demonstrated to be superior to outdated practices. Furthermore, educational programs must be evaluated and the results shared with other health care providers or educators from a lessons-learned context, even though publication is not a primary goal. Research, on the other hand, is designed to develop new knowledge or contribute to existing knowledge, by generating results that apply to more than one location—It informs us of something “enduring” about human and organizational processes, including learning processes.7,8 Whereas publication has a central role in research, research is not defined solely by the intent to publish but, rather, by the intent to generalize findings.9,10
Like QI, ME research can pose challenges to a working definition of generalizable knowledge. Because ME inquiries can be designed according to either the QI or research paradigm, the challenge is distinguishing “routine” educational activities from those that are designed to also understand what makes the new activities superior to existing practice—hence, applicable beyond the local environment and subject to IRB review.11 Although Table 1 does not capture the full range of methodologies used in ME evaluation and research, it summarizes characteristics that can help distinguish ME research from QI initiatives in ME.
The conceptual differences between QI and research provide a framework for determining whether an inquiry is human subject research—a process that involves answering two questions: (1) Do the activities proposed meet the federal definition of research, and (2) if so, are human subjects involved? Activities that meet both definitions will require further application of the human research regulations and policies of the local IRB.10,12–15
Step 1—Does my ME inquiry meet the definition of “research”?
Research is defined by the Department of Health and Human Services as “a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge”—an inquiry that has the purpose of learning something that may apply to populations beyond the primary site or context of the activity.12 There are two critical elements of this definition to consider: (1) The ME inquiry must be a systematic investigation, and (2) the design—goal, purpose, intent—of the investigation is to develop or contribute to generalizable knowledge. The two components have been described as having the effect of either “widening” or “narrowing” the definition of research relative to the inquiry under consideration.16
The first element—a systematic investigation—may be characteristic of both research and nonresearch ME inquiries. For example, practitioners routinely collect information for reporting purposes by using data collection methods and statistical techniques commonly employed in research to create quality dashboards, course evaluations, or resident performance evaluations. Characteristics of a systematic investigation include clearly stated objectives, a carefully designed methodology to achieve the objectives, and a reasoned approach to assessing outcomes of the project. Yet, a systematic investigation alone does not classify an ME inquiry as research.
The second element—intent— suggests that the primary reason for conducting the activity is to develop or contribute to generalizable knowledge. There must be an explicit connection between what is known (e.g., via a theoretical or conceptual framework) and what is not known, and a clear delineation of how this ME inquiry will fill the knowledge gap or contribute new knowledge. Figure 1 illustrates the relationship between the components of the definition of research. ME inquiry conducted in the QI paradigm, on the other hand, is a process of self-monitoring and self-assessment; results are applied in a “lessons-learned” format intended to improve a process or practice for a specific context or constituency.17
Step 2—Does my ME inquiry involve “human subjects”?
Determining whether an ME inquiry involves human subjects is less ambiguous than determining whether the activity is research. A human subject is defined as “a living individual about whom an investigator conducting research obtains: (1) data through intervention or interaction with the individual, or (2) identifiable private information.”12 Many educational activities involve human participants but do not meet the regulatory definition of research. Likewise, activities determined to meet the regulatory definition of research may not meet the definition of human subjects. Figure 2 illustrates the relationship between the components of the definition of human subjects.
The challenges of differentiating QI-driven ME inquiries from research
A proposed ME inquiry does not always fall neatly above or below the threshold definition of human subject research. Pritchard16 notes that efforts to contribute to generalizable knowledge in order to help future populations are often driven by curiosity as well as an initial need to solve an immediate problem. Training, education, QI, or case reviews are routine practices that may not begin as research but may evolve to a formal program evaluation or become research when an individual decides to take accidental discoveries or innovative practices a step further and engage in a systematic investigation with the intent to contribute to generalizable knowledge.
Confusing QI initiatives with research can, however, have far-reaching consequences. Medical educators can face unintended violations of the human research regulations when activities perceived as routine evaluation cross the threshold definition of research.13 Institutions are vulnerable to legal consequences for violating human research regulations or failing to properly educate members with respect to each paradigm.17 The IRB and the educator would benefit from institutional policies and procedures, such as those required for institutions accredited by the Association for the Accreditation of Human Research Protection Programs, that support a process for vetting ambiguous activities when seeking the appropriate application of the federally mandated regulatory framework at the local level.14,15 With a clear understanding that the proposed inquiry is human subject research, the IRB can proceed with its review, which begins with an evaluation of risk.
Examining the IRB Review Process
The paradox of risk in human subject research
A fundamental responsibility of an IRB is to assess the relative risks and potential benefits of a study and to apply the appropriate review process. Risk is commonly understood as the possibility of a negative consequence due to participation in a research activity. It is both the nature and magnitude of risk that dictate the IRB review process.
The Office of Human Research Protections has codified two general categories of risks to human subjects.18 The first focuses on the risk of direct physiological or psychological harm. For example, a research proposal may call for an invasive procedure such as a biopsy, or an interview that invokes unpleasant memories or distressing feelings such as residents describing their first patient death experience. The second category centers on the loss of privacy or breach of confidentiality that could place a subject at risk for negative social, professional, or economic consequences, or even criminal or civil liabilities, especially if data collection requires private, personal, identifiable information that could be publicly or financially damaging. Breaches to confidentiality, for example, can include the inappropriate use of data to evaluate residents when those data were collected as part of an experimental educational research intervention. Alternatively, failure to protect routinely collected data (such as course grades or evaluations) while conducting a secondary analysis of the data for research purposes constitutes a breach of the regulations. Medical educators must consider the level of risk and how it applies to their educational research studies. They should consider that students, trainees, and staff may be vulnerable to coercion under conditions in which there is a hierarchical power relationship between the investigators and the subjects.4,19,20
Although the regulations do not define risk, per se, they do define minimal risk: “the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.”12 Research exceeding the minimal-risk threshold requires review at a convened IRB meeting, whereas research posing no more than minimal risk may take alternate routes for review. Most ME research is likely to be considered minimal-risk research, where loss of privacy or breach of confidentiality is the principal risk.
Minimal-risk research can, ironically, present challenges to the IRB review process that differ from research that is greater than minimal risk. Research that poses more than minimal risk requires the IRB to be more prescriptive when applying the regulations, such as mandating strict requirements for ongoing data and safety monitoring and full written informed consent. Minimal-risk research, on the other hand, allows IRBs greater flexibility when applying the regulations; although created with the intent to deal with a range of human subject research, the IRB can unintentionally produce variability and inconsistency in review outcomes when trying to address ME proposals.16,21,22 This may be especially true when traditional biomedical IRBs lack members with sufficient expertise in social, behavioral, and educational research techniques to evaluate the proposed methodologies. The increasing interdisciplinary nature of human subject research in academic health care highlights the need for diverse IRB membership and IRB policies that invoke the flexibility the human research regulations already provide for minimal-risk research.
Further diffusing the confusion
After an IRB determines an inquiry is human subject research, the proposal is triaged to the appropriate review—exempt, expedited, or full committee. The categories of exempt and expedited protocols apply to minimal-risk human subject research—thus, we are not referring to those inquiries deemed to be nonresearch in keeping with the previously described guidelines—and reflect nuances in the meaning and implications of minimal-risk research as well as in IRB review processes. There are, however, some important differences between exempt and expedited research, which IRBs and ME researchers must consider.
Exempt human subject research
Exemptions are applied to minimal-risk proposals in which the study's procedures fall into one of the six exemption categories. Exemption categories are educational tests; surveys and instruments; observation of public behavior; existing documents and data/specimens if publicly available or unidentifiable; studies of federal benefits; and taste and food quality evaluation. “Exempt” does not equate to “no IRB review required.” Rather, the proposal has been determined to meet the definition of human subjects research and clearly conforms to a specific category defined in the regulations.23 Research activities codified in one of these six categories allow IRBs the latitude to review the research in a number of ways that effectively balance the resources required by the IRB with the risk of the proposed research; specifically, the degree of oversight required by the IRB and the demands associated with human subject consent procedures are relaxed if the proposal meets the criteria for exemption.24 Regulatory guidance encourages that IRBs make the exemption determination, not the individual investigator. Although human research studies that qualify for an exempt category are not subject to the same oversight criteria as nonexempt studies, such as ongoing monitoring, investigators are still obligated to report to the IRB any modifications to the study design or procedures prior to implementing them to determine whether the study continues to be exempt; in addition, the researcher must report any unanticipated problems involving risks to subjects that occur during the study. Although exempt research does not bear the requirement of an explicit written consent as may be required for nonexempt research, investigators are still obligated to ensure an appropriate informed consent process. Investigators must also protect the subjects' privacy and their confidential data—protections that are no different from what physicians and educators should provide for their patients and learners regardless of whether they are participants in a research study.24–26 Finally, an exempt determination does not relieve the researcher of his or her obligations to adhere to institution-specific research policies and procedures, which may impose requirements in addition to federally mandated requirements.
Human subject research eligible for expedited review procedures
Human subject research determined not eligible for exemption is triaged to the next level of IRB review—expedited review.27 An “expedited” review does not equate to “taking a shortcut,” or to an abbreviated review. A proposal that conforms to the regulatory definition of minimal risk and is not eligible for an exempt category, but conforms to one of nine defined expedited categories, is eligible for review through expedited procedures.28 The expedited review process allows an IRB chair to designate one or more experienced reviewers from among IRB committee members to perform the review, exercising the full authority of the IRB committee.27,29 Of the nine categories, seven apply to new research applications, and two apply to continuing review. Research activities eligible for the expedited review procedure are typically characterized by direct contact with research subjects, noninvasive physical procedures, or access to existing or routinely collected data, including private or confidential information for which a breach of confidentiality could pose the primary risk. One essential distinction is studies that include a target population defined by the federal regulations as a vulnerable population.23 If, however, the assigned IRB reviewer believes the study does not meet the criteria for expedited review, the convened IRB committee must review the study.
Illustrations and Suggestions
We have attempted to describe both the difficulties and nuances in determining whether an ME inquiry is QI or human subject research and, for the latter, whether research is eligible for exempt status or expedited review procedures. Table 2 illustrates these nuances.
The IRB is charged with the mission of ensuring the safe and ethical treatment of human research participants. In doing so it also plays a critical role in examining the scientific rigor of proposed investigations—including investigations that pose no more than minimal risk—thereby improving the probability that the results will be meaningful.11,30 As we reviewed the literature, listened to our colleagues, and drew on our own experience as educators, researchers, and IRB members, we became aware of the need to assist educators, researchers, and IRBs in reaching a common understanding of the definitions and interpretations needed to define the parameters for conducting ME research. We found that many educational activities intrinsic to the health care environment may not meet the regulatory definition of human subject research and, therefore, do not require IRB review and oversight.10 We also found that making the distinction between conducting ME inquiry as a QI project or as a research study can be confusing and can pose unique challenges for medical educators; similarly, medical educators may have difficulty delineating the level of IRB review required when the criteria for human subject research are met. To determine whether an ME inquiry is research, it is important to understand that nonresearch activities (e.g., QI projects) may use scientific methods and produce results suitable for publication. Nonresearch activities can be published as lessons learned but should never be referred to as research studies.17 In this context, we make the following recommendations:
* Institutions should provide comprehensive education to assist professionals and students in their efforts to distinguish between ME inquiries conducted as QI projects and proposals characterized as research studies; these efforts should also delineate the risks and implications of each paradigm.
* Institutions, through their IRBs, should provide their research communities with formal procedures for vetting ambiguous activities, including documentation from the IRB to support a determination that a proposed inquiry does or does not meet the regulatory definition of human subject research.
* Institutions should provide comprehensive programs to educate researchers about the implications of risk involved with educational inquiries and procedures or strategies to address those risks.
* IRBs should make use of the flexibility permitted by the current regulatory framework when reviewing minimal-risk research.
* IRBs managing predominantly biomedical research portfolios should include members with expertise in the social and behavioral sciences. These individuals should be familiar with a range of quantitative and qualitative methodologies as well as with education evaluation practice. Institutions with substantial social, behavioral, and educational research portfolios should have a separate IRB committee dedicated to reviewing this type of research.
* ME researchers should seek and receive instruction at an early stage of their careers about the IRB process. They should initiate conversations with their local IRB when contemplating educational innovation or evaluation so that proper consideration is given to the design of the proposed work, human studies requirements, and possible regulatory requirements.
Whether an activity is characterized as routine ME QI, human subject research that qualifies as exempt, or research that rises to a level at which the full protections of the human research regulations apply, consistent ethical principles prevail.8,9,31 We are reminded, however, that research has a unique ethical status in our society. Although participating in research is socially desirable, it is not required.16,32 Given the importance of ME research in our ability to produce doctors capable of practicing in an ever-changing and increasingly complex world, it is critical that IRBs and ME practitioners and researchers develop a common understanding of the current regulatory framework. To maintain the public's trust in the research enterprise, it is essential to conduct human investigations of the highest quality while ensuring ethical standards. This principle holds for ME as well as biomedical research.14,32 Our challenge is to aspire to high standards while obviating potentially unnecessary regulatory barriers.
The authors wish to thank the following individuals for valuable insight provided: Dan DeHainaut, protocol specialist-vulnerable populations, Committee on Clinical Investigations, Beth Israel Deaconess Medical Center; Jonathan Fisher, MD, vice chair, Committee on Clinical Investigations, Beth Israel Deaconess Medical Center; Grace Huang, MD, director of assessment, Shapiro Institute for Education and Research, Beth Israel Deaconess Medical Center; Julie Irish, PhD, director, Office of Educational Research, Shapiro Institute for Education and Research, Beth Israel Deaconess Medical Center; and Mary Williams, RN, MS, director of IRB operations, Committee on Clinical Investigations, Beth Israel Deaconess Medical Center.
The 2007 Millennium Conference was sponsored by the Shapiro Institute for Education and Research at Beth Israel Deaconess Medical Center and Harvard Medical School, and the Association of American Medical Colleges.
The views expressed herein are those of the authors and do not necessarily represent those of the Department of Defense.
1 Institute of Medicine. Health Professionals Education: A Bridge to Quality. Washington, DC: National Academy Press; 2003.
5 Huang GC, Gordon JA, Schwartzstein RM. Millennium conference 2005 on medical simulation: A summary report. Simul Healthc. 2007;2:88–95.
6 Huang GC, Newman LN, Anderson MB, Schwartzstein RM. Conference proceedings and consensus statements of the millennium conference 2007: A collaborative approach to educational research. Teach Learn Med. 2010;22:50–55.
7 Baily MA, Bottrell M, Lynn J, Jennings B; Hastings Center. The ethics of using QI methods to improve health care quality and safety. Hastings Cent Rep. 2006;36:S1–S40. www.thehastingscenter.org
. Accessed March 11, 2011.
8 Lynn J, Baily MA, Botrell M, et al. The ethics of using quality improvement methods in health care. Ann Intern Med. 2007;146:666–673.
9 Amdur JA, Speers M, Bankert EA. Identifying intent: Is this project research? In: Bankert EA, Amdur RJ, eds. Institutional Review Board Management and Function. 2nd ed. Sudbury, Mass: Jones and Bartlett Publishers, Inc.; 2006.
11 Brown J, Ryland I, Howard J, Shaw N. Views of National Health Service (NHS) Ethics Committee members on how education research should be reviewed. Med Teach. 2007;29:225–230.
14 Institute of Medicine. Preserving Public Trust: Accreditation and Human Research Participant Protection Programs. Washington, DC: National Academy Press; 2001.
15 Association for the Accreditation of Human Research Protection Programs. Tip Sheet: Determining Whether an Activity Is Research Involving Human Participants. www.aahrpp.org/Documents/D000130.PDF
. Accessed March 11, 2011.
16 Pritchard IA. Searching for “research involving human subjects”—What is examined? What is exempt? What is exasperating? IRB. May–June, 2001;23:5–12.
17 Newhouse RP, Pettit JC, Poe S, Rocco L. The slippery slope: Differentiating between quality improvement and research. J Nurs Admin. 2006;36:211–219.
19 McLachlan JC, McHarg J. Ethical permission for the publication of routinely collected data. Med Educ. 2005;39:944–948.
20 Willer D, Walker H. Building Experiments: Testing Social Theory. Stanford, Calif: Stanford University Press; 2007.
22 The Center for Advanced Study Project Steering Committee. Improving the System for Protecting Human Subjects: Counteracting IRB “Mission Creep.” Urbana-Champaign, Ill: University of Illinois; 2006.
24 Prentice ED, Oki GSF. Exempt from institutional review board. In: Bankert EA, Amdur RJ, eds. Institutional Review Board Management and Function. 2nd ed. Sudbury, Mass: Jones and Bartlett Publishers, Inc.; 2006.
25 Bedward J, Davison I, Field S, Thomas H. Audit, educational development and research: What counts for ethics and research governance? Med Teach. 2005;27:99–101.
26 Association for the Accreditation of Human Research Protection Programs. Tip Sheets: Exemptions: Determinations and Review. www.aahrpp.org/Documents/D000136.PDF
. Accessed March 11, 2011.
30 Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in medical education: A systematic review. Med Educ. 2007;41:737–745.
31 National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Washington, DC: U.S. Government Printing Office; 1978. DHEW Publication No. (OS) 78-0008.
Reference Cited Only in the Figures