Reiter, Harold I. MD, MEd; Lockyer, Jocelyn MHA, PhD; Ziola, Barry PhD; Courneya, Carol-Ann MSc, PhD; Eva, Kevin PhD; for the Canadian Multiple Mini-Interview Research Alliance (CaMMIRA)
Achieving greater diversity among medical students carries the promise of various benefits, such as shifting service patterns to better provide care for the underserved, and greater opportunities for race and language concordance between providers and patients.1 One strategy to enhance diversity among students is to increase the numbers of qualified applicants from traditionally underrepresented groups.2 In pipeline programs, high-achieving students from disadvantaged and underrepresented backgrounds are identified at the secondary or postsecondary school level and receive focused educational support and mentorship as they pursue medical school. The new Liaison Committee on Medical Education (LCME) accreditation standards, MS-8, have come down firmly in support of pipeline programs as effective means of increasing diversity among medical students. The standards promote
a variety of approaches, including, but not limited to, the development and institutionalization of pipeline programs, collaborations with institutions and organizations that serve students from disadvantaged backgrounds, community service activities that heighten awareness and interest in the profession, and academic enrichment programs for applicants who may not have taken traditional premedical coursework.3
Pipeline programs may be effective, but they require dedicated resources over time. For most medical schools wishing to enhance diversity, resources are limited. Could methods of selection sufficiently enhance diversity to obviate the need for pipeline programs? Reports regarding diversity implications of grade point average (GPA) and Medical College Admission Test (MCAT) scores hardly support that contention. Data available regarding both GPA4 and MCAT5–7 have suggested that relying on these traditional admissions measures may actually limit diversity. There are few data on other measures of applicant performance from which to find guidance. One ubiquitous measure, interviews, came into being for the precise purpose of limiting racial diversity.8 Our experience suggests that medical training programs have moved beyond such duplicity toward an honest effort to be more inclusive. However, the absence of any standardized, validated approach to interviews (or any other assessment of personal characteristics) has made it impossible to determine the interview's effect on the collective diversity of the group of applicants who are ultimately selected.
In a successful effort to improve on reliability and predictive validity,9–12 the Multiple Mini-Interview (MMI)13 has been adopted in the majority of medical schools in Canada and at dozens of other health professional schools scattered across the United States, Australia, Europe, the Middle East, Southeast and Central Asia, and the Far East. However, the influence of the MMI on diversity and access to medical school for underrepresented minorities has not been assessed.
The MMI was created in 2001 and fully implemented for the first time in 2004 at the Michael G. DeGroote School of Medicine at McMaster University. Like its predecessor, the objective structured clinical examination (OSCE),14 the MMI uses sampling to dilute biased results arising because of the context specificity of any single interaction. Like the OSCE, students rotate through a circuit of stations in bell-ringer fashion, each station introduced by a stem—a short paragraph read by the applicant outside the room before he or she enters—and each performance scored by a distinct rater. A set of several identical MMI circuits can be run simultaneously; many schools now run two, three, or even four sets of circuits in different time slots over the course of an interview day. The requirement for applicants to interact with many interviewers creates further potential to dilute bias in scoring arising from the idiosyncratic views of any one interviewer. But what effect does this diversity of perspective have on the diversity of the matriculant pool?
The purpose of our study was to address two questions: (1) What is the association between MMI scores and diversity measures (considering age, gender, size of community of origin, income level, and self-declared aboriginal status), and (2) if a standardized, validated interview methodology such as the MMI was found to be diversity-neutral, would it tend to dilute the diversity-limiting impact of other admissions measures, such as GPA, so as to obviate the need for upstream program implementation? For lack of a universally acknowledged categorization of diversity, we used the categories identified in the LCME May 2011 Standards for Accreditation, IS-16: gender, racial, cultural, and economic factors.
Do traditional performance measures affect diversity?
Our first question was whether MMI, GPA, and MCAT were diversity-neutral or diversity-limiting for our applicant pools. As a consortium of six Canadian medical schools at separate universities (McMaster University, University of Saskatchewan, University of Calgary, University of British Columbia, Dalhousie University, and University of Alberta), which have adopted the MMI as part of our selection procedures, we agreed to share anonymized data for admissions cycles ending in 2008 and in 2009. We obtained ethics approval for this study from the research ethics boards of Hamilton Health Science Corporation, the University of Calgary conjoint health ethics research board, the behavioral research ethics board of the University of Saskatchewan, the University of British Columbia behavioral research ethics board, and the Dalhousie University social sciences and humanities research ethics board. The protocol was submitted to the University of Alberta research ethics board, who declared that the study could proceed without formal ethics approval from that body.
All applicants invited for interviews at our institutions during the study period participated in the MMI. Table 1 provides a description of the MMI process used by each school involved in our group. With the rare exception of accommodation for an applicant disability, we used the format of a 10-minute station (2 minutes for reading the stem and 8 for the interview) at each of our schools.
To conduct the study, we removed all applicants' personal identifiers from admissions files before data collection, aggregation, and analysis. Because different schools used different numeric scales for MMI scoring, we converted each school's MMI scores to z scores before aggregation. Where available, from each school's interviewed applicant pool, we also collected GPA and overall or subsectional MCAT scores. We were only able to aggregate GPA data from schools that used a four-point scale because variability in the application thresholds used by each school precluded simply combining scores after z scoring within institutions. Not all schools reported data for each variable, so the sample sizes in each analysis were variable and are reported in Table 2.
Where available, from each school's interviewed applicant pool, we collected the following diversity variables: age at the time of application, gender, self-declared aboriginal status, and postal code. For the purpose of this study, we defined applicant postal code as the postal code of their residence during high school.
Through arrangements via the Data Liberation Initiative with Statistics Canada, we obtained conversion codes (Postal Code Conversion File Plus Version 5C, Statistics Canada, Ottawa, Ontario, Canada; catalog no. 82F0086-XDB) to convert applicants' postal codes first into dissemination codes and then into quintile groupings representing both community size and income levels. We elected to use community size and income levels because they tend to reflect socioeconomic status. Beyond those applicants who self-declare as aboriginal, collection of data on ethnic background is precluded by law in Canada. Constitutionally in Canada, the aboriginal peoples are the original peoples of North America and their descendants, comprising the three groups of First Nations (Indians), Metis, and Inuit, each with its own distinct history and culture.
We used Microsoft Excel (Redmond, Washington) for Mac, SPSS (IBM, Armonk, New York) for Mac, and SAS (Cary, North Carolina) for all data analyses. We calculated correlations between admissions test variables to ensure that they reflected previously published data. We used Pearson correlations and analysis of variance techniques to assess the relationships between diversity variables and MMI performance, GPA, and MCAT scores. To verify the consistency of the data, we analyzed these relationships independently for the 2008 and 2009 interviewed applicant groups.
Would applying the MMI affect diversity?
Our second question was to learn whether application of the MMI would tend to dilute the previously reported diversity-limiting effect of GPA.7 The best available data set to address this question were the 2009 data from McMaster University. Unlike the other schools and years, McMaster applied the simplest formulae to rank 2009 interviewees for offer of admission. That year, applicants were invited to interview at McMaster on the basis of a weighting formula of 60% GPA and 40% autobiographical essays (determined after z scoring). Of the 546 interviewed applicants, we sent offers of admission to the 221 top-ranked applicants on the basis of a formula weighting 30% GPA and 70% MMI (again, compiled after z scoring those two measures). All other schools in both years used less algorithmic formulae, thereby creating more potential for confounding results, such as possible bias from admissions committee members' reviews of files.
To determine whether MMI-based admissions decisions would affect the diversity of the accepted applicant pool differently than GPA-based decisions would, we created comparison rankings whereby (1) GPA was weighted 100% and (2) MMI was weighted 100%. In terms of underrepresentation, the outcome variables that held our greatest interest were self-declared aboriginal status, size of community of origin, and income level. Neither age nor gender is an issue of underrepresentation in Canada. The number of applicants with self-declared aboriginal status in this sample was too small for analysis. We assessed the effect of the different weighting schemes by examining the frequency with which applicants from different quintiles of income level and size of community of origin would have received an offer of admission according to each scheme. We used quintiles because this is the manner in which Statistics Canada provides its postal code conversion codes.
Traditional performance measures and diversity
The total number of interviewed applicants across all six schools was 5,253 (details provided in Table 1). Taking into account the number of interviews for each applicant at each school, the total number of interview interactions that we conducted was 53,471.
Correlations between diversity variables and admissions test variables are provided in Tables 2 and 3, excepting gender, whose correlations with MMI, GPA, and MCAT scores were all found to be neutral (i.e., nonstatistically significant). Specifically, standard MMI scores for females and males were z = 0.05 and −0.05 in 2008 and z = 0.08 and −0.10 in 2009, respectively. Average GPA scores were 3.47 and 3.48 in 2008 and 3.64 and 3.66 in 2009 for females and males, respectively, and average MCAT scores were 9.24 and 9.48 in 2008 and 9.66 and 10.13 in 2009 for females and males, respectively.
Other diversity variables did demonstrate differences. First, there was a positive correlation between MMI scores and age (P < .05); the correlation was neutral with respect to gender, size of community of origin, and income level, and there was a negative correlation (P < .05) for those with self-declared aboriginal status. Second, scores in both GPA and MCAT were negatively correlated with age (P < .001) and were lower for self-declared aboriginals (P < .05). GPA was positively correlated with income level (P < .05) and trended upward for larger community of origin, though both correlations were quite small; MCAT was positively correlated with size of community of origin (P < .05) but was not related to income level. Finally, all correlations between admissions test variables were closely in keeping with previously published results.13,15,16 GPA was positively correlated with MCAT total score, biological sciences subsection score, and physical science subsection score and was less well correlated with verbal reasoning subsection score. GPA and MMI were not correlated, nor were MMI and overall MCAT score correlated.
MMI and diversity
Of all outcome variables, the community size of origin, income levels, and self-declared aboriginal status were the most obviously reflective of underrepresentation in Canada. Because the overall number of applicants with self-declared aboriginal status was low in the cohort sampled for this analysis, we did not consider this variable. In moving from the 30% GPA:70% MMI weighted formula to a 100% GPA or 100% MMI weighting, there was a 46% and 16% change, respectively, in the students who would hypothetically have received offers of admission. The change in the diversity outcomes of the hypothetical accepted student pools, however, was virtually indistinguishable, as illustrated in Figures 1 and 2.
In this study, on the basis of data for over 5,000 applicants to six medical schools collected during two years, we examined the relationship between GPA, MCAT scores, and MMI scores with respect to diversity variables including age, gender, income, community size, and aboriginal status. It should be noted that the MMI is an assessment process rather than a tool, and, as such, any MMI has the potential to be biased or discriminatory, just as any test can be biased or discriminatory depending on the questions. It is, therefore, essential that admissions committees maintain vigilance and vet their stations, among other actions, to minimize such risk. That said, in the manner in which we executed the MMI in our Canadian medical schools, it appears for the most part that MMI scores did not consistently correlate with gender, size of community of origin, or income level.
Female gender has been found to correlate with better communication skills in other settings.17 We did not find this correlation with MMI, however—an outcome consistent with previously reported results.13 Similarly, we did not find any correlation between gender and either GPA or MCAT in our study.
Community of origin may be of interest for schools with a mandate to favor local applicants, or rural applicants in the case of medical schools with distributed campuses in rural areas. Our results showed no difference in MMI scores based on region of origin. However, there were small correlations indicating that lower GPA and lower MCAT scores tended to be seen in applicants from smaller communities.
In terms of access to medical school for the underprivileged, applicants' income level is of interest for most institutions. Our results again showed no difference in MMI scores based on income level. However, lower MCAT and lower GPA tended to be seen in applicants whose high school residence postal codes were associated with lower income levels.
We found a small, positive correlation between age and MMI scores, inverse to the correlation found between age and GPA and MCAT scores. We also found that GPA, MCAT scores, and performance on the MMI were lower in those with self-declared aboriginal status. The relationship between older age and higher MMI score may not be surprising because the MMI requires applicants to demonstrate judgment in numerous situations, and older applicants may thus have an advantage. The fact that younger applicants had higher GPA and MCAT scores may be an artifact of the application process whereby applicants will begin applying at their earliest opportunity. Those with the highest GPA and MCAT scores are likely to be accepted first and will, therefore, leave the applicant pool.
Correlations with self-declared aboriginal status are the most challenging to interpret, for a variety of reasons. National and intramural definitions of “aboriginal” are variable, even within defined Canadian constitutional parameters. It is tempting, on the basis of the relatively small sample of self-declared aboriginal applicants in our sample, to discount these results. However, small sample sizes should lessen, not increase, the likelihood of a statistically significant finding. On the other hand, diversity programs include recruitment and facilitative admissions processes; both attract candidates with generally lower admissions test scores, artificially strengthening the negative correlations.
A prior experimental study18 showed no difference in MMI scores regardless of the aboriginal or nonaboriginal identity of applicant and/or rater. We would require further data accumulation over many years to overcome the sample size limitation, but, until then, caution should be used in interpreting our results. Although no definitive conclusions can be drawn, admissions committees using the MMI (just as with any other interview process) should at the very least exercise abundant vetting of MMI stations with an eye toward aboriginal cultural acceptability.
Our greatest surprise in the results was the lack of influence the MMI had on diversity representation within the pool of accepted applicants. Our expectation was that the MMI's neutral relationship with diversity measures such as income level and size of community of origin would dilute the diversity-limiting effects of GPA and MCAT. This was clearly not the case because we observed near-identical distributions among hypothetical accepted student pools along the parameters of income level and size of community of origin, even in our imagined scenario of using only the MMI on the interviewed group. Changing selection criteria may markedly change those being admitted.19 Our results demonstrate that even marked changes in those admitted are no guarantor of benefit to diversity. We believe that the explanation likely lies in the methods by which applicants reach the interview stage. Medical schools tend to set grade thresholds for applicants, and applicants themselves further self-select, with those demonstrating predominantly high academic grades being most likely to apply. The result is that, generally, the GPA range for those coming to interview is restricted. As long as the decision of who comes to interview is significantly dependent on a diversity-limiting measure, applying any diversity-neutral measure at the time of interview cannot be expected to substantially improve diversity. We did not expect these results, despite harbingers published a generation ago,20,21 that underrepresentation was in large part due to the limited numbers of qualified applicants reaching stages from which they could be selected. These results ultimately support the contention, reflected in the LCME accreditation standard update, that efforts in favor of diversity should be focused upstream, on pipeline programs.
We recognize that there are limitations to the study which make extrapolation outside of Canada difficult. First, because of legal and historical convention in Canada, we do not have data on any ethnic group other than self-declared aboriginal people, for which the sample size is small. At the moment, the effect of the MMI on the representation of people of diverse ethnic backgrounds in matriculant pools is unknown, awaiting sufficient data accumulation from medical schools worldwide.
Additionally, the geocoding breakdown for different scenarios to determine the effect of different admissions weightings was limited to one school in one year because of database limitations. To determine family income, we converted data from postal codes, which do not necessarily represent the applicants' true family income levels. Specifically, use of geocoding is vulnerable to ecological inference fallacy.22 Nevertheless, the results we found were the same for community size of origin, for which no vulnerability to ecological inference fallacy exists.
Despite these limitations, our results are generally consistent across two years and across different jurisdictions, despite each of our schools having its own independently conducted MMI and its own uniquely defined mission statement, admission policies, and procedures. The absence of a clear benefit to diversity is the more worrisome outcome. Even at a time of heightened medical school awareness of the need to enhance diversity, we found that greater weight on the MMI at the time of interview may simply be too little, too late.
The authors acknowledge and thank Russell Wilkins of Statistics Canada for his provision and explanations of Canadian postal code conversion formulae data. The authors further acknowledge and thank the admissions coordinators of the six medical schools—Wendy Edge, Heather Mandeville, Catherine Macala, Anita MacDonald, Denis Hughes, Peter Romeo, and Diane Baker, who kindly provided the applicant data for the study.
1. Saha S, Shipman S. The Rationale for Diversity in the Health Professions: A Review of the Evidence. Rockville, Md: Health Resources and Services Administration; 2006.
2. Thomson WA, Denk JP. Promoting diversity in the medical school pipeline: A national overview. Acad Med. 1999;74:312–314.
3. Liaison Committee on Medical Education. Functions and Structure of a Medical School: Standards for Accreditation of Medical Education Programs Leading to the M.D. Degree. http://www.lcme.org/functions2011may.pdf
. Accessed December 21, 2011.
4. Tekian A. Minority students, affirmative action, and the admission process: A literature review, 1987–1998. Teach Learn Med. 2000;12:33–42.
5. Koenig JA, Sireci SG, Wiley A. Evaluating the predictive validity of MCAT scores across diverse applicant groups. Acad Med. 1998;73:1095–1106.
6. Huff KL, Koenig JA, Treptau MM, Sireci SG. Validity of MCAT scores for predicting clerkship performance of medical students grouped by sex and ethnicity. Acad Med. 1999;74(10 suppl):S41–S44.
7. Association of American Medical Colleges. MCAT Report 1999: Characteristics of the 1999 MCAT Examinees. Washington, DC: Association of American Medical Colleges; 1999.
8. Karabel J. The Chosen: The Hidden History of Admission and Exclusion at Harvard, Yale and Princeton. New York, NY: First Mariner Books; 2005.
9. Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009;43:767–775.
10. Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007;41:378–384.
11. Eva KW, Reiter HI, Rosenfeld J, Norman GR. The ability of the multiple mini-interview to predict pre-clerkship performance in medical school. Acad Med. 2004;79(10 suppl):S40–S42.
12. Hofmeister M, Lockyer J, Crutcher R. The multiple mini-interview for selection of international medical graduates into family medicine residency education. Med Educ. 2009;43:573–579.
13. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: The multiple mini-interview. Med Educ. 2004;38:314–326.
14. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination. Med Educ. 1979;13:41–54.
15. Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80:910–917.
16. Kulasegaram K, Reiter HI, Wiesner W, Hackett RD, Norman GR. Non-association between Neo-5 personality tests and multiple mini-interview. Adv Health Sci Educ Theory Pract. 2010;15:415–423.
17. Wood JT. Gender, communication, and culture. In: Samovar LA, Porter RE, eds. Intercultural Communication: A Reader. Stamford, Conn: Wadsworth; 1998.
18. Moreau K, Reiter H, Eva KW. Comparison of aboriginal and non-aboriginal applicants for admissions on the multiple mini-interview using aboriginal and non- aboriginal interviewers. Teach Learn Med. 2006;18:58–61.
19. Ziv A, Rubin O, Moshinsky A, et al.. MOR: A simulation-based assessment centre for evaluating the personal and interpersonal qualities of medical school candidates. Med Educ. 2008;42:991–998.
20. Sleeth BC, Mishell RI. Black under-representation in United States medical schools. N Engl J Med. 1977;297:1146–1148.
21. Kane TJ. Racial and ethnic preferences in college admission. In: Jencks C, Phillips M, eds. The Black-White Test Score Gap. Washington, DC: Brookings Institute; 1998.
22. Robinson WS. Ecological correlations and the behavior of individuals. Am Sociol Rev. 1950;15:351–357.
Canadian Multiple Mini-Interview Research Alliance (CaMMIRA):
Mark Moreau, MD, University of Alberta; Kevin Eva, PhD, University of British Columbia; Carol-Ann Courneya, PhD, University of British Columbia; Michael Clifford Fabian, MD, University of British Columbia; Jocelyn Lockyer, PhD, University of Calgary; Ian Walker, MD, University of Calgary; Evelyn Sutton, MD, Dalhousie University; Harold Reiter, MD, MEd, McMaster University; Mahan Kulasegaram, McMaster University; Kelly Dore, PhD, McMaster University; Ilana Horvath, McMaster University; Tanya Stone, McMaster University; Lauren Griffith, PhD, McMaster University; Barry Ziola, PhD, University of Saskatchewan.
Ethics approval for this study was obtained from the research ethics boards of Hamilton Health Science Corporation, University of Calgary conjoint health ethics research board, behavioral research ethics board of the University of Saskatchewan, University of British Columbia behavioral research ethics board, and the Dalhousie University social sciences and humanities research ethics board. The protocol was submitted to the University of Alberta research ethics board, who declared that the study could proceed without formal ethics approval from that body.