Moving Beyond Angoff: Options for Setting Cut Scores, Minimal Qualifications, and Performance Standards
Presenters: Steve Ferrara, CTB/McGraw Hill
Lorin Mueller, American Institutes for Research
Coordinator: Dwayne Norris, American Institutes for Research
The Angoff technique is the default option for setting cut scores in organizational settings when sufficient job performance data are not available to conduct an empirical standard setting process. The prevalence of the Angoff technique has three limitations. First, practitioners often ignore available data that can help to inform the standard-setting process, such as psychometric data and limited job performance data. Second, limited familiarity with other judgmental standard-setting techniques may discourage practitioners from applying these techniques to other important organizational standard-setting efforts, such as defining minimal qualifications and setting standards for job performance. Third, the Angoff technique has been criticized on the basis that subject matter experts (SMEs) often report difficulty in making the judgments required by the Angoff technique, such as defining a minimally competent candidate and estimating the marginal probability that a minimally competent candidate will answer the item correctly. Further, a large empirical research literature illustrates that people often do not make accurate probability judgments. Recent court cases have underscored the importance of conducting an understandable cut-score-setting procedure and having SMEs report confidence in their judgments (e.g., U.S. v. Vulcan Society, Lanning v. SEPTA). This workshop should be of interest to users and developers of tests and other types of assessments who would like to increase their understanding of and ability to implement more options for setting cut scores and establishing performance standards.
This workshop is designed to accomplish the following objectives:
• Instruct workshop participants in the basics of conducting judgmental standard setting procedures other than the Angoff technique, such as item mapping methods and the body of work method
• Describe the data requirements for each method
• Discuss critical elements of leading a standard setting panel, such as getting SME buy in and providing well-developed performance standards
• Broaden participants’ thinking about using judgmental standard setting techniques for assessments other than preemployment tests
Dr. Steve Ferrara has almost 30 years of experience in operational educational measurement programs. Since 1992, he conducted numerous standard settings using the Angoff, contrasting groups, borderline groups, bookmark, item-descriptor matching, body of work, and reasoned judgment methods. He originated and developed the item-descriptor matching method and has published research on articulating performance standards to enable inferences about achievement growth. Dr. Ferrara conducted measurement research at AIR for over 10 years prior to joining CTB. He received his PhD in educational psychology from Stanford University.
Dr. Lorin Mueller has nearly 15 years of experience setting standards and cut scores on a variety of assessments, both in employment settings and educational measurement settings. His work at the American Institutes for Research has given him the opportunity to research and apply multiple standard-setting techniques in high-stakes settings, including developing the simulated contrasting groups procedure. Prior to joining the AIR, Dr. Mueller worked as a consultant to several petrochemical companies, helping to set standards on preemployment and training measures using the Angoff method as well as empirical standard setting methods. He received his PhD in I-O psychology from the University of Houston and holds a Senior Professional in Human Resources certification.