A critical review of five language washback studies from 1995-2007:
by Yi-Ching Pan
Keywords: washback, examination consequences, test validity, construct validity, consequential validity
|"Washback has become a focal point of validity research . . ."|
[ p. 2 ]
|Studies:||Shohamy, et al. (1996)||Alderson & Hamp-Lyons (1996)||Cheng (1999)||Green (2007)||Shih (2007)|
|Exams studied:||An Israeli ASL test & ESL test||TOEFL®||Old & New HKCEE||IELTS Writing Test||GEPT|
|Purposes:||To examine the impact of 2 national tests in and beyond classroom settings||To ascertain influence of the TOEFL on class teaching||To compare teachers' perceptions toward both exams||To examine how preparation classes impact score gains||To explore the effects of GEPT exit requirements on learning|
|Methodologies:||1. Student questionnaires
2. Structured interviews with teachers and inspectors
3. Analysis of inspectorate bulletins
|1. Interviews with teachers and students
2. Classroom observations
|1. Teacher/student questionnaires
2. Structured interviews with teachers
3. Classroom observations
|1 . Two IELTS writing tests
2. Two questionnaires consisting of participant and process variables respectively
|1. Interviews with department heads, teachers, students, and family members
2. Classroom observations
|Collected evidence:||1. More positive washback found in ESL
2. More negative washback found in ASL
|1. More occurrences of teacher talk, the use of meta language in non-TOEFL classes
2. Fewer opportunities for pair work, laughter, and turn-taking in TOEFL classes
|1. An increased change in teaching content and activities
2. A lack of change in teaching methodologies
|An improvement in test scores for learners in test-preparation or academic-oriented classes, but those in the former progressed no more than those in the latter||1. Small but varied aspects of washback found in students at both schools with and without exit requirements
2. External, intrinsic and test factors explain GEPT's minor impact on students' learning
|Conclusions:||Washback changes over time because of factors including language status and test uses.||TOEFL affects both what and how teachers teach, but the effect varies with teachers.||The change on teaching content rather than methodology was attributed to inadequate training and qualifications of secondary English teachers.||Test preparation classes have no apparent benefit to improve test scores.||The current washback theory didn't account for GEPT washback, so a new learning washback model has been developed.|
[ p. 3 ]
[ p. 4 ]Cheng, however, determined from the questionnaires that although most teachers felt positively about the revised exam that enabled students to use English more practically and authentically, no major changes emerged in terms of actual pedagogic practices, which are still content-based and teacher-centered. The content of what was taught now focuses more on listening and speaking in accordance with the revised exam. As Cheng stated, "the change of the HKCEE toward an integrated and task-based approach showed teachers the possibility of something new, but it did not automatically enable teachers to teach something new" (p. 164). Cheng's study confirms Wall and Alderson's (1993) previous findings: while classroom content may change because of a test, the way teachers instruct does not change to any significant degree. The changes noted by Cheng (2005, p. 235) were "superficial".
[ p. 5 ]Interviews with 2 department heads, 6 teachers, 30 students, and 3 family members were conducted. Observations were made for a semester in test-preparation classes or in classes that taught skills tested on the GEPT. Departments' policies regarding the GEPT exit requirements were also reviewed. Findings indicated that the GEPT had elicited a varying but minor impact on learners at both schools, although a slightly higher degree of washback was found at the school with exit requirements. In addition, Shih generated a new washback model of students' learning, as illustrated in Figure 1. This model includes extrinsic, intrinsic, and test factors to help depict the complexity of learning washback.
[ p. 6 ]
[ p. 7 ]A second limitation of Alderson and Hamp-Lyons's study was their choice of participants. Alderson and Hamp-Lyons point out that the TOEFL affects both what and how teachers teach, but the effect differs considerably from teacher to teacher. However, given the varying backgrounds and amounts of teaching experience of the participants (a material developer with seventeen years of experience versus a first-time material teacher who has taught a TOEFL class only once), the disparities of the effects come as no surprise (Saif, 1999). It would be worthwhile to determine whether those effects are similar among teachers with comparable backgrounds.
|". . . washback is a complex phenomenon that involves a variety of intervening variables . . ."|
[ p. 8 ]Cheng's baseline study, by focusing on what occurred before the administration of the test and making a comparison between classroom activities and teachers' perspectives under the syllabus of both the old and new HKCEEs, helps us better understand what has changed. Qi (2004) points out that washback studies usually suffer from a lack of data collected before the test was first introduced. Cheng has thus provided us with a good starting point for more research. However, a longitudinal study with a longer timeframe than the one used by Cheng might shed better light on the effects of the new HKCEE. As Messick (1989) claims, the effects of tests on societies and educational systems only becomes apparent after a while.
if it is more generally found to be the case that 'teaching to test' is no more effective in boosting test scores than teaching the targeted skills, this will have profound implications for the relationship between teaching and testing. (p. 94)Washback on learning outcomes is a complicated issue. It often is difficult to detect whether washback is due primarily to a test-prep course itself or other factors such as motivation, learning experience, and age. Green's research focuses on both participant and process variables, providing a comprehensive list for factors thought to influence learning outcomes. Those factors can be used in questionnaires for those interested in exploring similar topics.
[ p. 9 ]In addition, it has provided us with a comprehensive list of extrinsic, intrinsic, and test factors that assist in the explanation of the intricacy of learning washback, while the previous washback theories of Alderson and Wall (1993) Bailey (1996)'s, and Hughes (1993) seem too simplistic in this respect. Shih's model also contributes to the explanation of how tests influence students' learning, especially applied to East-Asian contexts — foreign language education in Korea, Japan and Taiwan is remarkably similar.
|". . . most washback studies cover test effects on classroom settings or the educational contexts, while little attention is devoted to society at large."|
[ p. 10 ]However, we should remember that Hughes (2003) defines washback as the effects of testing not only on learners and teachers in a given educational context, but also on society at large. In addition, Bachman and Palmer (1996) have stressed that both classroom micro effects and macro social and educational system effects need to be examined. In light of this, overviews of washback should address both micro and macro levels.
[ p. 11 ]For example, to better understand why teachers change what they teach but not necessarily their methodology (Cheng, 1999, 2004, 2005) following the introduction of a test, their beliefs, perceptions of the test, and their levels of participation in its implementation may help us understand the phenomena of washback.
|"To gauge micro- and macro washback levels of washback, a triangulation of questionnaires, interviews, observations, pre-and-post tests, and document analysis need to be conducted."|
[ p. 12 ]References
[ p. 13 ]Green, A. (20007). Washback to learning outcomes: a comparative study of IELTS preparation and university pre-sessional language courses. Assessment in Education, 14 (1), 75-97.
[ p. 14 ]Stoneman, B. W. H. (2006). The impact of an exit English test on Hong Kong undergraduates: A study investigating the effects of test status on students' test preparation behaviours. Unpublished Ph. D. Dissertation, The Hong Kong Polytechnic University, Hong Kong, China.