Even When Test Scores Go Up, Some Cognitive Abilities Don’t
To evaluate school quality, states require students to take standardized tests designed to measure the knowledge and skills that students have acquired in school — what psychological scientists refer to as “crystallized intelligence.” These high-stakes tests have been shown to predict students’ future educational attainment and adult employment and income.
But new research shows that schools whose students have the highest gains on test scores do not produce similar gains in “fluid intelligence” — the ability to analyze abstract problems and think logically.
Data from nearly 1,400 eighth-graders in the Boston public school system showed that some schools have successfully raised their students’ scores on the Massachusetts Comprehensive Assessment System (MCAS). However, those schools had almost no effect on students’ performance on tests of fluid intelligence skills, such as working memory capacity, speed of information processing, and ability to solve abstract problems.
“It doesn’t seem like you get these skills for free in the way that you might hope, just by doing a lot of studying and being a good student,” says psychological scientist and senior author John Gabrieli, the Grover M. Hermann Professor of Health Sciences and Technology, professor of brain and cognitive sciences, at MIT.
This study grew out of a larger effort to find measures beyond standardized tests that can predict long-term success for students. “As we started that study, it struck us that there’s been surprisingly little evaluation of different kinds of cognitive abilities and how they relate to educational outcomes,” says Gabrieli, who is also a member of MIT’s McGovern Institute for Brain Research.
The data for the Psychological Science study came from students attending traditional, charter, and exam schools in Boston. Some of those schools have had great success improving their students’ MCAS scores — a boost that studies have found also translates to better performance on the SAT and Advanced Placement tests.
The researchers calculated how much of the variation in MCAS scores was due to the school that students attended. For MCAS scores in English, schools accounted for 24 percent of the variation, and they accounted for 34 percent of the math MCAS variation.
However, the schools accounted for very little of the variation in fluid cognitive skills — less than 3 percent for all three skills combined.
In one example of a test of fluid reasoning, students were asked to choose which of six pictures completed the missing pieces of a puzzle — a task requiring integration of information such as shape, pattern, and orientation.
“It’s not always clear what dimensions you have to pay attention to get the problem correct. That’s why we call it fluid, because it’s the application of reasoning skills in novel contexts,” says Amy Finn, an MIT postdoc and lead author of the paper.
The researchers plan to continue tracking these students, who are now in 10th grade, to see how their academic performance and other life outcomes evolve. They have also begun to participate in a new study of high school seniors to track how their standardized test scores and cognitive abilities influence their rates of college attendance and graduation.
Implications for education
Gabrieli notes that the study should not be interpreted as critical of schools that are improving their students’ MCAS scores. “It’s valuable to push up the crystallized abilities, because if you can do more math, if you can read a paragraph and answer comprehension questions, all those things are positive,” he says.
He hopes that the findings will encourage educational policymakers to consider adding practices that enhance cognitive skills. Although many studies have shown that students’ fluid cognitive skills predict their academic performance, such skills are seldom explicitly taught.
“Schools can improve crystallized abilities, and now it might be a priority to see if there are some methods for enhancing the fluid ones as well,” Gabrieli says.
Adapted from an MIT News Office press release.
The research was a collaboration with the Center for Education Policy Research at Harvard University, Transforming Education, and Brown University.
In addition to Gabrieli and Finn, co-authors on the research include Matthew A. Kraft of Brown University, Martin R. West of Harvard University, Julia A. Leonard and Rebecca E. Martin of MIT, Crystal E. Bish of the National Center on Time & Learning, Margaret A. Sheridan of Children's Hospital Boston, and Christopher F. O. Gabrieli of Harvard Unviersity and the National Center on Time & Learning.
This research was funded by the Bill & Melinda Gates Foundation (to J.D.E and C.F.O Gabrieli) and the National Institute of Health (NRSA to A.S.F).