I just listened to a talk about the potential benefits of computerized psychological assessment. Besides simplifying data managment and scoring, there are some things that can (almost) only be done with a computer, like adaptive testing or reaction time measurement — the overwhelming majority of studies in cognitive psychology depends on reaction time measures. But I also experienced a slight discomfort in listening to that talk. The speaker gave an example of the kind of research I react allergically to in psychological assessment: He devised a test which he called “visual comprehension” (in German it was actually “Seh-Verständnis” which is not quite identical with “visual comprehension”, but I can’t come up with a better translation). Taking that test, you will be shown short educational videos (4-5 minutes) for the natural sciences; there’s a speaker explaining, and what he is talking about is illustrated by the visual content. Afterwards you will be asked several questions about the content of the video.
So far no problem. But what I don’t like is that this is called “visual comprehension”. Performance in such a task will depend only slightly on visual abilities (as opposed to auditory, say). You can only watch the video once and you won’t be able to re-view portions of the video when you have to answer the questions. So I would say much of the performance depends on memory systems; how much you are able to memorize things while watching, and how good you are at recalling things when confronted with the question. In any textbook on learning and memory you will find that performance in such tasks depends greatly on previous subject knowledge: whenever you already know something about the subject at hand, it will facilitate encoding, because you already know what to look for, and retrieval, which can be explained by associationist models of memory, where specific nodes will be more easily activated the more they are connected to other nodes and the more such connections have been active in the past.
So how much does this test of “visual comprehension” differ from more traditional tests of learning abilities, as for example tests of “reading comprehension”, which have been included in international assessment programmes such as PISA (there called “reading literacy”)? Not much, you might have guessed already, and that is also what the speaker found out. However, it would obviously not be valid to say “visual comprehension is highly correlated with reading comprehension”, because it is just stupid to talk of “visual comprehension” for the kind of test used. But it is a general problem in psychological assessment that people design a test, then give it a name, and then make claims such as “[insert test name] correlates highly with [insert construct of choice]!”.
Sometimes you really should think before you act, or devise some clever test.