The Highs & Lows of Double-Blind Testing Page 10
The Double-Blind Debate #1
Editor: In JA's "Two-Cents' Worth" conclusion to "The Double-Blind & the Not-so-Blind" (Vol.9 No.5), he pondered the existence of an audio subculture believing that most pairs of similarly described audio components sound the same. I, for one, can tell you why I am a member of that subculture. Due to the margins for error correctly pointed out by Mr. Leventhal, I cannot at this time demand your acknowledgment that my audio subculture is not as blind as you say. However, I would like you to consider the counterproductive mudslinging tone of the article and its title. There is no benefit to audio in making Mr. Clark and Mr. Leventhal appear to be enemies.
Nor is my conclusion that amplifiers sound alike drawn from the statistical tests applied to the controlled listening test results, or from excessive blindness or deafness; rather, it is a personal one, drawn from being a subject in controlled test after controlled test. I learned from listening avidly and intently, believing each amplifier sounded different, hearing the difference during the test, confidently identifying each amplifier, and seeing my score tallied up. I was wrong as often as I was right. This did not happen just once. Time after time I thought I heard a difference, but was wrong.
I stopped testing amplifiers by ear years ago because I was wrong so often that there was no hope of proving a difference. But I am not deaf, nor do I have a tin ear. I hear small amounts of distortion and subtle frequency-response deviations as well as the best "golden" ears we've found.
How, then, do we explain the adamancy of a golden ear on the one hand, and my own confidence that amplifiers sound alike on the other?
The human brain is best at making sense out of nonsense. Humans tend to find differences and distinctions whether they exist or not. The research of Richard M. Warren, Diana Deutsch, and others, confirms that humans can decipher a word obscured by noise as much as a minute after a sentence was spoken (footnote 8). This same potential can, at times, create the wrong word. We can misunderstand and still believe firmly we heard a word different than the one spoken. This is not a defect. It allowed our ancestors to survive by detecting threats through noise. Sometimes they overreacted and called out defences when no mastodon approached. This did no harm.
This human power to make sense out of nonsense even when it is nonsense is what makes the golden ear so confident. I have recognized that, in the case of amplifiers, my ears and brain give me the signals for difference when there is none. My reasoned decision that amplifiers do not sound different does not influence these human signals for difference. Only last night I sat in on a double-blind test being set up. I heard the cues for difference and wrote down my identifications of the amplifiers. My responses were random once again.
Only the most disciplined scientist would believe the truth of the situation from the published statistics or the statistics of Dr. Leventhal. Most people are not disciplined scientists; certainly most readers of Stereophile are not. A confident golden ear is unlikely to believe that the differences he hears are not real unless he experiences the reality of controlled testing by himself. Likewise, confidence that a given comparison is set up to the standards of the listener who has learned to require cartridge "A" and interconnects "C" is practically accomplished only by that listener setting up the test himself.
It is to this end that the ABX Company was founded. Those who have used ABX Company equipment have found, as we have, that the exciting but dull-sounding conclusion that most equipment sounds alike is true. For the skeptics there is our longstanding challenge to attend a double-blind test and satisfy themselves that the testing is being done correctly and to their standards.
Yes, my conclusion that all amplifiers sound alike is a personal one, and I cannot ask you to take my word for it. The statistics so far published are good and lend support to my belief, but do not have sufficient strength of numbers to satisfy Mr. Leventhal. If the statistics did satisfy Mr. Leventhal's criteria, I would be the first to ask you to believe as I do on that evidence alone.
My claim is that you, the golden ear, cannot ask me to believe you without tests controlled by modern science. Your blind tests and Mr. Colloms' blind tests carried out by the UK section of the AES (footnote 9) do not satisfy me. The double-blind criteria is far more rigorous. I, too, cannot ask you to believe me without such evidence. As Mr. Leventhal explains, it is the nature of the statistics, for it to be far easier for you to obtain proof you heard something than for me to prove you did not.
In the absence of that evidence, we have two alternatives:
(1) Mr. Atkinson takes part in sufficient controlled double-blind tests to produce evidence to support his beliefs or to convince himself that he is not right often enough to continue, or
(2) I persevere with enough tests to convince Mr. Leventhal that the best listeners we can find cannot hear the difference between modest and "esoteric" amplifiers, and then I ask you to believe me.
In (2), it is unlikely that you will believe me. (1) has the advantage that you, the listener, are confident, whereas I am jaded and would have trouble doing more tests. Recall that I have had your golden-ear experience and found myself deceived. You have not had my experience in controlled tests that satisfied my every condition. I hope you will not ignore the possibility that you are crying "mastodon" and start to take science seriously enough to begin double-blind testing for your magazine.—David Carlstrom, ABX Company, Huntington Woods, MI
Footnote 8: Warren, Richard M., "Auditory Illusions and their Relation to Mechanisms Enhancing Accuracy of Perception," Journal of the Audio Engineering Society, Vol.321 No.9, September 1983.
Footnote 9: Which produced positive discrimination between nominally identical amplifiers.—JA