I was reading one of my copies of Skeptic Magazine, Vol. 16, No. 1 for 2010, and came across an article entitled, The Saturn-Mars Effect: How a Statistical Effect Explains the Astrological Claim for the Power of Mars.
It involved a statistical study using a computer program that looks for correlations in data sets, and which made use of an issue in the field of Genome-Wide Association Studies (GWAS), known as the problem of multiple comparisons, in which a misleading causative relationship can be alluded to in any statistical analysis involving genes and gene-sequences and the diseases they might code for used in the study, only some of them known and controlled for.
It noted that with any such study in which not all of the potential factors can be known of and controlled for, there is a very real danger of making a spurious causal inference, a Type 1 error, or false-positive, based on the factors not so controlled that could easily produce what seems to be statistically significant p values that are in fact worthless as evidence of a real effect.
This got me thinking about the data set of parapsychology, which interestingly enough consists solely of statistical anomalies from literally thousands of experiments over a period of more than 100 years. While the level of significance of the better done studies is superficially convincing, confirming the psi hypothesis is just as elusive as ever.
Not only have replication attempts by nonbelievers in psi proven ineffective, but even those conducted by believers conscientious enough to improve controls for previously unaccounted for flaws in the study predictably reduce the statistical significance to vanishingly within what could be expected by chance.
Regardless of one’s position on the existence of certain things, and this is true of meta-analyses as well, it’s easy to mislead oneself into equating a seeming numerical quirk in the data as constituting a genuine effect, especially a novel phenomenon not yet explained…or explainable…by science.
This is especially the case in large collections of data from multiple studies with large sample sizes, in which one could reasonably expect flukes in the results that even with adequate randomization could still be the result of thousands of different factors not involving anything outside of science…
…And in any study conducted with limited time and resources, it is simply not possible to account for all of these and rightly claim that they establish beyond rational doubt anything real, because of one underlying flaw in the psi hypothesis: the fact that even after over a century, there is no coherent consensus on a positive theory to tell us what we should expect to find if psi exists, where we should look, what we should fail to see if it doesn’t, or what its limits are.
Psi is defined by what it isn’t, not by what it is…
Until such a theory is proposed and tested, and nonbelievers can successfully replicate the results of such testing, the entire data set of parapsychology from over more than a century implies nothing outside of the normal, and is of no real value in supporting claims of anything beyond the mundane.