Below some graphs are depicted that may put practical merits into perspective of the statistical concept of ' significance'.
This in light of the actual purpose of statistical testing, which is to assess whether a hypothesis about a causal relation can reasonably be included in a theory. For the latter it is necessary to prove that the causal hypothesis has sufficient predictive power. A minimal criterion, or precondition, is that the theorem at least correctly predicts better than random, that is more then 50 percent, in the empirical world.

Curiously, on this point, we see an immense problem in many instances of scientific research, particularly in social sciences: widespread misconceptions - and frequent abuse - of significance. In short, a statistically significant outcome is often interpreted as evidence for the presence of a 'significant' causal relation, while it only points to a 'significant' deviation from mere coincidence in purely linear covariance of two or more sets of numbers.
As proof or indication of predictive power it is rather irrelevant.
The only legitimate interpretation of statistical significance of a given statistical measure is exactly the opposite: if there is no sufficient statistical significance we can not reasonably assume any meaningful statistical (i.e., computational) explanatory or predictive value based on the measurement data used, still less a real causal relationship in the relevant referential domain.
(And even then, two caveats are in order here: (a) significance can only be determined on the basis of a predefined value (alpha) of the maximum probability of chance accepted, which is essentially arbitrary and does not allow for rigorous conclusions; (b) the fact that a conclusion cannot be derived for the time being does not at all mean that the conclusion is thereby refuted).

The ratios between various statistical measures of predictive power have been elaborated in a number of detailed number tables. See eg for a summary overview:
' Predictive power of sample correlation for deduction to N=1 samples - Matrix (9) Values of first significance, optimized'.

Below some of those table data are put on a graph, in three parts. This graph provides clear insight into the huge problem at a single glance.

Subgraphs 1 to 3 show on the X-axis the spectrum of the obtained (first) sample correlation.
(N.b., correlation is no more than the degree of symmetric variation between two or more sets of numbers; it does not prove anything about a causal relationship. On the contrary, the converse applies: a causal relationship implies or requires that there is sufficient correlation between the values of cause and effect. In other words: if we can't demonstrate a correlation of any notable size, we can reasonably assume that there is no causal relationship ).

For the purpose of illumination, a reasonably customary sample size has been taken: N = 100. With other sample sizes, the ratios do not change in essence.

On the Y-axes of the subgraphs statistical results are represented, respectively:

The general practice - in the social sciences, but also media, opinion pollers, etc. - is that the outcome concerning statistical significance (ad 1a) is used rather extremely incorrectly:
- applied in theories as if it were (2b).
- applied in practical applications as if it were (3b).

Graph: Predictive power of sample correlation for N = 1 applications.

.
(at

N

=100, alpha =0.05, epsilon =0.05).

Conclusions

The usefulness of the statistical concept of significance as a criterion to test the validity of causal hypotheses, in the sense of sufficient deviation from pure chance, appears to be very limited. She can not in a reasonable way serve to confirm the validity of causal hypotheses in a positive sense.
The most meaningful applications lie on two points:

Zie ook ..

C.P. van der Velde © 2016, 2018.