UNDERSTANDING OF SIGNIFICANCE LEVEL AS CONDITIONAL PROBABILITY
Hashimoto, T. and Shigemasu, K., University of Tokyo, Japan
Significance level and p-value are conditional probabilities of observed values given null hypothesis. However, it is often interpreted as a probability of null hypothesis given observed values. People often confuse conditional probabilities, and one typical kind of confusion is called "base-rate fallacy". Base-rate fallacy is people's tendency to ignore base rates in favor of individuating information (when such is available), rather than integrate the two (Bar-Hillel 1980). One of the methods which can improve such fallacy is "frequency formats" (Gigerenzer and Hoffrage 1995), which uses integers or irreducible fractions rather than decimals or percentages to express probabilities. We assumed that frequency format is applied to improve the comprehension of significance level and p-value. First, we investigated how the university students interpret significance level and p-value as conditional probability. We found that even though they took statistics course, about a half of them misunderstood them. Namely, they interpreted them in the wrong way. Next, we investigated whether this misunderstanding is related to other probability fallacies, but we did not find the correlations among them. Finally, we investigated whether the frequency format helped the students to correct their misunderstanding, but we could not find an improvement in their understanding. Our conclusion is some statistical concepts such as significance level or p-values have unique features and should be taught more carefully than just a probability fallacy.