Skip to content

Treating a p-value as the probability of the null

The statement “p = 0.02 means the null is 2% likely” introduces a probability the test never calculated.

On this page

Identify the missing model inputs

A test starts with a null reference model and calculates a statistic’s tail area under it. It has not assigned prior probabilities to the null and alternatives. It has not specified a distribution of possible effects under an alternative.

Rewrite the sentence without reversing it

For a fictional upper normal test with p = 0.02, a suitable description is that the null model assigns probability 0.02 to a statistic at least as large as the observed one. The statement depends on the specified model and planned upper direction.

List the separate posterior ingredients

Questions a posterior analysis would need
IngredientQuestion
PriorWhat probability is assigned before this data?
Null likelihoodHow is the observed data modeled under the null?
Alternative likelihoodHow is it modeled under a specified alternative?
UpdatingHow are those pieces combined?

Keep the reported effect visible

A probability interpretation alone does not describe the estimated effect’s magnitude or usefulness. Report the estimate, uncertainty and design next to the p-value. Do not replace the mistaken 2% claim with an equally unsupported 98% probability of profit.

Further references