Statistical Significance Testing From Scratch

Let Zμ ∼ N (μ, 1) with unknown μ. Our “alternative” hypothesis is H1 : μ 6= 0 We want to decide if we should accept H1 based on a single sample zμ of Zμ. Since we don’t care about the exact nonzero value of μ that makes H1 true, we may consider a proof by contradiction. A null hypothesis is a statement that is false iff H1 is true, in this case H0 : μ = 0 Now we need to decide if we should reject H0 based on zμ. There are two possible errors. • Type I error: We accept a false H1 (i.e., reject a true H0). • Type II error: We reject a true H1 (i.e., accept a false H0). We want to especially avoid a type I error. To this end, we introduce a hyperparameter α ∈ (0, 1) called a significance level. We will define RejectNullα : R → {0, 1} that maps zμ to 1 iff it rejects H0 such that the associated type I error probability is α. Formally,