Hypothesis Testing With The Bootstrap

Hypothesis Testing with theBootstrapNoa HaasStatistics M.Sc. Seminar, Spring 2017Bootstrap and Resampling Methods

Bootstrap Hypothesis TestingA bootstrap hypothesis test starts with a teststatistic - 𝑡(𝒙) (not necessary an estimate of aparameter).We seek an achieved significance level𝐴𝑆𝐿 𝑃𝑟𝑜𝑏𝐻0 𝑡 𝒙 𝑡(𝒙)Where the random variable 𝒙 has a distributionspecified by the null hypothesis 𝐻0 - denote as 𝐹0 .Bootstrap hypothesis testing uses a “plug-in” styleto estimate 𝐹0 .

The Two-Sample ProblemWe observe two independent random samples:𝐹 𝐺 𝒛 𝑧1 , 𝑧2 , , 𝑧𝑛 �� 𝑜𝑓𝒚 𝑦1 , 𝑦2 , , 𝑦𝑚And we wish to test the null hypothesis of no differencebetween F and G,𝐻0 : 𝐹 𝐺

Bootstrap Hypothesis Testing 𝐹 𝐺 Denote the combined sample by 𝒙, and its empiricaldistribution by 𝐹0 . Under 𝐻0 , 𝐹0 provides a non parametric estimate forthe common population that gave rise to both 𝒛 and 𝒚.1. Draw 𝑩 samples of size 𝑛 𝑚 with replacementfrom 𝒙. Call the first n observations 𝒛 and theremaining 𝑚 – 𝒚 2. Evaluate 𝑡( ) on each sample - 𝑡(𝒙 𝒃 )3. Approximate 𝐴𝑆𝐿𝑏𝑜𝑜𝑡 by𝐴𝑆𝐿𝑏𝑜𝑜𝑡 # 𝑡 𝒙 𝒃 𝑡 𝒙 /𝐵* In the case that large values of 𝑡(𝒙 𝒃 ) are evidence against 𝐻0

Bootstrap Hypothesis Testing 𝐹 𝐺on the Mouse DataA histogram of bootstrapreplications of𝑡 𝒙 𝑧 𝑦for testing 𝐻0 : 𝐹 𝐺 on themouse data. The proportion ofvalues greater than 30.63 is .121.Calculating𝑧 𝑦𝑡 𝒙 𝜎 1 𝑛 1 𝑚(approximate pivotal) for thesame replications produced𝐴𝑆𝐿𝑏𝑜𝑜𝑡 .128

Testing Equality of Means Instead of testing 𝐻0 : 𝐹 𝐺, we wish to test H0 : 𝜇𝑧 𝜇𝑦 ,without assuming equal variances. We need estimates of 𝐹and 𝐺 that use only the assumption of common mean1. Define points 𝑧𝑖 𝑧𝑖 𝑧 𝑥, 𝑖 1, , 𝑛, and 𝑦𝑖 𝑦𝑖 𝑦 𝑥 , 𝑖 1, , 𝑚. The empirical distributions of 𝒛 and 𝒚shares a common mean.2. Draw 𝑩 bootstrap samples with replacement 𝒛 , 𝒚 from𝑧1 , 𝑧2 , , 𝑧𝑛 and 𝑦1 , 𝑦2 , , 𝑦𝑚 respectivly3. Evaluate 𝑡( ) on each sample 𝑦 𝑧𝑡 𝒙 𝒃 𝜎𝑧 1 𝑛 𝜎𝑦 1 𝑚4. Approximate 𝐴𝑆𝐿𝑏𝑜𝑜𝑡 by𝐴𝑆𝐿𝑏𝑜𝑜𝑡 # 𝑡 𝒙 𝒃 𝑡 𝒙 /𝐵

Permutation Test VS BootstrapHypothesis Testing Accuracy: In the two-sample problem, 𝐴𝑆𝐿𝑝𝑒𝑟𝑚 isthe exact probability of obtaining a test statisticas extreme as the one observed. In contrast, thebootstrap explicitly samples from estimatedprobability mechanism. 𝐴𝑆𝐿𝑏𝑜𝑜𝑡 has nointerpretation as an exact probability. Flexibility: When special symmetry isn’t required,the bootstrap testing can be applied much moregenerally than the permutation test. (Like in thetwo sample problem – permutation test is limitedto 𝐻0 : 𝐹 𝐺, or in the one-sample problem)

The One-Sample ProblemWe observe a random sample:𝐹 𝒛 𝑧1 , 𝑧2 , , 𝑧𝑛And we wish to test whether the mean of thepopulation equals to some predetermine value 𝜇0 𝐻0 : 𝜇𝑧 𝜇0

Bootstrap Hypothesis Testing 𝜇𝑧 𝜇0What is the appropriate way to estimate the nulldistribution?The empirical distribution 𝐹 is not anappropriate estimation, because it does notobey 𝐻0 .As before, we can use the empirical distributionof the points:𝑧𝑖 𝑧𝑖 𝑧 𝜇0 , 𝑖 1, , 𝑛Which has a mean of 𝜇0 .

Bootstrap Hypothesis Testing 𝜇𝑧 𝜇0The test will be based on the approximate𝑧 𝜇0distribution of the test statistic 𝑡 𝒛 𝜎/ 𝑛We sample 𝑩 times 𝑧1 , , 𝑧𝑛 with replacementfrom 𝑧1 , , 𝑧𝑛 , and for each sample compute𝑧 𝜇0 𝑡 𝒛 𝜎/ 𝑛And the estimated ASL is given by𝐴𝑆𝐿𝑏𝑜𝑜𝑡 # 𝑡 𝒛 𝒃 𝑡 𝒛 /𝐵* In the case that large values of 𝑡 𝒛 𝒃 are evidence against 𝐻0

Testing 𝜇𝑧 𝜇0 on the Mouse DataTaking 𝜇0 129, the observed value of the test statistic is86.9 129𝑡 𝒛 1.6766.8/ 7(When estimating 𝜎 with the unbiased estimator for standarddeviation). For 94 of 1000 bootstrap samples, 𝑡 𝒛 wassmaller than -1.67, and therefor𝐴𝑆𝐿𝑏𝑜𝑜𝑡 .094For reference, the student’s t-test result for the same nullhypothesis on that data gives us42.1𝐴𝑆𝐿 𝑃𝑟𝑜𝑏 𝑡6 0.0766.8/ 7

Testing Multimodality of a PopulationA mode is defined to be a local maximum or “bump” of thepopulation densityThe data: 𝑥1 , , 𝑥485 Mexican stamps’ thickness from 1872.The number of modes is suggestive of the number of distincttype of paper used in the printing.

Testing Multimodality of a PopulationSince the histogram is not smooth, it is difficult to tell from it whether there aremore than one mode.A Gaussian kernel density with window size ℎ estimate can be used in order toobtain a smoother estimate:1𝑓 𝑡; ℎ 𝑛ℎ𝑛𝑖 1𝑡 𝑥𝑖𝜙ℎAs 𝒉 increases, the number of modes in the density estimate is non-increasing

Testing Multimodality of a PopulationThe null hypothesis:𝐻0 : 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑜𝑑𝑒𝑠 1Versus 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑜𝑑𝑒𝑠 1. Since the number of modes decreases asℎ increases, there is a smallest value of ℎ such that 𝑓 𝑡; ℎ has one mode.Call it ℎ1 . In our case, ℎ1 .0068.

Testing Multimodality of a PopulationIt seems reasonable to use 𝑓(𝑡; ℎ1 ) as the estimated nulldistribution for our test of 𝐻0 . It is the density estimate thatuses least amount of smoothing among all estimated withone mode (conservative).A small adjustment to 𝑓 is needed because the formula artificiallyincreases the variance of the estimate with ℎ12 . Let 𝑔 ; ℎ1 be the rescaleestimate, that imposes variance equal to the sample variance.A natural choice for a test statistic is ℎ1 - a large value of ℎ1 isevidence against 𝐻0 .Putting all of this together, the achieved significance level is𝐴𝑆𝐿𝑏𝑜𝑜𝑡 𝑃𝑟𝑜𝑏𝑔 ;ℎ1 ℎ1 ℎ1Where each bootstrap sample 𝒙 is drawn from 𝑔 ; ℎ1

Testing Multimodality of a PopulationThe sampling from 𝑔 ; ℎ1 is given by:1 ℎ12 𝜎 2 2𝑥𝑖 𝑥 1 𝑦𝑖 𝑥 ℎ1 𝜖𝑖 ; 𝑖 1, , 𝑛Where 𝑦1 , , 𝑦𝑛 are sampled with replacement from 𝑥1 , , 𝑥𝑛 , and𝜖𝑖 are standard normal random variables. (called smoothedbootstrap)In the stamps data, out of 500 bootstrap samples, none hadℎ1 .0068, so 𝐴𝑆𝐿𝑏𝑜𝑜𝑡 0.The results can be interpreted in sequential manner, moving on tohigher values of the least amount of modes. (Silverman 1981)When testing the same for 𝐻0 : 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑜𝑑𝑒𝑠 2, 146samples out of 500 had ℎ2 .0033, which translates to 𝐴𝑆𝐿𝑏𝑜𝑜𝑡 0.292.In our case, the inference process will end here.

SummaryA bootstrap hypothesis test is carried out using the followings:a) A test statistic 𝑡(𝒙)b) An approximate null distribution 𝐹0 for the data under 𝐻0Given these, we generate 𝐵 bootstrap values of 𝑡(𝒙 ) under 𝐹0 andestimate the achieved significance level by𝐴𝑆𝐿𝑏𝑜𝑜𝑡 # 𝑡 𝒙 𝒃 𝑡(𝒙) /𝐵The choice of test statistic 𝑡 𝒙 and the estimate of the null distributionwill determine the power of the test. In the stamp example, if the actualpopulation density is bimodal, but the Gaussian kernel density does notapproximate it accurately, then the suggested test will not have highpower.Bootstrap tests are useful when the alternative hypothesis is not wellspecified. In cases where there is parametric alternative hypothesis,likelihood or Bayesian methods might be preferable.

Statistics M.Sc. Seminar, Spring 2017 Bootstrap and Resampling Methods . Bootstrap Hypothesis Testing A bootstrap hypothesis test starts with a test statistic - P( ) (not necessary an estimat