Simulation-based tests
There is only one test
We can use t.test() to check whether two means are equal or not. Instead of dealing with the assumptions of the data and finding the appropriate statistical test to test for equality of variance, we can use the power of bootstrapping, permutation, and simulation to construct a null distribution, calculate confidence intervals and run any kind of test for inference.
The beauty and power of simulation is that we can run hypothesis test not just on differences of means that we have fomuals for, but also on other parameters, like medians
According to Allen Downey, there is only one statistical test and that all statistical tests follow the same pattern:
- Step 1: Calculate a sample statistic from the observed data, or \(\delta^{*}\). This is the measure you care about: the difference in means, the mean, the median, the proportion, the difference in proportions, anything you want really!
- Step 2: Use simulation to invent a world where \(\delta\) is null. Simulate what the world would look like if there was no difference between two groups, or if there was no difference in medians or proportions, or where the average value is a specific number.
- Step 3: Look at \(\delta^{*}\) in the null world. Put the observed sample statistic in the null world and see if it fits well.
- Step 4: Calculate the probability that \(\delta^{*}\) could exist in null world. This is the p-value, or the probability that you’d see a \(\delta^{*}\) at least that high in a world where there’s no difference.
- Step 5: Decide if \(\delta^{*}\) is statistically significant. Choose some threshold, cutoff value (e.g., 0.05) for deciding if there’s sufficient proof for rejecting the null world.