permutation tests are robust

The recent replication crisis has led to a number of ad hoc suggestions to decrease the chance of making false-positive findings. Among them, Johnson (2013) and Benjamin et al. (2018) recommend using the significance level of alpha = 0.005 as opposed to the conventional 0.05 level. Even though their suggestion is easy to implement, it is unclear whether or not the commonly used statistical tests are robust and/or powerful at such a small significance level. Therefore, the main aim of our study is to investigate the robustness and power curve behaviors of independent (unpaired) two-sample tests for metric and ordinal data at nominal significance levels of alpha = 0.005 and alpha = 0.05. Through an extensive simulation study, it is found that the permutation versions of the Welch t-test and the Brunner-Munzel test are particularly robust and powerful while the commonly used two-sample tests which utilize t-distribution tend to be either liberal or conservative and have peculiar power curve behaviors under skewed distributions with variance heterogeneity.