A New Approach to Multiple Hypothesis Testing: Finite-Sample Strong Control with Arbitrary Dependency

Gengyuan Zhang, University of Pennsylvania


Multiple (simultaneous) hypothesis testing is key for avoiding "data dredging'' with repeated single hypothesis testing across competing hypotheses. But, existing multiple testing procedures (MTP's) face serious limitations. In small (finite) samples common to social sciences, existing MTP's assume independent statistics, e.g., uncorrelated returns to different trading strategies. They produce conservative control of Type I errors in weak control problems (where all nulls are true) and limited power in strong control problems (with a mixture of true and false nulls). Large (asymptotic) sample procedures allow for general dependence, greatly expanding testing applications. But they fail to produce control in smaller samples, or when the number of tests increases with the amount of data. They face other technical limitations and are otherwise unrelated to small sample procedures. We present a new method (NM) MTP and a generalized sequential step-down version (NM-GS) that produces strong control and accurate weak control even in a small sample with an assumed but arbitrary dependency of test statistics. NM and NM-GS nest previous small-sample MTP's. Assumed dependency can be replaced with an empirical sample estimate in a large sample, producing control similar to existing procedures but with fewer assumptions. Being parametric, obtaining control requires less data relative to the number of hypotheses. NM and NM-GS create a unified framework between small and large samples, even supporting hybrid solutions.

Subject Area

Statistics|Applied Mathematics|Finance

Recommended Citation

Zhang, Gengyuan, "A New Approach to Multiple Hypothesis Testing: Finite-Sample Strong Control with Arbitrary Dependency" (2020). Dissertations available from ProQuest. AAI27961226.