Sharper Confidence Intervals for Hochberg- and Hommel-Related Multiple Tests Based On an Extended Simes Inequality

This article concerns recent developments related to the now classical multiple-testing procedures (MTPs) of Holm, Hochberg, and Hommel based on marginal p-values. For a long time, the derivation of simultaneous confidence intervals (SCIs) corresponding to these MTPs was considered to be a difficult problem, but solutions were published in 2008 for Holm's MTP, and in 2012 for Hochberg's and Hommel's MTPs. These SCIs turned out to be as simple and easily implemented as the MTPs themselves, and to be remarkably similar. However, they also turned out to have the property/limitation, shared with other powerful stepwise MTPs, that no confidence assertions sharper than rejection assertions are possible unless all null hypotheses are rejected. A possibility is then to construct related families of MTPs that do not have this limitation but are somewhat less powerful, so users may choose among various such trade-off MTPs. It is shown in this article how an extended Simes inequality can be used to construct Hochberg- and Hommel-related MTPs of this kind that: (i) are more powerful than corresponding trade-off MTPs proposed previously, and (ii) lead to SCIs that are sharper than the ones proposed previously. Corresponding Holm-related MTPs and SCIs are considered for completeness and comparisons.


Introduction
This article concerns recent developments related to three now classical multiple-testing procedures (MTPs); namely Holm's (1979) step-down MTP, Hochberg's (1988) step-up MTP, andHommel's (1988) more powerful MTP that are based on marginal p-values p 1 , . . . , p m for the null hypotheses in a given family {H 1 , . . . , H m } of null hypotheses. These three MTPs control their Type-I family-wise error rate (FWER) in the strong sense, which nowadays is a regulatory requirement for confirmatory clinical trials aiming at supporting multiple claims.
Holm's MTP is more powerful than the Bonferroni MTP, and as generally valid as this simple MTP. Hommel's MTP is more powerful than Hochberg's MTP, which in turn is more powerful than Holm's MTP. Here "more powerful" is in the strong sense that for any outcome of p 1 , . . . , p m , at least as much is rejected, possibly more. However, an important aspect of Hommel's and Hochberg's MTPs in applications is that in contrast to Holm's MTP, they are not generally valid. Hommel's MTP is valid if the p-values p 1 , . . . , p m are independent or positively dependent in such a way that the Simes (1986) inequality holds for any given subset of p i 's. This condition on p-values is sufficient also for Hochberg's MTP to be valid. Much research has been devoted to determine situations under which the Simes inequality holds; see for example, Sarkar and Chang (1997) and Sarkar (1998Sarkar ( , 2008, with references. For a long time, the derivation of simultaneous confidence intervals (SCIs) that correspond to these three classical MTPs was considered to be a difficult problem. Here, correspond means that the SCIs imply the same rejections as those made by the MTP, and thus provide extra information "for free." A nice discussion can be found in Strassburger and Bretz (2008) of the state of the art before 2008 for Holm's MTP, and of certain incorrect SCIs previously proposed for that MTP that seem to have been widely used in practice. Solutions were, however, obtained by Guilbaud (2008) and Strassburger and Bretz (2008) for Holm's MTP, and by Guilbaud (2012) for Hochberg's and Hommel's MTPs.
The SCIs derived for these three MTPs turned out to be as simple and easily implemented as the MTPs themselves, and to be remarkably similar. However, these SCIs also turned out to have the property that no confidence assertions sharper than rejection assertions can be made unless all null hypotheses are rejected. This may be a disturbing limitation for practitioners, but it is not surprising-it is shared with SCIs for other powerful stepwise MTPs, including the fixed-sequence MTP (Hsu and Berger 1999, sec. 1), the step-down Dunnett MTP (Stefansson, Kim and Hsu 1988, sec. 3), and the step-up Dunnett MTP (Finner and Strassburger 2007, sec. 3).
It is then natural to try to construct families of MTPs related to Holm's, Hochberg's, and Hommel's MTPs that do not have this limitation, at the cost of some loss in rejection power. A user may then choose among various such trade-off MTPs. Such families of MTPs were considered in Guilbaud (2012). The Holm-related MTPs considered there had previously been used in the gatekeeping context (Dmitrienko, Tamhane and Wiens 2008, sec. 3.1) where they were called truncated Holm MTPs. These can be viewed as MTPs between the Holm MTP and the Bonferroni MTP. The corresponding SCIs had briefly been discussed (though in other terms) in Strassburger and Bretz (2008, sec. 4.2), and they correspond to a special case of the Guilbaud [2009, Equation (10)] SCIs. In contrast, the Hochberg-and Hommel-related MTPs and the corresponding SCIs proposed in Guilbaud (2012) were new.
This article shows how alternative Hochberg-and Hommel-related MTPs can be constructed that: (i) are more powerful than the Hochberg-and Hommel-related MTPs proposed previously, and (ii) lead to SCIs that are sharper than the ones proposed previously. This improvement became possible through the derivation of an extended Simes inequality that is closely related to a recent result by Sarkar (2008, sec. 3). The results are formulated in terms of t-distributions, but are valid (at least approximately) more generally. A nice practical and theoretical aspect is that underlying correlations are only assumed to be nonnegative, as in Sarkar's result. For completeness and comparison purposes, the Holm-related MTPs and corresponding SCIs just mentioned are also discussedfor simplicity under the same assumptions, though they are valid more generally.
The developments are in terms of one-sided null hypotheses, one-sided marginal tests/p-values, and onesided SCIs. This was the case also for the developments in Guilbaud (2012) and Strassburger and Bretz (2008), as well as, for instance, in Hsu and Berger (1999, sec. 1) for the fixed-sequence MTP, in Stefansson, Kim and Hsu (1988, sec. 3) for the step-down Dunnett MTP, and in Finner and Strassburger (2007, sec. 3) for the step-up Dunnett MTP. The reason for not considering two-sided null hypotheses in this article is discussed in Section 10.1. It has to do with the fact that practitioners are typically interested in (and almost always make) additional decisions about the directions that led to the two-sided rejections, and the challenging problem of proving whether such additional directional decisions actually are valid with stepwise MTPs. The discussion in Section 10.1 is relevant also for MTPs that are more elaborated than the ones considered in this article; see for example, Dmitrienko, D'Agostino, and Huque (2013) for various elaborated MTPs that have stepwise components.
The article is organized as follows. The setup is described in Section 2, including the notation, the family of null hypotheses, and the assumptions. The formulation of an MTP as a closed-testing procedure (CTP) is described in Section 3. In view of their simplicity, the SCIs for the original Holm, Hochberg, and Hommel MTPs are introduced and discussed already in Section 4. The Liu (1996) class of CTPs is then described in Section 5, and the trade-off-type Holm-, Hochberg-, and Hommelrelated MTPs are defined in Section 6 in terms of this class of CTPs. A simple algorithm is given in Section 7 that can be used to determine the null hypotheses rejected through these MTPs. The main results about the SCIs for these MTPs are given and discussed in Section 8. Section 9 provides illustrations based on the dose-finding trial example in Dmitrienko et al. (2009). Finally, some concluding comments and additional results are given in Section 10, including a discussion in Section 10.1 about why two-sided null hypotheses have not been considered, and a brief sketch in Section 10.2 of how the exact inferences based on t-distributions that have been developed can be extended to approximate inferences in other situations through large-sample arguments.
The supplementary materials for this article consist of four appendices (A-D). Appendix A provides the extended Simes inequality (Theorem 1) and its proof. Appendix B provides a discussion of technical details and ideas underlying the proposed SCIs. In particular, the role of the extended Simes inequality is discussed in Section B.4. Appendix C provides proofs of miscellaneous assertions made in the developments. Appendix D provides a further discussion about the original Simes (1986) inequality and related results; including some historical details and an early result by Daniels (1945) that should be remembered, as well as a simple argument showing that in Simes' proof based on independent p-values, one can drop the restrictive condition that p-values are uniformly distributed under their null hypotheses. Here "direct argument" means that no particular null hypotheses or test statistics underlying the p-values are introduced in the argument.
We refer to Guilbaud (2012) for more about SCIs for MTPs, and to Dmitrienko et al. (2009), Dmitrienko, D'Agostino, and, , and Alosh, Bretz, and Huque (2013) for basic results, key issues, and recent advances in multiple testing in clinical drug development. It should also be mentioned that the pioneering article by Stefansson, Kim, and Hsu (1988) now has had its 25th birthday. This article was the first to show that SCIs corresponding to powerful stepwise MTPs were at all possible; see also the related further results and discussions in Hayter and Hsu (1994).
For convenience, no notational distinction is made between random quantities and the corresponding realizations in this article.

Setup, Notation, and Assumptions
The simultaneous inferences considered in this article concern m ≥ 2 specified real-valued quantities, θ 1 . . . , θ m . Typically these unknown quantities θ i reflect comparisons of interest; for example, comparisons in a clinical trial in terms of differences in true means between two or more treatments; see the illustration in Section 9, and the extensions in Section 10.2. It is supposed that we are interested in making: (i) simultaneous assertions of the form where θ i,0 is a prespecified target boundary; and (ii) if possible, additional simultaneous confidence assertions about the θ i 's. To simplify the subsequent developments, though, all target assertions are supposed to be of the form "θ i > θ i,0 ". An assertion of the alternative form "θ i < θ i,0 " can be reformulated to be of the form "θ i > θ i,0 " through an appropriate redefinition of θ i and θ i,0 . A target assertion "θ i > θ i,0 " may, for example, correspond to a noninferiority or superiority assertion aimed at in a confirmatory clinical trial to support an intended claim about a treatment. In the following, α ∈ (0, 1/2) is a prespecified level, M denotes the index set {1, 2, . . . , m}, and for each subset I of M, |I | denotes the number of elements in I . We use the following standard notation. The normal distribution with mean a and variance b is denoted N (a, b), and the q-quantile and distribution function of the standard normal distribution N (0, 1) are denoted z q and (·), so that (z q ) = q. The m-variate normal distribution with mean vector μ and covariance matrix is denoted N m (μ, ). The central t-distribution with ν ≥ 1 degrees of freedom is denoted T ν , its q-quantile is denoted t ν,q , and its distribution function is denoted F ν (·), so that F ν (t ν,q ) = q. Moreover, χ 2 ν denotes a chi-square distributed random variable with ν ≥ 1 degrees of freedom.
The restriction α < 1/2 ensures that certain sign restrictions underlying the extended Simes inequality in Appendix A are fulfilled when this inequality is applied in the developments. Typically α is chosen to be much smaller than 1/2, so this is not really a restriction in practice.
Let us now introduce, for i = 1, . . . , m, the null and alternative hypotheses, H i and H c i , about θ i ∈ (−∞, ∞) given by where θ i,0 ∈ (−∞, ∞) is a prespecified target value, see the discussion following (1). Note that it is the alternative hypothesis H c i that corresponds to the target assertion "θ i > θ i,0 " aimed at.
The data we have at our disposal to make inferences about the quantities θ 1 , . . . , θ m are supposed to be summarized in: (i) point estimatorsθ 1 , . . . ,θ m of these quantities; and (ii) standard errors sθ 1 , . . . , sθ m of these point estimators, respectively. The following Assumption A about these random quantities leads to exact tdistribution-based inferences, including rejection decisions about H i s and SCIs for θ i s.
These distributional assumptions may seem rather strong, but they are in accordance with those used by Sarkar (2008, sec. 3) in his proof of the validity of the Simes inequality for t-distributions. Standard "large sample" arguments can, however, be invoked that show that approximate inferences can be made in situations where approximately normally distributed estimatorsθ 1 , . . . ,θ m and associated standard errors are available, see Section 10.2.
Assumption A. (i) the data at our disposal for inferences about θ 1 , . . . , θ m are summarized in the two independent statistics (θ 1 , . . . ,θ m ) and S > 0 that are such that ⎛ where σ > 0 is an unknown scale factor; (ii) the empirical and theoretical standard error, sθ i > 0 and σθ i > 0, of the point estimatorθ i of θ i are of the form and (iii) in the (c i, j )-matrix in (3), the diagonal elements c 1,1 > 0, . . . , c m,m > 0 are known and possibly different, and the off-diagonal elements c i, j , i = j, are nonnegative, possibly unknown, and possibly different.
Remark 1. The fact that the off-diagonal elements c i, j , i = j, are allowed to be different constitutes a considerable weakening of the earlier assumption (that offdiagonal elements are equal) used by Sarkar (1998, prop. 3.2) in his proof of the validity of the Simes inequality for t-distributions. Unbalanced designs are now covered by these assumptions, for example, many-to-one multiple comparisons based on samples of different size. See Sarkar (2008, sec. 3) for an interesting historical account of difficulties and advances in this context. Assumption A implies that marginally for each i = 1, . . . , m: (i)θ i and sθ i are independent (observable) statistics such thatθ so that and (iii) the ordinary p-value p i of the test of H i versus H c i in (2) based on the t-statistic (θ i − θ i,0 )/sθ i is given by so we have the relation that p i ≤ u if and only if θ i,0 ≤ L i;1−u . We use the natural convention that L i;1−0 = −∞.
Finally, we note that under Assumption A, the m random quantities L i;(1−α) 1/m , i ∈ M, constitute simultaneous (1 − α) confidence lower bounds for θ 1 , . . ., θ m in that The multiplicity adjustment of the level of each lower bound in (8) is as if the bounds were independent. Relation (8) can be shown through an application of Slepian's inequality in combination with Kimball's inequality (Hsu 1996, theor. A.3.1 and cor. A.1.1) by conditioning on the random variable S in (3) and (4). The confidence bounds in (8) are sharper than the corresponding Bonferronitype bounds L i;1−α/m that do not use the assumption that correlations betweenθ i s in (3) are nonnegative. As will be seen, these Bonferroni-type bounds, as well as the independence-type bounds in (8), occur in the expressions for the SCIs considered in Sections 4 and 8.

The Given MTP Viewed as a Closed-Testing Procedure (CTP)
The simultaneous 1 − α confidence assertions about θ 1 , . . . , θ m to be considered correspond to a given MTP based on the marginal p-values (7) that rejects null hy- Here multiple-level α means that the Type-I FWER is strongly controlled to be at most α, that is, the probability of rejecting any true H i in the family through the MTP is at most α, irrespective of the actual constellation of true and false H i 's. The potential rejection assertions "θ i > θ i,0 " made through the MTP are thus equal to the target assertions aimed at.
It is assumed that the given MTP can be viewed as a CTP based on given local (i.e., marginal) level-α tests for intersection hypotheses H I = ∩ i∈I H i , ∅ = I ⊂ M; and by definition, the CTP rejects any specific H i if and only if each H I with i ∈ I ⊂ M is rejected by its local/marginal level-α test. Then: (i) the two complementary index sets are well-defined random quantities; (ii) their number of elements satisfy |I Reject | + |I Accept | = m; and (iii) the probability is less than or equal to α that I Reject contains the index of any true H i , irrespective of the actual constellation of true and false H i 's in the family. A simple algorithm is available to determine the index sets (9) for MTPs/CTPs considered in this article; see Section 7. The index sets (9) and the rejection assertions "θ i > θ i,0 ", i ∈ I Reject , constitute the basis for the proposed simultaneous 1 − α confidence assertions about θ 1 , . . . , θ m . The rejection assertions are always included in these simultaneous confidence assertions, so these latter assertions provide additional information "for free."

The SCIs Corresponding to the Original Holm, Hochberg, and Hommel MTPs
The SCIs for these three classical MTPs are so simple and remarkably similar that we describe and discuss them already in this section. It is well known how these MTPs are defined in terms of the p-values p 1 , p 2 , . . . , p m given by (7), so it is easy to determine the index sets I Reject and I Accept in (9) for these MTPs; see for example, Dmitrienko, D'Agostino, and Huque (2013, sec. 5) and Dmitrienko and D'Agostino (2013, app. A.3). Algorithm 1 in Section 7 can also be used (with d r,s 's given by (19)) to determine these index sets.
The SCIs are defined in terms of: (i) the index sets I Reject and I Accept , (ii) the target values θ i,0 in (2), and (iii) the lower bounds L i;1−u given by (5). We use the notation x ∨ y for max(x, y). Recall from (2) that the rejection of H i in favor of H c i corresponds to the assertion "θ i > θ i,0 ," whereas the acceptance (i.e., nonrejection) of H i corresponds to the noninformative assertion "θ i > −∞." Let I Reject and I Accept be the index sets (9) for Holm's (1979) MTP based on the p-values (7), and consider the following intervals for θ i , i ∈ M, respectively, These intervals simultaneously cover θ 1 , . . . , θ m , respectively, with probability at least 1 − α. They were derived by Guilbaud (2008) and Strassburger and Bretz (2008) through different methods.
Let I Reject and I Accept be the index sets (9) for Hochberg's (1988) MTP based on the p-values (7), and consider the following intervals for These intervals simultaneously cover θ 1 , . . . , θ m , respectively, with probability at least 1 − α. They were derived by Guilbaud (2012); see eq. (58) and remark 9 in that article.
Let I Reject and I Accept be the index sets (9) for Hommel's (1988) MTP based on the p-values (7), and consider the following intervals for if i ∈ I Reject and |I Reject | < m, These intervals simultaneously cover θ 1 , . . . , θ m , respectively, with probability at least 1 − α. They were derived by Guilbaud (2012), see eq. (58') and remark 11 in that article.
The expressions for the SCIs in (10)-(12) are very simple and remarkably similar. The only formal difference is that the Bonferroni-type lower bound L i;1−α/m occurs in the first row in (10), whereas the independencetype lower bound L i;(1−α) 1/m occurs in the first row of (11) and (12). Intuitively it is surprising that the same Bonferroni-type adjustment α/|I Accept | occurs in the third row of (10)- (12), because Hochberg's and Hommel's MTPs are not Bonferroni-based CTPs. The reason behind this common Bonferroni-type adjustment is actually far from evident; see Remark 3 in Section 8.
The SCIs in (10)-(12) have the following properties. For each i ∈ I Reject , the confidence assertion about θ i is at least as sharp as the rejection assertion "θ i > θ i,0 " made through the MTP, possibly sharper-though the important point here is that such a sharper assertion is not possible in case |I Reject | < m. For each i ∈ I Accept , the confidence assertion about θ i is sharper than the noninformative acceptance assertion "θ i > −∞" made through the MTP, but less sharp than the rejection assertion "θ i > θ i,0 ," thus indicating by how much one missed this target assertion.
It is interesting to note that in (10)-(12), the lower confidence bounds for θ i 's with i ∈ I Accept are always at least as sharp as the corresponding Bonferroni bounds L i;1−α/m in that where the inequality ≤ is strict if at least one null hypothesis is rejected by the MTP. In case no null hypothesis is rejected by the MTP, the inequality ≤ in (13) is an equality, and the m lower bounds in (10)-(12) are then all equal to the Bonferroni bounds L i;1−α/m . This could be used to construct lower confidence bounds that can beat Bonferroni bounds also in right neighborhoods of target values of interest by prespecifying θ i,0 's to be larger than these target values (instead of equal to them). This would obviously require various considerations regarding the choice of θ i,0 's, and we do not develop this idea in this article. The essential difference between the SCIs (10)-(12) is of course that the three underlying MTPs may lead to different index sets (9). As is well known, the index sets (9) of these MTPs satisfy the relations It can be verified using these relations that the SCIs (12) are always at least as sharp as the SCIs (11), and that the SCIs (11) are always at least as sharp as the SCIs (10).
The fact that no confidence assertions sharper than rejection assertions can be made through the SCIs (10)-(12) in case not all null hypotheses are rejected may be a disturbing limitation for practitioners. It is, therefore, of interest to consider MTPs with SCIs that do not have this limitation. The main results in this article concern MTPs of this kind that are related to Holm's, Hochberg's, and Hommel's MTPs. These MTPs are defined in Section 6 in terms of the Liu (1996) class of CTPs that is described next.

The Liu (1996) Class of CTPs Based on Ordered p-Values and a Critical Matrix
Consider the set of all p-values p 1 , p 2 , . . . , p m given by (7). For each nonempty I ⊂ M, let p (1):I ≤ p (2):I ≤ · · · ≤ p (|I |):I denote the ordered values of the subset of p s 's with index s ∈ I . When I = M, we sometimes simplify the p (s):I -notation to p (1) ≤ p (2) ≤ · · · ≤ p (m) .
Each CTP considered in this article for the family {H s ; s ∈ M} of null hypotheses given by (2)  Here the critical constants d |I |,1 ≥ d |I |,2 ≥ · · · ≥ d |I |,|I | are prespecified and such that the test (15) of H I has local (i.e., marginal) level α. There is no loss of generality in letting the critical constants d |I |,s be nonincreasing in s, because the ordered p-values p (s):I in (15) are nondecreasing in s.
The CTP based on the local level-α tests (15) is determined by the prespecified critical constants. Such CTPs were considered by Liu (1996) the critical matrix of the CTP. The local test (15) of H I is thus based on the critical constants in row r = |I | of the matrix (16). It is important to note that these critical constants depend on I only through the number |I | of elements in I. It is understood (without loss of generality) that d r,1 ≥ d r,2 ≥ · · · ≥ d r,r for each row r = 1, . . . , m.
Moreover, for all CTPs considered in this article, the diagonal elements in (16) satisfy The inequalities (18) are important for the subsequent developments in that they: (i) ensure the validity of Algorithm 1 described in Section 7 through which the index sets (9) can easily be determined; and (ii) lead to certain simplifications, notably the one discussed in Remark 3 in connection with (29). The inequalities (18) Interestingly, it follows from (19) that the three MTPs have the same diagonal elements (18), and that these common diagonal elements are equal to We recognize these values as the critical values that occur in the step-down algorithm of Holm (1979), and in the step-up algorithm of Hochberg (1988). This relation between the critical matrix of Holm's MTP, Hochberg's MTP, and Hommel's MTP was noted and discussed by Liu (1996).

The Holm(W ), Hochberg(W ), and
Hommel(W ) MTPs with Parameter 0 ≤ W ≤ 1 We now introduce certain MTPs related to Holm's (1979) MTP, Hochberg's (1988) MTP, and Hommel's (1988) MTP that also belong to Liu's (1996) class of CTPs, and define them in terms of their critical matrix. These MTPs depend on a parameter 0 ≤ W ≤ 1 that specifies the closeness to the original Holm, Hochberg, and Hommel MTPs based on (19), with W = 1 corresponding to these original MTPs. The value of 0 ≤ W ≤ 1 is to be chosen and prespecified by a user.
Let the Holm(W ) MTP, the Hochberg(W ) MTP, and the Hommel(W ) MTP, be the CTPs defined through (15)-(18) with critical constant d r,s given for 0 ≤ W ≤ 1 and 1 ≤ s ≤ r ≤ m by where A r (W ) and B r (W ) are the nondecreasing functions of 0 ≤ W ≤ 1 given by It can be verified that as W increases in the interval [0, 1]: (i) the value of A r (W ) increases from αr/m to α if 1 ≤ r < m, and equals α if r = m; and (ii) the value of B r (W ) increases from 1 − (1 − α) r/m to α if 1 ≤ r < m, and equals α if r = m. In particular, A r (W ) ≤ α, B r (W ) ≤ α, and A r (1) = B r (1) = α.
The critical constants d r,s in (21) satisfy the important inequalities (18) for any given 0 ≤ W ≤ 1. For the Holm(W )-related d r,s 's this follows from the fact that A r (W )/r equals α [W/r + (1 − W ) /m], which is nonincreasing as r increases. For the Hochberg(W )-, and Hommel(W )-related d r,s s, this follows from the fact shown in Appendix C.1 that the ratio B r (W )/r is nonincreasing as r increases.
It is also clear from (22) that with W = 0, the Holm(W ) MTP has d r,s 's that are all equal to α/m, as the Bonferroni MTP. This means that the Holm(0) MTP is equal to the Bonferroni MTP.
The rejection index sets I Reject for the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs are nondecreasing as W increases, because A r (W ) and B r (W ) in (21) and (22) are nondecreasing as W increases. It follows in particular that for any given 0 ≤ W ≤ 1, these MTPs reject, respectively: (i) at most as much as the Holm It is shown in Appendix C.2 that for any given 0 ≤ W ≤ 1, the index sets (9) This generalizes the relations (14). It follows in particular that the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs reject at least as much as the Holm (0) (24) for B r (W ) in the last two rows of (21). In the sequel we will refer to these MTPs based on (24) as the Hochberg * (W ) and Hommel * (W ) MTPs.
It is clear from (22) and (24) that B * r (1) = B r (1) = α for all 1 ≤ r ≤ m. It follows that the Hochberg * (1) and Hommel * (1) MTPs are equal to the original Hochberg and Hommel MTPs, respectively, as the Hochberg(1) MTP and the Hommel(1) MTPs. Moreover, it is shown in Appendix C.3 that for W α,m ≤ W < 1, It follows that for W α,m ≤ W < 1, the Hochberg(W ), and Hommel(W ) MTPs are sharper than the Hochberg * (W ) and Hommel * (W ) MTPs, respectively, in that at least as much is rejected, possibly more. Another advantage of the Hochberg(W ) and Hommel(W ) MTPs is that they are defined also for 0 ≤ W < W α,m . It is understood in the sequel that the Hochberg(W ) MTP and the Hommel(W ) MTP are the MTPs based on (21) and (22) that are defined for all 0 ≤ W ≤ 1.

Algorithm to Determine the Index Sets I Reject and I Accept in (9)
The following simple algorithm can be used to determine the index sets (9) for MTPs/CTPs that can be represented in terms of a critical matrix (16) satisfying (17) and (18). In particular, the index sets (9) for the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs with d r,s 's given by (21) can be determined through this algorithm for any given 0 ≤ W ≤ 1.
Algorithm 1 (Klein (1998)). Assuming that the inequalities (18)  This is a corrected version of the corresponding algorithm in Guilbaud (2012, p. 327). It originates from Klein's (1998) unpublished diploma thesis, but its published version mentioned as Klein's theorem in Bernhard, Klein, and Hommel (2004, p. 8) contained a recently detected index error that unfortunately was reproduced in Guilbaud (2012); see the corrections in Bernhard, Klein and Hommel (2013) and Guilbaud (2013). Only the numerical illustrations in Guilbaud (2012, sec. 7) were affected by this index error, and a corrected version of that section was provided in the Supporting Information for Guilbaud (2013).

The SCIs Corresponding to the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs
The SCIs for θ 1 , . . . , θ m considered here are defined in terms of: (i) the target values θ i,0 in (2); (ii) the lower bounds L i;1−u in (5); (iii) the functions A r (W ) and B r (W ) given by (22); and (iv) the index-sets I Reject and I Accept in (9) that can be determined through Algorithm 1 with d r,s s given by (21) for the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs. Some technical details and underlying ideas are discussed in Appendix B; but briefly, these SCIs are special cases of the SCIs in Guilbaud (2012, eq. (44)) for CTPs in the Liu (1996) class, which in turn follow from the more general SCIs in Guilbaud (2012, eq. (19)). Recall that the value of 0 ≤ W ≤ 1 is to be prespecified by a user, and that W = 1 corresponds to the original Holm (1979), Hochberg (1988), and Hommel (1988) MTPs.
Let I Reject and I Accept be the index sets (9) determined through Algorithm 1 with d r,s = d Holm(W ) r,s in (21) for the Holm(W ) MTP based on the p-values (7), and consider the following intervals for θ i , i ∈ M, respectively, These intervals simultaneously cover θ 1 , . . . , θ m , respectively, with probability at least 1 − α. These SCIs are not new. For W = 1, they equal the SCIs in (10) derived by Guilbaud (2008) and Strassburger and Bretz (2008). For 0 ≤ W < 1, they were briefly discussed (though in other terms) in Strassburger and Bretz (2008, Section 4.2), and they can be viewed as a special case of the Guilbaud (2009, eq. (10)) SCIs with weights B i = 1/m.
Let I Reject and I Accept be the index sets (9) determined through Algorithm 1 with d r,s = d Hommel(W ) r,s in (21) for the Hommel(W ) MTP based on the p-values (7), and consider the following intervals for θ i , i ∈ M, respectively, These intervals simultaneously cover θ 1 , . . . , θ m , respectively, with probability at least 1 − α. For W = 1, they equal the SCIs in (12) derived by Guilbaud (2012) for Hommel's (1988) MTP; whereas for 0 ≤ W < 1, they are new.
For W = 1, the SCIs (26)-(28) have the properties discussed in Section 4. For 0 ≤ W < 1, the SCIs have the following properties. For each i ∈ I Reject , the confidence assertion about θ i is at least as sharp as the rejection assertion "θ i > θ i,0 " made through the MTP, possibly sharper-the important point here is that such a sharper assertion is possible even in case |I Reject | < m. For each i ∈ I Accept , the confidence assertion about θ i is sharper than the noninformative acceptance assertion "θ i > −∞" made through the MTP, but less sharp than the rejection assertion "θ i > θ i,0 ," thus indicating by how much one missed this target assertion.
For W = 0, the SCIs (26) reduce to the Bonferroni bounds L i;1−α/m for all 1 ≤ i ≤ m, irrespective of the outcome of the index sets I Reject and I Accept , because A r (0)/r = α/m. This reduction result is quite appealing because the Holm(0) MTP equals the Bonferroni MTP.
Remark 3. The confidence intervals in the third row of (10)-(12) and (26)-(28) have a simple relation to the diagonal elements d r,r of the critical matrix (16) defining each underlying MTP. It can be verified from (19) and (21) that each of these confidence intervals is of the simple form The reason is that the SCIs (10)-(12) and (26)-(28) are special cases of the SCIs in Guilbaud (2012, eq. (44)) for MTPs in the Liu (1996) class, and that the expression given there for the confidence intervals for θ i 's with i ∈ I Accept reduce to (29) in case the inequalities (18) hold. The common Bonferroni-type adjustment in (10)-(12) then follows from the additional fact that the ordinary Holm, Hochberg, and Hommel MTPs have the common diagonal elements d r,r = α/r in (20).
Remark 4. Let 0 ≤ W < W ≤ 1 be given, and let H(W ) denote for a moment any one of the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs. Then H(W ) can be said to be better than H(W ) in that it always rejects at least as much, possibly more; see Section 6. In contrast, one cannot say that the H(W )-based SCIs in (26)-(28) are better than the H(W )-based SCIs, nor vice versa. Here "are better than" is in the strong sense "are contained within" with probability one. Rather, one can say that the H(W )-based SCIs and the H(W )-based SCIs correspond to different tradeoffs between the aim of rejecting as many H i 's as possible and the aim of getting sharper confidence assertions about θ i s associated with rejected H i 's. Note, for example, that in case H(W ) rejects some H i 's and H(W ) does not reject all H i 's, then the H(W )-based lower confidence bound in the second row of (26)- (28) can be larger than the corresponding H(W )-based lower confidence bound; see the illustration in Section 9.2.
Remark 5. For any given 0 ≤ W ≤ 1, the SCIs (28) for the Hommel(W ) MTP are at least as sharp, possibly sharper than, the SCIs (27) for the Hochberg(W ) MTP, which in turn are at least as sharp as, possibly sharper than, the SCIs (26) for the Holm(W ) MTP. This can be verified using the relations (23), the inequality 1 − (1 − α) 1/m > α/m, and the fact shown in Appendix C.2 that the difference B r (W ) − A r (W ) given by (22) is nonnegative. It should be remembered in this context that the Holm(W )-based inferences (rejections and SCIs) are valid under weaker assumptions than the Hochberg(W )and Hommel(W )-based inferences.
Remark 6. The SCIs (27) and (28) are at least as sharp as, possibly sharper (in case W < 1) than, the SCIs in Guilbaud (2012, eqs. (58) and (58')) that correspond to the Hochberg * (W ) and Hommel * (W ) MTP discussed in Remark 2. The latter SCIs are given for any given W α,m ≤ W ≤ 1 by the expressions (27) and (28) with: (i) B r (W ) in the third row replaced by B * r (W ) in (24); and (ii) index sets I Reject and I Accept for the Hochberg * (W ) and Hommel * (W ) MTP obtained through Algorithm 1 and the d r,s 's described in Remark 2. The improvement over the Hochberg * (W )-and Hommel * (W )-based inferences is a consequence of the relations (25) between B * r (W ) and B r (W ). These relations, and the role in this context of the extended Simes inequality derived in Appendix A, are discussed in Appendix B.4; see also the illustration of the improvement in Section 9.3.

Illustrations Based on the Dose-Finding Trial Example in Dmitrienko et al. (2009)
We use the same illustration as in Guilbaud (2012, sec. 7) based on the dose-finding trial example in Dmitrienko et al. (2009). We can then make some comparisons with previous results. The trial is a parallel-group study where patients with dyslipidemia are treated either with placebo (labeled D 0 ), or with one of four doses (labeled D 1 , D 2 , D 3 , D 4 , in increasing dose order) of the drug under investigation. The study is balanced in that the size of each treatment group is 77. The inferences of interest concern the four (D i − D 0 )-differences, i = 1,2,3,4, in true means, θ 1 , θ 2 , θ 3 , θ 4 , with respect to increase in high-density lipoprotein cholesterol after 12 weeks of treatment in patients. These differences are estimated by the corresponding four (D i − D 0 )-differences in sample means,θ 1 ,θ 2 ,θ 3 ,θ 4 , with standard errors, sθ 1 , sθ 2 , sθ 3 , sθ 4 , based on a pooled estimator S 2 of an underlying common variance σ 2 . This pooled S 2 is based on the five withingroup variances.

Outcome Scenario
The outcome scenario considered here is the one labeled Scenario 1 in Dmitrienko et al. (2009), and Outcome Scenario 1 in Guilbaud (2012). In this outcome scenario, the observed (D i − D 0 )-differences,θ 1 ,θ 2 ,θ 3 , θ 4 , are equal to 2.90, 3.14, 3.56, 3.81, and their standard errors, sθ 1 , sθ 2 , sθ 3 , sθ 4 , are each equal to 1.44. The observed differenceθ i is thus positive for 1 ≤ i ≤ 4, and increases as i increases. The marginal p-values p 1 , p 2 , p 3 , p 4 , based on these observed differences and standard errors are equal to 0.0224, 0.0149, 0.0069, 0.0042; these values differ slightly from those for Scenario 1 in Dmitrienko et al. (2009, tab. 2.1) due to the rounding in the observed values of differences and standard errors just given.

Inferences Based on (26)-(28) for Some Values
of 0 ≤ W ≤ 1 Table 1 provides the rejection index set for the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs, as well as the corresponding simultaneous 1 − α confidence lower bounds for θ 1 , θ 2 , θ 3 , θ 4 in (26)-(28), in the outcome scenario just described. With these data, the results for the Hommel(W ) MTP turn out to be equal to those for the Hochberg(W ) MTP, so they are not repeated in the table. Recall that the W -value is to be prespecified by a user.
First, consider the results in the left part of Table 1. With W = 1 (original Holm MTP), only H 3 and H 4 are rejected, so no confidence assertion sharper than a rejection assertion "θ i > 0" is made about θ 3 and θ 4 . Note also that as W decreases in the seven rows with I Reject = {3, 4}, the assertions about θ 1 and θ 2 become less sharp. Moreover, as W further decreases in the four rows with I Reject = {4}: (i) the assertions about θ 1 , θ 2 , and θ 3 become less sharp; whereas (ii) the assertion about θ 4 becomes sharper, even though not all hypotheses are rejected. In particular, with W = 0 (Bonferroni MTP), the assertion about θ 4 is sharper than the assertions made with larger W -values, including the one made with the original Holm MTP. These results are in accordance with the discussion in Remark 4.
Next, consider the results in the right part of Table 1. With W = 1 (original Hochberg MTP), all four hypotheses are rejected, and a confidence assertion about θ 4 is made that is sharper than a rejection assertion. Interestingly, an equally sharp assertion about θ 4 is made with W = 0.9. Otherwise, similar comments can be made about the monotonic behavior of rows as W decreases as those made about the left part of Table 1. Again, these results are in accordance with Remark 4.
The fact that the SCIs in the right part of Table 1 are at least as sharp, often sharper, than those in the left part is in accordance with the discussion in Remark 5.

Improvement Over the Hochberg-and
Hommel-Related Inferences in Guilbaud (2012) As discussed in Remarks 2 and 6, for any given W α,m ≤ W ≤ 1: (i) the Hochberg(W ) and Hommel(W ) MTPs reject at least as much as, possibly more (in case W < 1) than, the Hochberg * (W ) and Hommel * (W )    MTPs in Guilbaud (2012, sec. 4.3); and (ii) the SCIs (27) and (28) are at least as sharp, possibly sharper (in case W < 1) than, the SCIs in Guilbaud (2012, eqs. (58) and (58')) that correspond to the Hochberg * (W ) and Hommel * (W ) MTPs. Recall also that the Hochberg(W ) and Hommel(W ) MTPs and associated SCIs (27) and (28) have the further advantage that they are defined also for 0 ≤ W < W α,m . Table 2 illustrates the improvement in the outcome scenario described in Section 9.1. With these data, the results for the Hommel(W ) and Hommel * (W ) MTPs turn out to be equal to those for the Hochberg(W ) and Hochberg * (W ) MTPs, respectively, so they are not repeated in the table. We restrict considerations to W -values equal/close to 0.86, 0.79, 0.33, 0.32; because for these four values, the Hochberg(W ) MTP rejects more than the Hochberg * (W ) MTP.
The improvement in the left part (compared to the right part) of Table 2 may be small, except when the Hochberg(W ) reject more than the Hochberg * (W ). Anyhow, whenever there is a difference, small or large, it is always in favor of the Hochberg(W ) MTP and its associated SCIs. This illustrates the fact discussed in Remarks 2 and 6 that the Hochberg(W ) and Hommel(W ) MTPs and their associated SCIs are preferable to the Hochberg * (W ) and Hommel * (W ) MTPs and their associated SCIs.

Concluding Comments and Additional Results
If the aim in a particular situation is to reject as many null hypotheses as possible, then it may be reasonable to restrict considerations to the original Holm, Hochberg, and Hommel MTPs and their associated SCIs given by (10)-(12). This may be the case in confirmatory clinical trials where target assertions may correspond, for example, to noninferiority and/or superiority assertions aimed at to support intended claims about treatments. The confidence assertions associated with nonrejected hypotheses that can then be made may be of some value in that they indicate by how much one missed the target assertions aimed at. Otherwise, if one is interested in the possibility of making assertions sharper than rejection assertions also when not all hypotheses are rejected, then one may consider trade-off procedures such as the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs with 0 ≤ W <1 and their associated SCIs given by (26) and (28). The choice of 0 ≤ W < 1 to be used then depends on the anticipated behavior of the MTPs and their SCIs. A further discussion of these aspects is outside the scope of this article.
We emphasize that the main results in this article are based on the setup and assumptions made in Section 2. The assumption that correlations betweenθ i s in (3) are nonnegative is important for the Hochberg(W )and Hommel(W )-related inferences. If this assumption is questionable, then one may consider the Holm(W )related inferences instead, because these are valid without this assumption.
We end this article with: (i) a discussion about why we have not considered two-sided null hypotheses in this article, and (ii) a brief description of how large sample arguments can be invoked that show that approximate inferences can be made under rather weak distributional assumptions in many situations.

About One-Sided Directional Decisions Supplementing Two-Sided Rejections
An MTP may of course be applied to a family of two-sided null-hypotheses under appropriate assumptions. For example, under the setup of Section 2 (with two-sided null-hypotheses and p-values instead of onesided), Holm's (1979) MTP is valid without any assumption aboutθ i -correlations, whereas the assumption about nonnegativeθ i -correlations does not seem to be sufficient some pairs of one-sided null hypotheses, and accordingly redefine theθ i 's, θ i 's, and θ i,0 's, so that the resulting correlation structure of theθ i is appropriate. This is always possible if m = 2, and is sometimes possible if m > 2. In the situation considered in Section 9, no such reformulation and exchange of one-sided null hypotheses within pairs is necessary. We omit the details.

Inferences Based on "Large Sample" Approximations
The rather strong Assumption A in Section 2 leads to exact inferences about θ 1 , . . . , θ m based on the tdistribution. Standard "large sample" arguments can, however, be invoked to show that corresponding approximate inferences can be made in various situations.
The data available for inferences are then supposed to be summarized in the estimatorsθ 1 , . . . ,θ m with associated standard errors s 1 , . . . , s m . Letting n denote the sample size (or some other suitable information measure), it is assumed that as n → ∞: (i) the distribution of √ n(θ 1 − θ 1 , . . . ,θ m − θ m ) T tends to an m-variate normal distribution with mean (0, . . . , 0) T and covariance matrix (σ i, j ) having positive diagonal elements σ i,i and nonnegative off-diagonal elements; and (ii) each standardized difference √ n(s i − √ σ i,i ) tends in probability to zero.
This holds in many common situations. Then, for large values of n: (i) the vector (θ 1 , . . . ,θ m ) T is approximately distributed according to an m-variate normal distribution with mean (θ 1 , . . . , θ m ) T and covariance matrix having diagonal elements (s 1 , . . . , s m ) and nonnegative off-diagonal elements; (ii) L i;1−u ≡ θ i − z 1−u s 2 i is such that Pr[L i;1−u < θ i ] is approximately equal to 1 − u; (iii) p i ≡ 1 − ([θ i − θ i,0 ]/s 2 i ) can be used to test the null hypothesis H i in (2) at an approximate level 0 < u < 1 by rejecting H i if and only if p i ≤ u; and (iv) with these normal-approximation-based L i;1−u s and p i s, the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs defined through (21) have multiple-level approximately equal to α, and their associated SCIs in (26)-(28) have simultaneous confidence level approximately equal to 1 − α.
As a simple illustration of these approximate inferences, consider the situation described in Section 9, with the modification/complication that: (i) sample sizes are not equal; and (ii) the within-group variability is anticipated to differ so much between groups that we are reluctant to pool sample variances S 2 i . Such a situation was discussed by Dunnett and Tamhane (1991, p. 994) who mentioned the possibility of using Holm's (1979) MTP based on the (Group i − Group 0 )-differenceŝ θ i in sample means and standard errors s i of the form (S 2 i /n i + S 2 0 /n 0 ) 1/2 . It can be verified from the two pre-ceding paragraphs that with theseθ i 's and standard errors s i , and if the ratios n i /n tend to given positive proportions as n ≡ n 0 + · · · n m tends to infinity and the normalapproximation-based L i;1−u 's and p i 's are used, then for any given 0 ≤ W ≤ 1, the Holm(W ), Hochberg(W ), and Hommel(W ) MTPs, as well as their associated SCIs in (26)- (28), are approximately valid for large values of n. Suppose that in the preceding situation we are interested in group comparisons of, say, survival (or death) rates P i (t) after a given time t on treatment instead of means. The comparisons may then be in terms of, for example, differences, ratios, or odds ratios; with θ i s specified for all 1 ≤ i ≤ m either as, respectively, θ i = P i (t) − P 0 (t), θ i = log(P i (t)/P 0 (t)), or θ i = log([P i (t)/Q i (t)]/[P 0 (t)/Q 0 (t)]) where Q i (t) ≡ 1 − P i (t). LetP i (t) denote the Kaplan-Meier estimator of P i (t), and letσ 2 i (t) denote the associated Greenwood variance estimator. Approximate inferences about the θ i s may then be based on: (i) estimatorsθ i obtained by substi-tutingP i (t) for P i (t) in the θ i -definitions; and (ii) standard errors s i in terms ofσ 2 i (t)s obtained (if necessary) through the "delta-method." For example, if θ i s are specified as rate differences, then each s 2 i is simply equal toσ 2 i (t) + σ 2 0 (t); whereas if θ i s are specified as log-odds-ratios as just described, then according to the delta-method, each s 2 i equalsσ 2 i (t)/[P i (t)Q i (t)] 2 +σ 2 0 (t)/[P 0 (t)Q 0 (t)] 2 wherê Q i (t) ≡ 1 −P i (t). We omit technical details.