A Multi-Stage Drop-the-Loser Design With Superiority Boundaries.

Introduction

Multi‐arm multi‐stage trials have the potential to reduce the duration and large cost of clinical trials and therefore have become increasingly popular [1,2,3]. Traditional multi‐arm multi‐stage (MAMS) trials involve comparing multiple active treatments to a common control treatment at predefined interim stages [4,5,6,7,8]. The interim stages can be defined to allow for the stopping of a treatment early for superiority or futility or both. Additionally in some designs, the entire trial can stop early for superiority [9,10].

While traditional MAMS designs are efficient in terms of expected sample size, they often require a larger maximum sample size compared to running a multi‐arm study. Academic investigator led trials can find this design feature particularly difficult, as normal funding mechanisms are often not flexible enough to accommodate variable sample sizes [11]. As a result, applicants typically need to request funding for the maximum possible sample size, so making the trials appear disproportionately expensive [12].

This led to the development of the drop‐the‐loser design, where at each interim a fixed number of active treatments are dropped, therefore reducing the maximum sample size of the design. A well‐studied approach is the two‐stage drop‐the‐loser design, in which a single interim analysis is used to only let the top performing treatment continue to the second stage [13]. Thall et al. [14] propose a two‐stage design where only one treatment may continue to the second stage, and it must also demonstrate sufficient efficacy, otherwise, the trial is terminated. Two‐stage drop‐the‐loser designs with further flexibility have also been proposed using closed testing procedures and combination tests [15,16]. These two‐stage designs have also been extended to consider multiple doses and historical control arms with different endpoints [17,18]. The idea of allowing for multiple interim analyses in which treatments can be dropped was considered in Stallard and Friede [19] with this work focusing on controlling the family‐wise error of these designs in a conservative manner. Wason et al. [12], propose a multi‐stage drop‐the‐loser approach in which a set number of treatments is dropped at the end of each stage, with the final treatment being tested for superiority.

Motivated by an academic investigator‐led trial into Post Operative Preventative Therapy for AtRial fibrillation after Thoracic Surgery (POPTARTS), we developed methodology building on Wason et al. [12] for a drop‐the‐loser design which also allows for the termination of the entire trial if all remaining treatments are found superior. This design therefore aims to gain the benefits of lower maximum sample size of the drop‐the‐loser design as well as decrease expected sample size seen in traditional MAMS designs. The motivating trial is further discussed in Section3.

As seen in the motivating example, this type of design can be used when one has multiple active treatments of interest, a common control arm, and the same primary outcome of interest for each treatment. This design supports the evaluation of multiple hypotheses with one for each active treatment. As discussed above, this design could be used when funding mechanisms are based on the maximum sample size; however, there is a desire to be able to stop the trial early if all remaining treatments are found superior.

Building on this trial, we present the methodology to calculate the required stopping boundaries and sample size for this type of multi‐stage superiority drop‐the‐loser design in order to control the pairwise error rate (PWER) [20,21,22] and the power under the least favorable configuration (LFC) [10,12,23]. The PWER is used as this ensures that the probability of making a type I error, for a given active treatment is controlled at the desired level. PWER is the focus as the motivating trial is designed to evaluate distinct treatments, therefore, it is argued that PWER should be used [21,24,25,26,27]. However, there are scenarios where multiplicity adjustments, such as family‐wise error rate (FWER) control or false discovery rate (FDR) control, may be necessary [28], and it is worth noting that this work does not cover these error controls. The power under the LFC is used as it controls the probability that a treatment that has a clinically relevant effect is found if one exists when the other active treatments have an uninteresting treatment effect. The methodology given in Section2accommodates a trial with any number of active arms and can be used to drop multiple treatments at each interim analysis. This methodology is then applied to the motivating trial example of POPTARTS in Section3and is compared to some alternative designs. Finally, the paper will conclude with a discussion.

Design

Consider a clinical trial with$K$experimental arms being tested against one common control arm. Each active treatment is tested at up to$J$analyses, with there being an equal number of analyses as treatments. Therefore, the maximum number of analyses,$J$, equals$K$. Let$n_{k , j}$denote the number of patients recruited to treatment$k \in \left\{0 , \ldots , K\right\}$by the end of stage$j \in \left\{1 , \ldots , J\right\}$. The null hypotheses of interest are$H_{1} : \delta_{1} \leq 0 , H_{2} : \delta_{2} \leq 0 , \ldots , H_{K} : \delta_{K} \leq 0 ,$where$\delta_{k}$is the difference in treatment effect between treatment$k$and the control treatment (treatment 0). We denote the set of all$\delta_{k}$as$\Delta$, so$\Delta = \left(\delta_{1} , \ldots , \delta_{K}\right)$. At each analysis$j$for treatment$k$,$H_{k}$is tested using the test statistic

Z_{k , j} = \frac{\overset{\overline}{\delta}_{k , j}}{\sqrt{V_{k , j}}} ,

where\overset{\overline}{\delta}_{k , j}is the difference in treatment effects of the observed patients on that given treatment$k$and the control treatment, up to stage$j$and$V_{k , j}$is the variance of the observed difference in treatment effects. It is assumed that$Z_{k , j}$follows a normal distribution$Z_{k , j} \sim \left(\frac{\delta_{k , j}}{\sqrt{V_{k , j}}} , 1\right)$.

At each analysis$j \leq J - 1$, the treatment with the smallest test statistic is dropped. Therefore, treatment$k^{'}$is dropped at stage$j$if$Z_{k^{'} , j} \leq Z_{k , j}$for all$k \in \left\{1 , \ldots K\right\} / \left\{m_{1} , m_{2} , \ldots , m_{j - 1}\right\}$, where$m_{i}$is the treatment dropped at stage$i$. For the remaining treatments if$Z_{k , j} \geq u_{k , j}$for all treatments left, then the trial stops for superiority, where$u_{k , j}$is the predefined boundary for treatment$k \in \left\{1 , \ldots , K\right\}$at stage$j \in \left\{1 , \ldots , J\right\}$. One can use the approach defined in this paper even if one wants to drop multiple treatments at each interim. In this case, one simply sets$n_{k , j} = n_{k , j^{'}}$and$u_{k , j} = u_{k , j^{'}}$for any interims that one wants to drop multiple treatments, similarly seen in Wason et al. [12].

PWER

The pairwise error rate (PWER) is the probability of recommending an ineffective treatment$k^{\star}$by the end of the study regardless of other experimental arms in the trial [29]. The PWER is$1 - P \left(\cap_{j = 1}^{J} B_{k^{\star} , j}\right)$with$B_{k , j}$being the event$Z_{k , j} \leq u_{k , j}$for all$k \in 1 , \ldots , K$. Therefore to control the PWER at a given level$\alpha$, one needs to find$u_{k , 1} , \ldots , u_{k , J}$so that$1 - P \left(\cap_{j = 1}^{J} B_{k^{\star} , j}\right) \leq \alpha$. When one has equal sample size for each active treatment, then the PWER is equal for every arm given they have the same boundaries, therefore, one just needs to calculate$u_{1} , \ldots , u_{J}$where$u_{j} = u_{k , j}$for all$k \in 1 , \ldots K$. This can be calculated using the multivariate normal distribution function similar to [10,30]. In theSupporting InformationSection1, the multivariate normal distribution equations used to calculate the PWER is given for the motivating example.

If one controls the PWER, this ensures that the type I error for a given treatment$k^{\star}$is controlled no matter what the treatment effect of the other treatments are, as shown in Theorem1.

The proof for Theorem1is given in theSupporting InformationSection2. The underlying principles of the proof are that if$1 - P \left(\cap_{j = 1}^{J} Z_{k^{\star} , j} \leq u_{k^{\star} , j}\right) \leq \alpha$then the PWER is controlled assuming no dropping of any treatment arms, but, includes accounting for the ability to stop the trial early for superiority. By additionally accounting for the possibility that treatment$k^{\star}$may be dropped at an intermediate stage, the type I error for treatment$k^{\star}$is further reduced. Furthermore, incorporating the fact that the arm can only stop early if all other treatments are found to be superior to the control further decreases the type I error for treatment$k^{\star}$. Overall, controlling the PWER therefore results in overly conservative control of the type I error for each treatment. However, this allows practical flexibility. For example, ignoring the requirement that all treatments must be found superior to the control means that, if needed, other treatments in the trial can be stopped earlier (e.g., for safety concerns).

It is worth noting that we have defined a type I error for a given treatment as occurring when, at the point the trial ends, its test statistic exceeds the superiority boundary, regardless of whether that treatment is the best‐performing option overall. This definition is used because, even if the treatment is not the best‐performing, we still wish to avoid drawing the conclusion that it is superior to the control when it is not. Requiring that the treatment also be the best‐performing option would further reduce the type I error. Originally, we require$Z_{k^{\star} , j} > u_{j}$and$Z_{k , j} > u_{j}$for all remaining treatments$k$. If we additionally require that treatment$k^{\star}$be the best for a type I error to occur, then a type I error can occur only if$Z_{k^{\star} , j} > u_{j}$,$Z_{k , j} > u_{j}$for all remaining treatments$k$, and$Z_{k^{\star} , j} > Z_{k , j}$for all remaining treatments$k$, which is a subset of the original event. Consequently, any procedure that controls the pairwise error rate (PWER) under the original definition of type I error will also control it under this more restrictive definition that requires$k^{\star}$to be the best‐performing treatment.

Power Under LFC

In this trial design, the trial can stop early at each of the$j$stages for superiority. This will happen when all the remaining treatments are found superior to the control. As recommended in the literature [9,10,12,31] when this happens one wants to ensure that if there is a treatment that is superior to the other active treatments, it is found to be the best performing of the treatments and is therefore the one that is recommended. We assume that any given treatment$k^{'}$is recommended when (i) its test statistic and the test statistics for all other treatments left at that given stage cross the corresponding upper boundar, and (ii) its test statistic is the largest one of the remaining treatments. The sample size is found such that the probability of rejecting$H_{k^{'}}$achieves power$1 - \beta$when$\delta_{k^{'}} = \theta^{'}$and$\delta_{k} = \theta_{0}$for$k \neq k^{'}$, where$\theta^{'}$is the minimum clinically interesting treatment effect, and$\theta_{0}$is the highest uninteresting treatment effect. This setting is known as the least favorable configuration (LFC). When calculating the power under the LFC one will likely assume equal sample size for each active treatment [10,12] which is assumed for the remainder of this section, in this case the power under the LFC is the same for every treatment.

The calculation of power under the LFC can be broken down into the event that the treatment of interest is recommended at each stage$j \in 1 , \ldots , J$, defined by$\Phi_{j}$. Without loss of generality assume that treatment 1 is the treatment of interest. The event that treatment 1 is found to be the best treatment and the trial stops for superiority at a given stage$j$, where$j < J$, can be split into five events. The first event is that treatment 1 is found superior to the control at the given stage,$A_{1 , j}$, where$A_{k , j} : Z_{k , j} > u_{k , j}$. The second event is that the test statistic for treatment 1 is larger than the rest of the treatments being tested at that stage. This event equals

$$\cap_{k \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{j - 1}\right\}} C_{1 , k , j} ,$$

where$m_{1} , \ldots , m_{j - 1}$are the treatments that have already been dropped at earlier stages and$C_{k , k^{\star} , j}$is the event that the test statistic for treatment$k$is greater than the test statistic for treatment$k^{\star}$at stage$j$,$C_{k , k^{\star} , j} : Z_{k , j} > Z_{k^{\star} , j}$. It is worth noting that$\cap_{k \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{j - 1}\right\}}$is the same as$\cap_{k \in \left\{2 , \ldots , K\right\}}$if$j = 1$.

The third event is all the test statistics of the other treatments being tested at the end of stage$j$are greater than the test statistic of treatment$m_{j}$, and they are also found superior to the control treatment at stage$j$,

$$\cap_{k \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{j}\right\}} \left(A_{k , j} \cap C_{k , m_{j} , j}\right) .$$

The fourth event is that treatment$m_{i}$for$i = 1 , \ldots , j - 1$is dropped at stage$i$,

$$\cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i}\right\}} C_{k , m_{i} , i}\right) .$$

The fifth event is that the trial did not stop for superiority at an earlier stage,

$$\cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cup_{k \in \left\{1 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i}\right\}} B_{k , i}\right) .$$

Using these five events and every possible$m_{1} , \ldots m_{j}$, one can calculate$\Phi_{j}$. Therefore, the event that treatment 1 is declared superior to the control at a given stage$j$, given$j < J$, is

$$\begin{aligned}\Phi_{j} & = \cup_{\genfrac{}{}{0pt}{}{m_{i} \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i - 1}\right\}}{i = 1 , \ldots , j}} \left[A_{1 , j} \cap \cap_{k \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{j - 1}\right\}} C_{1 , k , j} \cap\right) \\ & \cap_{k \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{j}\right\}} \left(A_{k , j} \cap C_{k , m_{j} , j}\right) \cap \cap_{i \in \left\{1 , \ldots , j - 1\right\}} \\ & \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i}\right\}} C_{k , m_{i} , i}\right) \\ & \left(\cap \cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cup_{k \in \left\{1 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i}\right\}} B_{k , i}\right)\right] .\end{aligned}$$

It is worth noting that

$$\cup_{\genfrac{}{}{0pt}{}{m_{i} \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i - 1}\right\}}{i = 1 , \ldots , j}}$$

represents the union taken over all possible permutations of$m_{1} , \ldots , m_{i - 1}$.

For treatment 1 to be found superior at the final stage, it must be the last treatment being tested. This therefore simplifies the calculation, so, the event that treatment 1 is declared superior to the control at stage$J$is

$$\begin{aligned}\Phi_{J} & = \cup_{\genfrac{}{}{0pt}{}{m_{i} \in \left\{2 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i - 1}\right\}}{i = 1 , \ldots , J - 1}} \left[A_{1 , J} \cap \cap_{i \in \left\{1 , \ldots , J - 1\right\}}\right) \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i}\right\}} C_{k , m_{i} , i}\right) \\ & \left(\cap \cap_{i \in \left\{1 , \ldots , J - 1\right\}} \left(\cup_{k \in \left\{1 , \ldots , K\right\} / \left\{m_{1} , \ldots , m_{i}\right\}} B_{k , i}\right)\right] .\end{aligned}$$

As$\Phi_{1} , \ldots , \Phi_{J}$are mutually exclusive events, the power under the LFC is

$$\sum_{j = 1}^{J} P \left(\Phi_{j}\right) ,$$

with$P \left(\Phi_{j}\right)$being calculated using the multivariate normal distribution.Supporting InformationSection3gives the multivariate normal equations used for the motivating example. One then calculates the$n_{j}$and$n_{0 , j}$to ensure the power under LFC is$\geq 1 - \beta$, where$n_{j} = n_{k , j}$for all$k \in 1 , \ldots , K$. The maximum sample size can then be found to be,$max \left(N\right) = \sum_{j = 1}^{J} n_{j} + n_{0 , J}$, where$N$is the sample size of the trial.

Expected Sample Size

The expected sample size can be calculated in a similar way to the power. First by calculating the probability of the event that the trials stops at each given stage. The event that the trial stops at each stage$j$is defined as$\Psi_{j}$. Given$j < J$,$\Psi_{j}$equals

$$\begin{aligned}\Psi_{j} & = \cup_{\genfrac{}{}{0pt}{}{k_{i} \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{i - 1}\right\}}{i = 1 , \ldots , j}} \left[\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{j}\right\}} \left(A_{k , j} \cap C_{k , k_{j} , j}\right)\right) \\ & \cap \cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{i}\right\}} C_{k , k_{i} , i}\right) \\ & \left(\cap \cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{i}\right\}} B_{k , i}\right)\right] .\end{aligned}$$

The event that the trial stops at stage$J$is

$$\begin{aligned}\Psi_{J} & = \cup_{\genfrac{}{}{0pt}{}{k_{i} \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{i - 1}\right\}}{i = 1 , \ldots , J}} \left[\cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{i}\right\}} C_{k , k_{i} , i}\right)\right) \\ & \left(\cap \cap_{i \in \left\{1 , \ldots , j - 1\right\}} \left(\cap_{k \in \left\{1 , \ldots , K\right\} / \left\{k_{1} , \ldots , k_{i}\right\}} B_{k , i}\right)\right] .\end{aligned}$$

To calculate the expected sample size one needs to include the sample size required for each event$P \left(\Psi_{j}\right)$. The expected sample size is therefore,

$$E \left(\right. N \left|\right. \Delta \left.\right) = \sum_{j = 1}^{J} \left(P \left(\Psi_{j}\right) \left(\sum_{i = 1}^{j - 1} n_{i} + \left(K - j + 1\right) n_{j} + n_{0 , j}\right)\right) .$$

Once again$P \left(\Psi_{j}\right)$can be calculated using the multivariate normal distribution, with theSupporting InformationSection4giving the equations used for the motivating example.

Pre‐Specified Design Parameters and Implementation

In order to use this design, one must pre‐specify several key parameters based on medical expertise. One must specify the desired level of control of the PWER,$\alpha$, and the desired power under the LFC,$1 - \beta$. Additionally, the number of treatments of interest should be prespecified.

To calculate the power, one must also specify the minimal clinically relevant effect, the highest uninteresting effect, and the variability of the outcome. These three parameters should be determined through discussion with clinicians and by reviewing the literature. One can use the approaches outlined in Whitehead and Jaki [32] to obtain the normal approximations for these parameters, which can then be used in the calculation of the stopping boundaries and the power.

Overall, to implement the approach discussed above with the defined parameters, one must first calculate the type I error for the given boundaries. This can be done using an iterative approach in which the PWER is calculated for the current boundaries, and if PWER is greater than$\alpha$, the boundaries are increased; if PWER is less than$\alpha$, the boundaries are decreased. This process is repeated until the boundaries control the PWER at a level between$\alpha$and$\alpha - \omega$, where$\omega$is a predefined tolerance. For example,$\omega$was set to 0.00001 in the motivating example. Therefore, the bounds were selected to achieve PWER within the range 0.02499–0.025.

Once the boundaries are calculated, the required sample size per stage can then be determined. This can be done by calculating the power for a small sample size, then increasing the sample size, one patient per stage until the study achieves at least$1 - \beta$power. Using this approach both the sample size and the boundaries are found. One can then calculate the expected sample size under any configuration of interest using the method set out in Section2.3.

Motivating Example

This design was motivated by a trial in Post Operative Preventative Therapy for AtRial fibrillation after Thoracic Surgery (POPTARTS). Post‐operative atrial fibrillation (POAF) is the most common arrhythmia encountered after thoracic surgery, with a prevalence of 10%–15% after lobectomy, 20%–30% after pneumonectomy, and 12%–37% after esophagectomy [33,34]. In the short term, the development of POAF is associated with worsened hemodynamic instability, increased risk of thromboembolic events, and prolonged intensive care unit and hospital stay [35]. In the long‐term, development of POAF confers a three to four‐fold increased risk of stroke, a four‐fold increase in myocardial infarction, and a three‐fold increase in mortality [36,37]. POAF is associated with significantly increased healthcare costs and resource utilization, adding over 10000 dollars to the cost of care when patients develop the arrhythmia [38]. Despite some evidence of efficacy of pharmacological prevention of POAF, as well as recommendations for use of prophylaxis in thoracic surgery guidelines, recent surveys have demonstrated poor uptake of these interventions by thoracic surgeons. In a 2014 survey by the American Association of Thoracic Surgeons, 56% of respondents did not routinely give any medication for the prevention of POAF [39]. Similarly, in a recent, yet to be published, survey of Canadian thoracic surgeons, 67% of respondents stated they did not routinely prescribe pharmacologic prophylaxis for POAF. Therefore, this motivated the desire to study the available pharmacologic treatments to see if it can be shown that any are superior to the current standard of care of no treatment. Due to this being a publicly funded, academic investigator led trial with a limited fixed maximum budget a multi‐stage drop‐the‐loser design was recommended. However, due to the severity of the POAF, the ability to stop early for superiority was also desired.

There are three active treatments of interest, carvedilol, magnesium sulphate, amiodarone, and the control treatment is the current standard of care of no treatments. One treatment will be dropped at each equally spaced stage, so there are an equal number of patients on each treatment at each stage. A clinically relevant difference is categorized as a 5% absolute decrease in atrial fibrillation—so a risk difference of interest of 5%—and an uninteresting treatment effect being less than 1% absolute decrease. The incidence of POAF after major thoracic surgery is approximately 12%. Therefore, a treatment with a clinically relevant effect would have incidence of POAF of less than 7%. The power of the trial was set to be 90% and PWER controlled at 2.5% one‐sided. Therefore, using the approach discussed in Whitehead and Jaki [32] and Jaki and Magirr [40], the normal approximation for the binary endpoint can be used. This gives a clinically relevant effect of$\theta^{'} = 0 . 594$based on the log‐odds ratio and an uninteresting treatment effect of$\theta_{0} = 0 . 98$based on the log‐odds ratio and an approximation for the variance of the observed difference in treatment effect of$V_{k , j} = \sigma^{2} \left(n_{k , j}^{- 1} + n_{0 , j}^{- 1}\right)$, where$\sigma^{2} = 9 . 47$[32,40,41]. The O'Brien and Fleming boundaries are used as they preserve a nominal significance level at the final analysis that is close to that of a single test procedure [42,43].

The calculations were carried out using R [44] with the method given here having the multivariate normal probabilities being calculated using the packagemvtnorm[45]. The code is available in theSupporting Information.

Alternative Approaches

The approach proposed in this paper was compared to five alternative designs. The first alternative design is to use a multi‐stage drop‐the‐loser design, without early termination for superiority. The second is to run multiple single stage trials with each one comparing an active treatment to the control. Building on this, we will also consider the alternative design of running separate trials but now with three stages for each trial. At each stage, the treatments will be tested for superiority using the O'Brien and Fleming boundaries. We will consider two versions of this approach, the first is using symmetric futility boundaries and the second is using futility boundaries set to zero as recommended in Magirr et al. [10]. Each individual trial controls power at 90% and controls the PWER at 2.5%. Because the trials are independent, it is possible for multiple trials to conclude that different active treatments are superior to the control. If, instead, one were to require control of the LFC across all trials simultaneously, the sample sizes shown in Table1would increase. In that scenario, the design would need to demonstrate not only that the treatment with the clinically relevant effect is superior to the control but also that it outperforms all other treatments. However, such cross‐trial calibration is not realistic, as power is typically controlled within each individual study rather than across multiple independent studies. For this reason, Section3.2considers designs in which each trial is powered independently at the 90% level.

Table: Operating characteristics of the proposed multi‐stage drop‐the‐loser design along with the operating characteristics of the alternative approaches.

The next alternative design will be a multi‐arm design. The final alternative design will be a multi‐arm multi‐stage (MAMS) design where the trial can stop early for superiority using the O'Brien and Fleming boundaries. This will follow the design given in Magirr et al. [10] now with PWER control, so the trial will stop once a treatment is found superior to the control, and the best performing treatment will be recommended. We will consider two versions of this MAMS design, the first using symmetric futility boundaries and the second using futility boundaries set to zero as recommended in Magirr et al. [10]. For the MAMS designs, we will control the power under the LFC at 90% and control the PWER at 2.5%.

Results

The O'Brien and Fleming stopping boundaries required to control the PWER at the 2.5% significance level are:

$$u_{1} = 3 . 47 , u_{2} = 2 . 45 , u_{3} = 2 . 0 ,$$

Table1reports the power under the LFC, the type I error of treatment$k^{\star}$under the global null, and the PWER. The PWER is equal to the type I error of treatment$k^{\star}$when its true effect is equal to that of the control and all the other active treatments are dropped from the trial before the first interim. For example, the other active treatments may be dropped early if they exhibit a strongly negative effect and therefore raise safety concerns. The equations used to calculate the type I error of treatment$k^{\star}$under the global null are given in theSupporting Informationsection 5. To calculate the type I error under the global null$\delta_{k} = 0$for all$k \in \left\{1 , 2 , 3\right\}$. Additionally,Supporting Informationsection 6 gives simulation results of the type I error for a given treatment under multiple different scenarios.

Table1shows the operating characteristics of the proposed design. These operating characteristics are calculated using pre‐defined configurations detailed at the start of Section3and in Table2. Table2gives the treatment effects used to ensure the designs control the PWER and power. It is worth noting that when calculating PWER, one only needs to consider the treatment effect of the treatment of interest as proven in Section2.1. This table also includes the pre‐defined stopping boundaries calculated for the designs based on a target PWER control of 2.5% (one‐sided). In Table1, the maximum sample size is given,$max \left(N\right)$. The expected sample size is studied under three configurations. The first is all the active treatment effects are the same as the control treatment which is known as the global null hypothesis, so$\delta_{k} = 0$for all$k \in \left\{1 , 2 , 3\right\}$, which we denote as$\Delta_{0} = \left(0 , 0 , 0\right)$. The second is$\delta_{1} = \theta^{'}$and$\delta_{2} = \delta_{3} = \theta_{0}$therefore under the configuration studied when looking at the power under the LFC, which we denote as$\Delta_{1} = \left(\theta^{'} , \theta_{0} , \theta_{0}\right)$. The final configuration is every active treatment has a clinically relevant effect, so$\delta_{k} = \theta^{'}$for all the$k \in \left\{1 , 2 , 3\right\}$, which we denote as$\Delta_{2} = \left(\theta^{'} , \theta^{'} , \theta^{'}\right)$. The results presented in Table1and Table2, including the stopping boundaries and operating characteristics, were calculated in R [44] using the code provided in theSupporting Information, together with the predefined inputs given in Section3, Table2and the design configurations described above for the expected sample size.

Table: Design parameters of the proposed multi‐stage drop‐the‐loser design along with the design parameters of the alternative approaches.

As can be seen in Table1, the maximum sample size of the proposed approach is 1854 patients which corresponds to 206 patients being recruited per remaining arm per stage. The expected sample size under the null configuration is 1846.5 which is very close to the maximum sample size, as the trial will only stop if all the remaining treatments are found superior. Therefore, under the null most of the time, the trial will continue to the final stage. Under the LFC, however, the sample size now drops by over 250 patients. This is because now there is an increased probability, 62.5%, that the trial stops early. This reduction in expected sample size is even greater when all the treatments have a clinically relevant effect, as now there is a 83.7% chance that the trial will stop early.

The alternative design of a multi‐stage drop‐the‐loser design without the ability to stop early for superiority reduces the maximum sample size by 27 patients compared to the proposed approach, as shown in Table1. There is also a decrease in expected sample size of 19.5 patients when under the global null, however, this design comes with a significant increase in expected sample size under the LFC, with an increase of 231.0 patients, or when all the treatments have a clinically relevant effect, with an increase of 342.3 patients.

Table1also shows the operating characteristics of the other alternative designs. For the multi‐arm design, there is an increase in both maximum sample size and in expected sample size compared to the proposed approach. The maximum sample size increases by 422 patients. Under the LFC, the expected sample size increases by 680.0. The MAMS approach with symmetric boundaries also results in a further increase in maximum sample size of 498 patients. Additionally, the expected sample sizes studied are greater than that of the proposed approach. The MAMS approach with futility boundaries of zero has the smallest expected sample size under the null, with a decease of 237.9 compared to the proposed design. The expected sample size under both the LFC and when all the treatments have a clinically relevant effect is greater than the proposed approach, with an increase of 7 and 107.5 patients, respectively. The MAMS approach with futility boundaries of zero has an increased maximum sample size of 582 patients compared to the proposed approach. Table2gives the superiority and futility boundaries used at the interim analyses for the multi‐stage designs.

For all the separate trials designs, the sample size is increased further. This is because now there is the need to recruit a different control group for each comparison. The maximum sample size increases by 1530 patients for the single stage separate trials design compared to the proposed approach. Under the LFC, the expected sample size increases by 1788.0 patients. Compared to the proposed approach, the maximum and expected sample sizes under the LFC increased by 1584 and 1577.5 patients for the multi‐stage separate trials with symmetric futility boundaries, and by 1728 and 1002.3 patients for the design with zero futility boundaries, respectively. As discussed in Section3.1, Table2shows that, when calculating the power in the separate trials case, only the treatment effect of the treatment of interest is used.

Discussion

Overall, this paper has presented a multi‐stage drop‐the‐loser design which allows for early stopping for superiority which was motivated from a trial in atrial fibrillation. In Section2, generalized methodology was presented to control both the PWER and power under the LFC of the study, along with equations needed to analytically calculate the expected sample size. Section3then compared the proposed approach to the alternative designs considered in the development of the trial. This section demonstrates that the proposed approach requires fewer patients than comparable multi‐arm or separate trial designs. For the motivating example, the early stopping for superiority does result in an increase in the maximum sample size compared to doing a drop‐the‐loser design with no early stopping. This increase is 27 patients, compared to a potential saving of 342.3 patients if all the treatments have a clinically relevant effect when using the superiority boundaries compared to not. There was also a potential increase in expected sample size of the proposed design compared to the MAMS design with futility boundaries equal to zero. This increase was 237.9 under the null configuration, however, the maximum sample size decreased by 582 patients by using the proposed approach.

Throughout this work, the focus has been on controlling the PWER, as this trial is designed to evaluate distinct treatments, therefore, this type I error control is appropriate [21,24,25,26,27]. However, there are scenarios where multiplicity adjustments, such as family‐wise error rate (FWER) control or false discovery rate (FDR) control, may be necessary [28]. This could include cases where multiple doses of the same treatment are tested or where regulatory guidelines demand such control. The proposed approach could be applied in this setting by using a Bonferroni correction [46,47], although this would be overly conservative. Consequently, future work could focus on extending the current methodology to control these errors exactly. Additional research could also evaluate whether the drop‐the‐loser design can be integrated with an all‐pairwise design, thereby removing the need for a control treatment [48,49].

This paper introduces a framework for a drop‐the‐loser design with superiority boundaries, centered around normally distributed test statistics. Using the methodology proposed by Whitehead and Jaki [32] and Jaki and Magirr [40], this approach can accommodate other endpoints, including binary endpoints which was used for our motivating example. When applying this methodology, it is important to be aware of potential computational difficulties in the evaluation of high‐dimensional multivariate normal distributions when considering trials with many arms. In such cases, one may consider the techniques proposed in Blondell et al. [50] for managing high‐dimensional multivariate normal computations, or alternatively, use a simulation‐based approach. A simulation‐based approach also has a further benefit when using non‐normally distributed endpoints as it can help remove the bias caused from the normality assumptions and the normality approximations used in calculating the variance and covariance matrix [32,40,41]. As done in Wason et al. [12] in this work when calculating the covariance matrix, it has been assumed that$V_{k , j} = \sigma^{2} \left(n_{k , j}^{- 1} + n_{0 , j}^{- 1}\right)$, where$n_{k , j} = n_{k^{\star} , j}$for all$k , k^{\star}$. Therefore, an area for further work is calculating the multivariate normal equations for when$\sigma_{k} \neq \sigma_{k^{\star}}$where$\sigma_{k}^{2}$is the variance of treatment$k$and$n_{k , j} \neq n_{k^{\star} , j}$. It is worth noting that the boundaries for PWER control can be calculated using pre‐existing R packages such asgsDesign[51] andrpact[52].

To use such a design, one must have multiple active treatments of interest, a common control arm, and the same primary outcome for each treatment. To implement the design, several key parameters must be prespecified, as detailed in Section2.4. It is also worth noting, as with other drop‐the‐loser designs, that if all the treatments have a clinically relevant effect, there is a high probability that one of these effective treatments may be dropped from the study before reaching the superiority boundaries. Therefore, if the aim is to identify all clinically relevant treatments, one should consider whether a MAMS design would be more appropriate.

Overall, the proposed design incorporates some of the advantages of both MAMS and drop‐the‐loser approaches. It retains the MAMS benefit of allowing early stopping for superiority, which can lead to substantial reductions in expected sample size. At the same time, the drop‐the‐loser component helps limit the maximum sample size required by systematically eliminating less promising arms during the trial.

Author Contributions

Peter Greenstreet derived the equations and wrote the paper with the support of Manel Khan. Salmaan Kanji, Motacedian Pouya, Andrew Seely, and Stephanie Sibley were key to the design of this trial with their clinical expertise, and reviewed the manuscript. Tim Ramsay oversaw the entire project and contributed greatly to the manuscript development.

Funding

This work was supported by the Canadian Institutes of Health Research (Grant Nos. 184898, 262556).

Conflicts of Interest

The authors declare no conflicts of interest.

Acknowledgments

Peter Greenstreet wishes to acknowledge CAN TAP TALENT for its role in supporting the completion of this manuscript. The CAN TAP TALENT CTTP is funded by the Canadian Institutes of Health Research (CIHR) – Grant 184898. Peter Greenstreet was also supported by a CANSTAT trainee award funded by CIHR grant 262556. We would like to thank the editor, associate editor, and the two reviewers for their useful comments and suggestions.

Data Availability Statement

The data that supports the findings of this study are available in theSupporting Informationof this article.

Associated Data

Data Availability Statement

The data that supports the findings of this study are available in theSupporting Informationof this article.

References

1. Mullard A., “How Much Do Phase III Trials Cost?,” Nature Reviews. Drug Discovery 17, no. 11 (2018): 777. doi.org/10.1038/nrd.2018.198
2. Noor N. M., Love S. B., Isaacs T., Kaplan R., Parmar M. K. B., and Sydes M. R., “Uptake of the Multi‐Arm Multi‐Stage (MAMS) Adaptive Platform Approach: A Trial‐Registry Review of Late‐Phase Randomised Clinical Trials,” BMJ Open 12, no. 3 (2022): e055615. doi.org/10.1136/bmjopen-2021-055615
3. Stallard N., Hampson L., Benda N., et al., “Efficient Adaptive Designs for Clinical Trials of Interventions for COVID‐19,” Statistics in Biopharmaceutical Research 12, no. 4 (2020): 483–497. doi.org/10.1080/19466315.2020.1790415
4. Greenstreet P., Jaki T., Bedding A., and Mozgunov P., “Design of Platform Trials With a Change in the Control Treatment Arm,” Biometrics 81, no. 2 (2025): ujaf073. doi.org/10.1093/biomtc/ujaf073
5. Royston P., Parmar M. K. B., and Qian W., “Novel Designs for Multi‐Arm Clinical Trials With Survival Outcomes With an Application in Ovarian cancer,” Statistics in Medicine 22, no. 14 (2003): 2239–2256. doi.org/10.1002/sim.1430
6. Serra A., Mozgunov P., and Jaki T., “An Order Restricted Multi‐Arm Multi‐Stage Clinical Trial Design,” Statistics in Medicine 41, no. 9 (2022): 1613–1626. doi.org/10.1002/sim.9314
7. Urach S. and Posch M., “Multi‐Arm Group Sequential Designs With a Simultaneous Stopping Rule,” Statistics in Medicine 35, no. 30 (2016): 5536–5550. doi.org/10.1002/sim.7077
8. Wason J. M. S. and Jaki T., “Optimal Design of Multi‐Arm Multi‐Stage Trials,” Statistics in Medicine 31, no. 30 (2012): 4269–4279. doi.org/10.1002/sim.5513
9. Greenstreet P., Jaki T., Bedding A., Harbron C., and Mozgunov P., “A Multi‐Arm Multi‐Stage Platform Design That Allows Preplanned Addition of Arms While Still Controlling the Family‐Wise Error,” Statistics in Medicine 43, no. 19 (2024): 3613–3632. doi.org/10.1002/sim.10135
10. Magirr D., Jaki T., and Whitehead J., “A Generalized Dunnett Test for Multi‐Arm Multi‐Stage Clinical Studies With Treatment Selection,” Biometrika 99, no. 2 (2012): 494–501.
11. Kairalla J. A., Coffey C. S., Thomann M. A., and Muller K. E., “Adaptive Trial Designs: A Review of Barriers and Opportunities,” Trials 13 (2012): 1–9. doi.org/10.1186/1745-6215-13-145
12. Wason J., Stallard N., Bowden J., and Jennison C., “A Multi‐Stage Drop‐The‐Losers Design for Multi‐Arm Clinical Trials,” Statistical Methods in Medical Research 26, no. 1 (2017): 508–524. doi.org/10.1177/0962280214550759
13. Sampson A. R. and Sill M. W., “Drop‐The‐Losers Design: Normal Case,” Biometrical Journal: Journal of Mathematical Methods in Biosciences 47, no. 3 (2005): 257–268. doi.org/10.1002/bimj.200410119
14. Thall P. F., Simon R., and Ellenberg S. S., “A Two‐Stage Design for Choosing Among Several Experimental Treatments and a Control in Clinical Trials,” Biometrics 45 (1989): 537–547.
15. Bretz F., Schmidli H., König F., Racine A., and Maurer W., “Confirmatory Seamless Phase II/III Clinical Trials With Hypotheses Selection at Interim: General Concepts,” Biometrical Journal: Journal of Mathematical Methods in Biosciences 48, no. 4 (2006): 623–634. doi.org/10.1002/bimj.200510232
16. Schmidli H., Bretz F., Racine A., and Maurer W., “Confirmatory Seamless Phase ii/iii Clinical Trials With Hypotheses Selection at Interim: Applications and Practical Considerations,” Biometrical Journal 48, no. 4 (2006): 635–643. doi.org/10.1002/bimj.200510231
17. Abbas R., Wason J., Michiels S., and Le Teuff G., “A Two‐Stage Drop‐The‐Losers Design for Time‐To‐Event Outcome Using a Historical Control Arm,” Pharmaceutical Statistics 21, no. 1 (2022): 268–288. doi.org/10.1002/pst.2168
18. Joshua Chen Y., DeMets D. L., and Gordon Lan K., “Some Drop‐The‐Loser Designs for Monitoring Multiple Doses,” Statistics in Medicine 29, no. 17 (2010): 1793–1807. doi.org/10.1002/sim.3958
19. Stallard N. and Friede T., “A Group‐Sequential Design for Clinical Trials With Treatment Selection,” Statistics in Medicine 27, no. 29 (2008): 6209–6227. doi.org/10.1002/sim.3436
20. Choodari‐Oskooei B., Bratton D. J., Gannon M. R., Meade A. M., Sydes M. R., and Parmar M. K., “Adding New Experimental Arms to Randomised Clinical Trials: Impact on Error Rates,” Clinical Trials 17, no. 3 (2020): 273–284. doi.org/10.1177/1740774520904346
21. Howard D. R., Hockaday A., Brown J. M., et al., “A Platform Trial in Practice: Adding a New Experimental Research Arm to the Ongoing Confirmatory FLAIR Trial in Chronic Lymphocytic Leukaemia,” Trials 22, no. 1 (2021): 38. doi.org/10.1186/s13063-020-04971-2
22. Sydes M. R., Parmar M. K. B., James N. D., et al., “Issues in Applying Multi‐Arm Multi‐Stage Methodology to a Clinical Trial in Prostate cancer: The MRC STAMPEDE Trial,” Current Controlled Trials in Cardiovascular Medicine 10, no. 1 (2009): 39. doi.org/10.1186/1745-6215-10-39
23. Pushpakom S. P., Taylor C., Kolamunnage‐Dona R., et al., “Telmisartan and Insulin Resistance in HIV (TAILoR): Protocol for a Dose‐Ranging Phase II Randomised Open‐Labelled Trial of Telmisartan as a Strategy for the Reduction of Insulin Resistance in HIV‐Positive Individuals on Combination Antiretroviral Therapy,” BMJ Open 5, no. 10 (2015): e009566. doi.org/10.1136/bmjopen-2015-009566
24. Cook R. J. and Farewell V. T., “Multiplicity Considerations in the Design and Analysis of Clinical Trials,” Journal of the Royal Statistical Society, Series A 159, no. 1 (1996): 93–110.
25. Howard D. R., Brown J. M., Todd S., and Gregory W. M., “Recommendations on Multiple Testing Adjustment in Multi‐Arm Trials With a Shared Control Group,” Statistical Methods in Medical Research 27, no. 5 (2018): 1513–1530. doi.org/10.1177/0962280216664759
26. Molloy S. F., White I. R., Nunn A. J., Hayes R., Wang D., and Harrison T. S., “Multiplicity Adjustments in Parallel‐Group Multi‐Arm Trials Sharing a Control Group: Clear Guidance Is Needed,” Contemporary Clinical Trials 113 (2022): 106656. doi.org/10.1016/j.cct.2021.106656
27. Parker R. A. and Weir C. J., “Non‐Adjustment for Multiple Testing in Multi‐Arm Trials of Distinct Treatments: Rationale and Justification,” Clinical Trials 17, no. 5 (2020): 562–566. doi.org/10.1177/1740774520941419
28. Wason J. M. S., Stecher L., and Mander A. P., “Correcting for Multiple‐Testing in Multi‐Arm Trials: Is It Necessary and Is It Done?,” Trials 15, no. 1 (2014): 364. doi.org/10.1186/1745-6215-15-364
29. Bratton D. J., Parmar M. K., Phillips P. P., and Choodari‐Oskooei B., “Type I Error Rates of Multi‐Arm Multi‐Stage Clinical Trials: Strong Control and Impact of Intermediate Outcomes,” Trials 17 (2016): 1–8. doi.org/10.1186/s13063-016-1382-5
30. Greenstreet P., Jaki T., Bedding A., and Mozgunov P., “A Preplanned Multi‐Stage Platform Trial for Discovering Multiple Superior Treatments With Control of FWER and Power,” Biometrical Journal 67, no. 1 (2025): e70025. doi.org/10.1002/bimj.70025
31. Dunnett C. W., “A Multiple Comparison Procedure for Comparing Several Treatments With a Control,” Journal of the American Statistical Association 50, no. 272 (1955): 1096–1121, 10.1080/01621459.1955.10501294. doi.org/10.1080/01621459.1955.10501294
32. Whitehead J. and Jaki T., “One‐and Two‐Stage Design Proposals for a Phase II Trial Comparing Three Active Treatments With Control Using an Ordered Categorical Endpoint,” Statistics in Medicine 28, no. 5 (2009): 828–847. doi.org/10.1002/sim.3508
33. Seesing M. F., Borggreve A. S., Ruurda J. P., and van Hillegersberg R., “New‐Onset Atrial Fibrillation After Esophagectomy for cancer,” Journal of Thoracic Disease 11, no. Suppl 5 (2019): S831–S834. doi.org/10.21037/jtd.2019.02.03
34. Vaporciyan A. A., Correa A. M., Rice D. C., et al., “Risk Factors Associated With Atrial Fibrillation After Noncardiac Thoracic Surgery: Analysis of 2588 Patients,” Journal of Thoracic and Cardiovascular Surgery 127, no. 3 (2004): 779–786. doi.org/10.1016/j.jtcvs.2003.07.011
35. Ivanovic J., Maziak D. E., Ramzan S., et al., “Incidence, Severity and Perioperative Risk Factors for Atrial Fibrillation Following Pulmonary Resection,” Interactive Cardiovascular and Thoracic Surgery 18, no. 3 (2014): 340–346. doi.org/10.1093/icvts/ivt520
36. Albini A., Malavasi V. L., Vitolo M., et al., “Long‐Term Outcomes of Postoperative Atrial Fibrillation Following Non Cardiac Surgery: A Systematic Review and Metanalysis,” European Journal of Internal Medicine 85 (2021): 27–33. doi.org/10.1016/j.ejim.2020.12.018
37. AlTurki A., Marafi M., Proietti R., et al., “Major Adverse Cardiovascular Events Associated With Postoperative Atrial Fibrillation After Noncardiac Surgery: A Systematic Review and meta‐Analysis,” Circulation: Arrhythmia and Electrophysiology 13, no. 1 (2020): e007437. doi.org/10.1161/CIRCEP.119.007437
38. LaPar D. J., Speir A. M., Crosby I. K., et al., “Postoperative Atrial Fibrillation Significantly Increases Mortality, Hospital Readmission, and Hospital Costs,” Annals of Thoracic Surgery 98, no. 2 (2014): 527–533. doi.org/10.1016/j.athoracsur.2014.03.039
39. Frendl G., Sodickson A. C., Chung M. K., et al., “2014 Aats Guidelines for the Prevention and Management of Peri‐Operative Atrial Fibrillation and Flutter (Poaf) for Thoracic Surgical Procedures,” Journal of Thoracic and Cardiovascular Surgery 148, no. 3 (2014): e153. doi.org/10.1016/j.jtcvs.2014.06.036
40. Jaki T. and Magirr D., “Considerations on Covariates and Endpoints in Multi‐Arm Multi‐Stage Clinical Trials Selecting All Promising Treatments,” Statistics in Medicine 32, no. 7 (2013): 1150–1163. doi.org/10.1002/sim.5669
41. Jaki T. F., Pallmann P. S., and Magirr D., “The R Package MAMS for Designing Multi‐Arm Multi‐Stage Clinical Trials,” Journal of Statistical Software 88, no. 4 (2019): 1–25.
42. Chen L. M., Ibrahim J. G., and Chu H., “Flexible Stopping Boundaries When Changing Primary Endpoints After Unblinded Interim Analyses,” Journal of Biopharmaceutical Statistics 24, no. 4 (2014): 817–833. doi.org/10.1080/10543406.2014.901341
43. O'Brien P. C. and Fleming T. R., “A Multiple Testing Procedure for Clinical Trials,” Biometrics 35, no. 3 (1979): 549–556.
44. R Core Team , R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2021),https://www.R‐project.org/.
45. Genz A., Bretz F., Miwa T., et al., “mvtnorm: Multivariate Normal and t Distributions,” (2021),https://CRAN.R‐project.org/package=mvtnorm.
46. Bonferroni C., “Teoria statistica delle classi e calcolo delle probabilita,” Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8 (1936): 3–62.
47. Dunn O. J., “Multiple Comparisons Among Means,” Journal of the American Statistical Association 56, no. 293 (1961): 52–64.
48. Greenstreet P., Jaki T., Bedding A., and Mozgunov P., “A Multi‐Arm Multi‐Stage Design for Trials With All Pairwise Testing,” (2025), arXiv Preprint arXiv:2502.07013.
49. Whitehead J., Desai Y., and Jaki T., “Estimation of Treatment Effects Following a Sequential Trial of Multiple Treatments,” Statistics in Medicine 39, no. 11 (2020): 1593–1609. doi.org/10.1002/sim.8497
50. Blondell L., Kos M. Z., Blangero J., and Göring H. H. H., “Genz and Mendell‐Elston Estimation of the High‐Dimensional Multivariate Normal Distribution,” Algorithms 14, no. 10 (2021): 296.
51. Anderson K., “gsDesign: Group Sequential Design,” (2025),https://CRAN.R‐project.org/package=gsDesign.
52. Wassmer G., Pahlke F., Jensen T., Bove D. S., Schueuerhuis S., and Muetze T., “rpact: Confirmatory Adaptive Clinical Trial Design and Analysis,” (2025),https://cran.r‐project.org/package=rpact.

---

Republished from the open web under CC-BY. Authors: Greenstreet P, Khan M, Kanji S, Motazedian P, Seely A, Sibley S, Ramsay T. Read the original.