By Exploristics CDSO, Kim Hacquoil
Aiming for 80% or 90% power for a clinical study is standard practice, however, it is possible that using power as a proxy for probability of success for decision-making is partly to blame for the higher-than-expected failure rates in Phase III studies.
What is power?
Statistical power is the probability of achieving a small p-value conditional on an assumed effect size. The FDA guidance says that the Type II error (1-power, or probability of trial success given the treatment is ineffective) is conventionally set to 10/20%. A key part is that the FDA says that it’s in the Sponsor’s interest to keep this as low as possible i.e., keep power high. A low power is really a Sponsor’s risk, as a low power may lead to not declaring success or progressing a drug to the next stage of development, when in fact it actually works.
Why is power not probability of success?
Often people think that high power means a high likelihood of success for a trial. However, power is only half the story. It is not the same as probability of success because it is conditional on knowing, for certain, the effect size of a treatment in the full patient population. The true effect size will never be known as we are not able to measure every single patient in every situation of interest. We can only estimate it through carrying out clinical trials. Drug discovery and development is a process of navigating the fact that we will never be certain of a treatment’s true effect, and we are constantly trying to build evidence to be as certain as we can (or feel comfortable with).
Using power for decision-making
Power is often used as a proxy for probability of success in decision-making – this is fraught with errors and can lead to ineffective, unproductive, or inaccurate decisions. Imagine having two different compounds in development. One which has a novel mechanism with very limited historical data and information but is a potential treatment for an indication with a high unmet medical need. A second which has a well-established mechanism of action and lots of historical data both in the target patient population but also other indications. You could design a Phase III study with the same power for these two compounds, but it’s probably safe to say you have a different belief on which is more likely to succeed, and which one carries far more risk. So only considering power within the planning stages doesn’t help you in quantifying the understanding you have regarding risks and potential for success/failure. Clearly you will want to approach the development differently but using only power isn’t the answer here!
A more holistic and informed approach to decision-making
The power for a clinical trial can vary greatly depending on the assumptions we make for the (unknown) true effect size. We should build into the decision-making framework the chance that we are wrong with our assumption. We need to think about how certain we are in the true effect size and what our prior belief is before we start the study. Statisticians have methods which can incorporate different beliefs and use this to quantify the unconditional probability of success for a study, i.e., which is not conditional on assuming we know the true treatment effect with 100% certainty. Using these methods helps project teams to better quantify how likely a study is to succeed and form a more holistic view of the overall development program. There is no magic number that is required for the probability of success, it will be dependent on many different factors. But starting to quantify this in a more transparent way is the first step in a better decision-making process. It ensures that the studies we design are done so with more realisation and appreciation of the risks involved and will mean we can set them up to succeed. Which ultimately gives us more “power” to make the right decisions for our patients.
Read more:
P value pitfalls in clinical trials
Statistical Consulting Services
Watch more: