Sex Differences in Clinical Trial Recruiting

The following article investigates several systematic reviews into sex and gender representation in individual clinical trial patient populations. In these studies sex ratios are assessed and evaluated by various factors such as clinical trial phase, disease type under investigation and disease burden in the population. Sex differences in the reporting of safety and efficacy outcomes are also investigated. In many cases safety and efficacy outcomes are pooled, rather than reported individually for each sex, which can be problematic when findings are generalised to the wider population. In order to get the dosage right for different body compositions and avoid unforeseen outcomes in off label use or when a novel therapeutic first reaches the market, it is important to report sex differences in clinical trials. Due to the unique nuances of disease types and clinical trial phases it is important to realise that a 50-50 ratio of male to female is not always the ideal or even appropriate in every clinical study design. Having the right sex balance in your clinical trial population will improve the efficiency and cost-effectiveness of your study. Based upon the collective findings a set of principles are put forth to guide the researcher in determining the appropriate sex ratio for their clinical trial design.

Sex difference by clinical trial phase

  • variation in sex enrolment ratios for clinical trial phases
  • females less likely to participate in early phases, due to increased risk of adverse events
  • under-representation of women in phase III when looking at disease prevalence

It has been argued that female representation in clinical trials is lacking, despite recent efforts to mitigate the gap. US data from 2000-2020 suggests that trial phase has the greatest variation in enrolment when compared to other factors, with median female enrolment being 42.9%, 44.8%, 51.7%, and 51.1% for phases I, I/II to II, II/III to III, and IV4. This shows that median female enrolment gradually increases as trials progress, with the difference in female enrolment between the final phases II/III to III and IV being <1%. Additional US data on FDA approved drugs including trials from as early as 1993 report that female participation in clinical trials is 22%, 48%, and 49% for trial phases I, II, and III respectively2. While the numbers for participating sexes are almost equal in phases II and III, women make up only approximately one fifth of phase I trial populations in this dataset2. The difference in reported participation for phase I trials between the datasets could be due to an increase in female participation in more recent years. The aim of a phase I trial is to evaluate safety and dosage, so it comes as no surprise that women, especially those of childbearing age, are often excluded due to potential risks posed to foetal development.

In theory, women can be included to a greater extent as trial phases progress and the potential risk of severe adverse events decreases. By the time a trial reaches phase III, it should ideally reflect the real-world disease population as much as possible. European data for phase III trials from 2011-2015 report 41% of participants being female1, which is slightly lower than female enrolment in US based trials. 26% of FDA approved drugs have a >20% difference between the proportion of women in phase II & III clinical trials and the prevalence of women in the US with the disease2, and only one of these drugs shows an over-representation of women.

Reporting of safety and efficacy by sex difference

  • Both safety and efficacy results tend to differ by sex.
  • Reporting these differences is inconsistent and often absent
  • Higher rates of adverse events in women are possibly caused by less involvement or non stratification in dose finding and safety studies.
  • There is a need to enforce analysis and reporting of sex differences in safety and efficacy data

Sex differences in response to treatment regarding both efficacy and safety have been widely reported. Gender subgroup analyses regarding efficacy can reveal whether a drug is more or less effective in one sex than the other. Gender subgroup analyses for efficacy are available for 71% of FDA approved drugs, and of these 11% were found to be more efficacious in men and 7% in women2. Alternatively, only 2 of 22 European Medicines Agency approved drugs examined were found to have efficacy differences between the sexes1. Nonetheless, it is important to study the efficacy of a new drug on all potential population subgroups that may end up taking that drug.

The safety of a treatment also differs between the sexes, with women having a slightly higher percentage (p<0.001) of reported adverse events (AE) than men for both treatment and placebo groups in clinical trials1. Gender subgroup analyses regarding safety can offer insights into the potential risks that women are subjected to during treatment. Despite this, gender specific safety analyses are available for only 45% of FDA approved drugs, with 53% of these reporting more side effects in women2. On average, women are at a 34% increased risk of severe toxicity for each cancer treatment domain, with the greatest increased risk being for immunotherapy (66%). Moreover, the risk of AE is greater in women across all AE types, including patient-reported symptomatic (female 33.3%, male 27.9%), haematologic (female 45.2%, male 39.1%) and objective non-haematologic (female 30.9%, male 29.0%)3. These findings highlight the importance of gender specific safety analyses and the fact that more gender subgroup safety reporting is needed. More reporting will increase our understanding of sex-related AE and could potentially allow for sex-specific interventions in the future.

Sex differences by disease type and burden

  • Several disease categories have recently been associated with lower female enrolment
  • Men are under-represented as often as women when comparing enrolment to disease burden proportions
  • There is a need for trial participants to be recruited on a case-by-case basis, depending on the disease.

Sex differences by disease type

When broken down by disease type, the sex ratio of clinical trial participation shows a more nuanced picture. Several disease categories have recently been associated with lower female enrolment, compared to other factors including trial phase, funding, blinding, etc4. Women comprised the smallest proportions of participants in US-based trials between 2000-2020 for cardiology (41.4%), sex-non-specific nephrology and genitourinary (41.7%), and haematology (41.7%) clinical trials4. Despite women being

proportionately represented in European phase III clinical studies between 2011-2015 for depression, epilepsy, thrombosis, and diabetes, they were significantly under-represented for hepatitis C, HIV, schizophrenia, hypercholesterolaemia, and heart failure and were not found to be overrepresented in trials for any of the disease categories examined1. This shows that the gap in gender representation exists even in later clinical trial phases when surveying disease prevalence, albeit to a lesser extent. Examining disease burden shows that the gap is even bigger than anticipated and includes the under-representation of both sexes.

Sex Differences by Disease Burden

It is not until the burden of disease is considered that men are shown to be under-represented as often as women. Including burden of disease can depict proportionality relative to the variety of disease manifestations between men and women. It can be measured as disability-adjusted life years (DALYs), which represent the number of healthy years of life lost due to the disease. Despite the sexes each making up approximately half of clinical trial participants overall in US-based trials between 2000-2020, all disease categories showed an under-representation of either women or men relative to disease burden, except for infectious disease and dermatologic clinical trials4. Women were under-represented in 7 of 17 disease categories, with the greatest under-representation being in oncology trials, where the difference between the number of female trial participants and corresponding DALYs is 3.6%. Men were under-represented compared with their disease burden in 8 of 17 disease categories, with the greatest difference being 11.3% for musculoskeletal disease and trauma trials.4 Men were found to be under-represented to a similar extent to women, suggesting that the under-representation of either sex could be by coincidence. Alternatively, male under-representation could potentially be due to the assumption of female under-representation leading to overcorrection in the opposite direction. It should be noted that these findings would benefit from statistical validation, although they illustrate the need for clinical trial participants to be recruited on a case-by-case basis, depending on the disease.

Takeaways to improve your patient sample in clinical trial recruiting:

  1. Know the disease burden/DALYs of your demographics for that disease.
  2. Try to balance the ratio of disease burden to the appropriate demographics for your disease
  3. Aim to recruit patients based on these proportions
  4. Stratify clinical trial data by the relevant demographics in your analysis. For example: toxicity, efficacy, adverse events etc should always be analyses separately for male and female to come up wit the respective estimates.
  5. Efficacy /toxicity etc should always be reported separately for male and female. reporting difference by ethnicity is also important as many diseases differentially affect certain ethnicity and the corresponding therapeutics can show differing degrees of efficacy and adverse events.

The end goal of these is that medication can be more personalised and any treatment given is more likely to help and less likely to harm the individual patient.

Conclusions

There is room for improvement in the proportional representation of both sexes in clinical trials and knowing a disease demographic is vital to planning a representative trial. Assuming the under-representation is on the side of female rather than male may lead to incorrect conclusions and actions to redress the balance. Taking demographic differences in disease burden into account when recruiting trial participants is needed. Trial populations that more accurately depict the real-world populations will allow a therapeutic to be tailored to the patient.

Efficacy and safety findings highlight the need for clinical study data to be stratified by sex, so that respective estimates can be determined. This enables more accurate, sex/age appropriate dosing that will maximise treatment efficacy and patient safety, as well as minimise the chance of adverse events. This also reduces the risks associated with later off label use of drugs and may avoid modern day tragedies resembling the thalidomide tragedy. Moreover, efficacy and adverse events should always be reported separately for men and women, as the evidence shows their distinct differences in response to therapeutics.

References:

1. Dekker M, de Vries S, Versantvoort C, Drost-van Velze E, Bhatt M, van Meer P et al. Sex Proportionality in Pre-clinical and Clinical Trials: An Evaluation of 22 Marketing Authorization Application Dossiers Submitted to the European Medicines Agency. Frontiers in Medicine. 2021;8.

2. Labots G, Jones A, de Visser S, Rissmann R, Burggraaf J. Gender differences in clinical registration trials: is there a real problem?. British Journal of Clinical Pharmacology. 2018;84(4):700-707.

3. Unger J, Vaidya R, Albain K, LeBlanc M, Minasian L, Gotay C et al. Sex Differences in Risk of Severe Adverse Events in Patients Receiving Immunotherapy, Targeted Therapy, or Chemotherapy in Cancer Clinical Trials. Journal of Clinical Oncology. 2022;40(13):1474-1486.

4. Steinberg J, Turner B, Weeks B, Magnani C, Wong B, Rodriguez F et al. Analysis of Female Enrollment and Participant Sex by Burden of Disease in US Clinical Trials Between 2000 and 2020. JAMA Network Open. 2021;4(6):e2113749.

Estimating the Costs Associated with Novel Pharmaceutical development: Methods and Limitations.

Data sources for cost analysis of drug development R&D and clinical trials

Cost estimates for pre-clinical and clinical development across the pharmaceutical industry differ based on several factors. One of these is the source of data used by each costing study to inform these estimates. Several studies use private data, which can include confidential surveys filled out by pharmaceutical firms/clinical trial units and random samples from private databases3,9,10,14,15,16. Other studies have based their cost estimates upon publicly available data, such as data from the FDA/national drug regulatory agencies, published peer-reviewed studies, and other online public databases1,2,12,13,17.

Some have questioned the validity of using private surveys from large multinational pharmaceutical companies to inform cost estimates, saying that survey data may be artificially inflated by pharmaceutical companies to justify high therapeutic prices 18,19,20. Another concern is that per trial spending by larger pharmaceutical companies and multinational firms would far exceed the spending of start-ups and smaller firms, meaning cost estimates made based on data from these larger companies would not be representative of smaller firms.

Failure rate of R&D and clinical trial pipelines

Many estimates include the cost of failures, which is especially the case for cost estimates “per approved drug”. As many compounds enter the clinical trial pipeline, the cost to develop one approved drug/compound includes cost of failures by considering the clinical trial success rate and cost of failed compounds. For example, if 100 compounds enter phase I trials, and 2 compounds are approved, the clinical cost per approved drug would include the amount spent on 50 compounds.

The rate of success used can massively impact cost estimates, where a low success rate or high failure rate will lead to much higher costs per approved drug. The overall probability of clinical success may vary by year and has been estimated at a range of values including 7.9%21, 11.83%10, and 13.8%22. There are concerns that some studies suggesting lower success rates have relied on small samples from industry curated databases and are thereby vulnerable to selection bias12,22.

Success rates per phase transition also affects overall costs. When more ultimately unsuccessful compounds enter late clinical trial stages, the higher the costs are per approved compound. In addition, success rates are also dependent on therapeutic area and patient stratification by biomarkers, among other factors. For example, one study estimated the lowest success rate at 1.6% for oncological trials without biomarker use compared with a peak success rate of 85.7% for cardiovascular trials utilising biomarkers22. While aggregate success rates can be used to estimate costs, using specific success rates will be more accurate to estimate the cost of a specific upcoming trial, which could help with budgeting and funding decisions.

Out-of-pocket costs vs capitalised costs & interest rates

Cost estimates also differ due to reporting of out-of-pocket and capitalised costs. An out-of-pocket cost refers to the amount of money spent or expensed on the R&D of a therapeutic. This can include all aspects of setting up therapeutic development, from initial funding in drug discovery/device design, to staff and site costs during clinical trials, and regulatory approval expenses.

The capitalised cost of a new therapeutic refers to the addition of out-of-pocket costs to a yearly interest rate applied to the financial investments funding the development of a new drug. This interest rate, referred to as the discount rate, is determined by (and is typically greater than) the cost of capital for the relevant industry.

Discount rates for the pharmaceutical industry vary between sources and can dramatically alter estimates for capitalised cost, where a higher discount rate will increase capitalised cost. Most studies place the private cost of capital for the pharmaceutical industry to be 8% or higher23,24, while the cost of capital for government is lower at around 3% to 7% for developed countries23,25. Other sources have suggested rates from as high as 13% to as low as zero13,23,26, where the zero cost of capital has been justified by the idea that pharmaceutical firms have no choice but to invest in R&D. However, the mathematical model used in many estimations for the cost of industry capital, the CAPM model, tends to give more conservative estimates23. This would mean the 10.5% discount rate widely used in capitalised cost estimates may in fact result in underestimation.

While there is not a consensus on what discount rate to use, capitalised costs do represent the risks undertaken by research firms and investors. A good approach may be to present both out-of-pocket and capitalised estimated costs, in addition to rates used, justification for the rate used, and the estimates using alternative rates in a sensitivity analysis26.

Costs variation over time

The increase in therapeutic development costs

Generally, there has been a significant increase in the estimated costs to develop a new therapeutic over time26. One study reported an exponential increase of capitalised costs from the 1970s to the mid-2010s, where the total capitalised costs rose annually 8.5% above general inflation from 1990 to 201310. Recent data has suggested that average development costs reached a peak in 2019 and had decreased the following two years9. This recent decrease in costs was associated with slightly reduced cycle times and an increased proportion of infectious disease research, likely in response to the rapid response needed for COVID-19.

Recent cost estimates

Costs can range with more than 100-fold differences for phase III/pivotal trials alone1. One of the more widely cited studies on drug costs used confidential survey data from ten multinational pharmaceutical firms and a random sample from a database of publicly available data10. In 2013, this study estimated the total pre-approval cost at $2.6 billion USD per approved new compound. This was a capitalised cost, and the addition of post-approval R&D costs increased this estimate to $2.87 billion (2013 USD). The out-of-pocket cost per approved new compound was reported at $1.395 billion, of which $965 million were clinical costs and the remaining $430 million were pre-clinical.

Another estimate reported the average cost to develop an asset at $1.296 billion in 20139. Furthermore, it reported that this cost had increased until 2019 at $2.431 billion and had since decreased to $2.376 billion in 2020 and $2.006 billion in 2021. While comparable to the previous out-of-pocket estimate for 2013, this study does not state whether their estimates are out-of-pocket or capitalised, making it difficult to meaningfully compare these estimates.

Figure 1: Recent cost estimates for drug development per approved new compound. “Clinical only” costs include only the costs of phase 0-III clinical trials, while “full” costs include pre-clinical costs. The colour of each bubble indicates the study, while bubble size indicated relative cost. A dashed border indicated the study used private data for their estimations, while a solid border indicates the study utilised publicly available data. Figure represents studies 9, 10, 12, 13 and 17 from the reference list in this report.

Publicly available data of 63 FDA-approved new biologics from 2009-2018 was used to estimate the capitalised (at 10.5%) R&D investment to bring a new drug to market at median of $985.3 million and a mean of $1.3359 billion (inflation adjusted to 2018 USD)12. These data were mostly accessible from smaller firms, smaller trials, first-in-class drugs, and further specific areas. The variation in estimated cost was, through sensitivity analysis, mostly explained by success/failure rates, preclinical expenditures, and cost of capital.

Publicly available data of 10 companies with no other drugs on the market in 2017 was used to estimate out-of-pocket costs for the development of a single cancer drug. This was reported at a median of $648 million and a mean of $719.8 million13. Capitalised costs were also reported using a 7% discount rate, with a median of $754.4 million and mean of $969.4 million. By focusing on data from companies without other drugs on the market, these estimates may better represent the development costs per new molecular entity (NME) for start-ups as the cost of failure of other drugs in the pipeline were included while any costs related to supporting existing on-market drugs were systematically impossible to include.

One study estimated the clinical costs per approved non-orphan drug at $291 million (out-of-pocket) and $412 million (capitalised 10.5%)17. The capitalised cost estimate increased to $489 million when only considering non-orphan NMEs. The difference between these estimates for clinical costs and the previously mentioned estimates for total development costs puts into perspective the amount

spent on pre-clinical trials and early drug development, with one studynoting their pre-clinical estimates comprised 32% of out-of-pocket and 42% of capitalised costs10.

Things to consider about cost estimates

The issue with these estimates is that there are so many differing factors affecting each study. This complicates cost-based pricing discussions, especially when R&D cost estimates can differ orders of magnitude apart. The methodologies used to calculate out-of-pocket costs differ between studies9,17, and the use of differing data sources (public data vs confidential surveys) seem to impact these estimates considerably.

There is also an issue with the transparency of data and methods from various sources in cost estimates. Some of this is a result of using confidential data, where some analyses are not available for public scrutiny8. This study in particular raised questions as estimates were stated without any information about the methodology or data used in the calculation of estimates. The use of confidential surveys of larger companies has also been criticised as the confidential data voluntarily submitted would have been submitted anonymously without independent verification12.

Due to the limited amount of comprehensive and published cost data17, many estimates have no option but to rely on using a limited data set and making some assumptions to arrive at a reasonable estimate. This includes a lack of transparent available data for randomised control trials, where one study reported that only 18% of FDA-approved drugs had publicly available cost data18. Therefore, there is a need for transparent and replicable data in this field to allow for more plausible cost estimates to be made, which in turn could be used to support budget planning and help trial sustainability18,26.

Despite the issues between studies, the findings within each study can be used to gather an idea of trends, cost drivers, and costs specific to company/drug types. For example, studies suggest an increasing overall cost of drug development from 1970 to peak in 201910, with a subsequent decrease in 2020 and 20219.

For a full list of references used in this article, please see the main report here: https://anatomisebiostats.com/biostatistics-blog/how-much-does-developing-a-novel-therapeutic-cost-factors-affecting-drug-development-costs-in-the-pharma-industry-a-mini-report/

How much does developing a novel therapeutic cost? – Factors Affecting Drug Development Costs across the Pharma Industry: A mini-Report

Introduction

Data evaluating the costs associated with developing novel therapeutics within the pharmaceutical industry can be used to identify trends over time and can inform more accurate budgeting for future research projects. However, the cost to develop a drug therapeutic is difficult to accurately evaluate, resulting in varying estimates ranging from hundreds of millions to billions of US dollars between studies. The high cost of drug development is not purely because of clinical trial expenses. Drug discovery, pre-clinical trials, and commercialisation also need to be factored into estimates of drug development costs.

There are limitations in trying to accurately assess these costs. The sheer number of factors that affect estimated and real costs means that studies often take a more specific approach. For example, costs will differ between large multinational companies with multiple candidates in their pipeline and start-ups/SMEs developing their first pharmaceutical. Due to the amount and quality of available data, many studies work mostly with data from larger multinational pharmaceutical companies with multiple molecules in the pipeline. When taken out of context, the “$2.6 billion USD cost for getting a single drug to market” can seem daunting for SMEs. It is very important to clarify what scale these cost estimates represent, but cost data from large pharma companies are still relevant for SMEs as they can used to infer costs for different scales of therapeutic development.

This mini-report includes what drives clinical trial costs, methods to reduce these costs, and then explores what can be learned from varying cost estimates.

What drives clinical trial costs?

There is an ongoing effort to streamline the clinical trial process to be more cost and time efficient. Several studies report on cost drivers of clinical trials, which should be considered when designing and budgeting a trial. Some of these drivers are described below:

Study size

Trial costs rise exponentially with an increasing study size, which some studies have found to be the single largest driver in trial costs1,2,3. There are several reasons for varying sample sizes between trials. For example, study size increases with trial phase progression as phases require different study sizes based on the number of patients needed to establish the safety and/or effectiveness of a treatment. Failure to recruit sufficient patients can result in trial delays which also increases costs4.

Trial site visits

A large study size is also correlated with a larger overall number of patient visits during a trial, which is associated with a significant increase in total trial costs2,3. Trial clinic visits are necessary for patient screening, treatment and treatment assessment but include significant costs for staff, site hosting, equipment, treatment, and in some cases reimbursement for patient travel costs. The number of trial site visits per patient varies between trials where more visits may indicate longer and/or more intense treatment sessions. One estimate for the number of trial visits per person was a median of 11 in a phase III trial, with $2 million added to estimated trial costs for every +1 to the median2.

Number & location of clinical trial sites

A higher number of clinical trial study sites has been associated with significant increase in total trial cost3. This is a result of increased site costs, as well as associated staffing and equipment costs. These will vary with the size of each site, where larger trials with more patients often use more sites or larger sites.

Due to the lower cost and shorter timelines of overseas clinical research5,6, there has been a shift to the globalisation of trials, with only 43% of study sites in US FDA-approved pivotal trials being in North America7. In fact, 71% of these trials had sites in lower cost regions where median regional costs were 49%-97% of site costs in North America. Most patients in these trials were either in North America (39.7%), Western Europe (21%), or Central Europe (20.4%).

Median cost per regional site as a percentage of North American median cost for comparison.

However, trials can face increased difficulties in managing and coordinating multiple sites across different regions, with concerns of adherence to the ethical and scientific regulations of the trial centre’s region5,6. Some studies have reported that multiregional trials are associated with a significant increase in total trial costs, especially those with sites in emerging markets3. It is unclear if this reported increase is a result of lower site efficiency, multiregional management costs, or outsourcing being more common among larger trials.

Clinical Trial duration

Longer trial duration has been associated with a significant increase in total trial costs3,4, where many studies have estimated the clinical period between 6-8 years8,9,10,11,12,13. Longer trials are sometimes necessary, such as in evaluating the safety and efficacy of long-term drug use in the management of chronic and degenerative disease. Otherwise, delays to starting up a trial contribute to longer trials, where delays can consume budget and diminish the relevance of research4. Such delays may occur as a result of site complications or poor patient accrual.

Another aspect to consider is that the longer it takes to get a therapeutic to market (as impacted by longer trials), the longer the wait is before a return of investment is seen by both the research organisation and investors. The period from development to on-market, often referred to as cycle time, can drive costs per therapeutic as interest based on the industry’s cost of capital can be applied to investments.

Therapeutic area under investigation

The cost to develop a therapeutic is also dependent on the therapeutic area, where some areas such as oncology and cardiovascular treatments are more cost intensive compared with others1,2,5,6,12,14. This is in part due to variation in treatment intensity, from low intensity treatments such as skin creams to high intensity treatments such as multiple infusions of high-cost anti-cancer drugs2. An estimate for the highest mean cost for pivotal trials per therapeutic area was $157.2M in cardiovascular trials compared to $45.4M in oncology, and a lowest of $20.8M in endocrine, metabolic, and respiratory disease trials1. This was compared to an overall median of $19M. Clinical

trial costs per therapeutic area also varied by clinical trial phase. For example, trials in pain and anaesthesia have been found to have the lowest average cost of a phase I study while having the highest average cost of a phase III study6.

It is important to note that some therapeutic areas will have far lower per patient costs when compared to others and are not always indicative of total trial costs. For example, infectious disease trials generally have larger sample sizes which will lead to relatively low per patient costs, whereas trials for rare disease treatment are often limited to smaller sample sizes with relatively high per patient costs. Despite this, trials for rare disease are estimated to have significantly lower drug to market costs.

Drug type being evaluated

As mentioned in the therapeutic areas section above, treatments may vary in intensity from skin creams to multiple rounds of treatment with several anti-cancer drugs. This can drive total trial costs due to additional manufacturing and the need for specially trained staff to administer treatments.

In the case of vaccine development, phase III/pivotal trials for vaccine efficacy can be very difficult to run unless there are ongoing epidemics for the targeted infectious disease. Therefore, some cost estimates of vaccine development include from the pre-clinical stages to the end of phase IIa, with the average cost for one approved vaccine estimated at $319-469 million USD in 201815.

Study design & trial control type used

Phase III trial costs vary based on the type of control group used in the trial1. Uncontrolled trials were the least expensive with an estimated mean of $13.5 million per trial. Placebo controlled trials had an estimated mean of $28.8 million, and trials with active drug comparators had an estimated mean cost of $48.9 million. This dramatic increase in costs is in part due to manufacturing and staffing to administer a placebo or active drug. In addition, drug-controlled trials require more patients compared to placebo-controlled, which also requires more patients than uncontrolled trials2.

Reducing therapeutic development costs

Development costs can be reduced through several approaches. Many articles recommend improvements to operational efficiency and accrual, as well as deploying standardised trial management metrics4. This could include streamlining trial administration, hiring experienced trial staff, and ensuring ample patient recruitment to reduce delays in starting and carrying out a study.

Another way to reduce development costs can take place in the thorough planning of clinical trial design by a biostatistician, whether in-house or external. Statistics consulting throughout a trial can help to determine suitable early stopping conditions and the most appropriate sample size. Sample size calculation is particularly important as underestimation undermines experimental results, whereas overestimation leads to unnecessary costs. Statisticians can also be useful during the pre-clinical stage to audit R&D data to select the best available candidates, ensure accurate R&D data analysis, and avoid pursuing unsuccessful compounds.

Other ways to reduce development costs include the use of personalised medicine, clinical trial digitisation, and the integration of AI. Clinical trial digitisation would lead to the streamlining of clinical trial administration and would also allow for the integration of artificial intelligence into clinical trials. There have been many promising applications for AI in clinical trials, including the use of electronic health records to enhance the enrolment and monitoring of patients, and the potential use of AI in trial diagnostics. More information about this topic can be found in our blog “Emerging use-cases for AI in clinical trials”.

For more information on the methodology by which pharmaceutical development and clinical trials costs are estimated and what data has been used please see the article: https://anatomisebiostats.com/biostatistics-blog/estimating-the-costs-associated-with-novel-pharmaceutical-development-methods-and-limitations/

Cost breakdown in more detail: How is a clinical trial budget spent?

Clinical trial costs can be broken down and divided into several categories, such as staff and non-staff costs. In a sample of phase III studies, personnel costs were found to be the single largest component of trial costs, consisting of 37% of the total, whereas outsourcing costs made up 22%, grants and contracting costs at 21%, and other expenses at 21%3.

From a CRO’s perspective, there are many factors that are considered in the cost of a pivotal trial quotation, including regulatory affairs, site costs, management costs, the cost of statistics and medical writing, and pass-through costs27. Another analysis of clinical trial cost factors determined clinical procedure costs made up 15-22% of the total budget, with administrative staff costs at 11-29%, site monitoring costs at 9-14%, site retention costs at 9-16%, and central laboratory costs at 4-12%5,6. In a study of multinational trials, 66% of total estimated trial costs were spent on regional tasks, of which 53.3% was used in trial sites and the remainder on other management7.

Therapeutic areas and shifting trends

Therapeutic area had previously been mentioned as a cost driver of trials due to differences in sample sizes and/or treatment intensity. It is however worth mentioning that, in 2013, the largest number of US industry-sponsored clinical trials were in oncology (2,560/6,199 active clinical trials with 215,176/1,148,340 patients enrolled)4,14. More recently, there has been a shift to infectious disease trials, in part due to the needed COVID-19 trials9.

Clinical trial phases

Due to the expanding sample size as a trial progresses, average costs per phase increase from phase I through III. Median costs per phase were estimated in 2016 at $3.4 million for phase I, $8.6 million for phase II, and $21.4 million for phase III3. Estimations of costs per patient were similarly most expensive in phase III at $42,000, followed by phase II at $40,000 and phase I at $38,50014. The combination of an increasing sample size and increasing per patient costs per phase leads to the drastic increase in phase costs with trial progression.

In addition, studies may have multiple phase III trials, meaning the median estimated cost of phase III trials per approved drug is higher than per trial costs ($48 million and $19 million respectively)2. Multiple phase III trials can be used to better support marketing approval or can be used for therapeutics which seek approval for combination/adjuvant therapy.

There are fewer cost data analyses available on phase 0 and phase IV on clinical trials. Others report that average Phase IV costs are equivalent to Phase III but much more variable5,6.

Orphan drugs

Drugs developed for the treatment of rare diseases are often referred to as orphan drugs. Orphan drugs have been estimated to have lower clinical costs per approved drug, where capitalised costs per non-orphan and orphan drugs were $412 million and $291 million respectively17. This is in part due to the limit to sample size imposed upon orphan drug trials by the rarity of the target disease and the higher success rate for each compound. However, orphan drug trials are often longer when compared to non-orphan drug trials, with an average study duration of 1417 days and 774 days respectively.

NMEs

New molecular entities (NMEs) are drugs which do not contain any previously approved active molecules. Both clinical and total costs of NMEs are estimated to be higher when compared to next in class drugs13,17. NMEs are thought to be more expensive to develop due to the increased amount of pre-clinical research to determine the activity of a new molecule and the increased intensity of clinical research to prove safety/efficacy and reach approval.

Conclusion & take-aways

There is no one answer to the cost of drug or device development, as it varies considerably by several cost drivers including study size, therapeutic area, and trial duration. Estimates of total drug development costs per approved new compound have ranged from $754 million12 to $2.6 billion10 USD over the past 10 years. These estimates do not only differ based on the data used, but also due to methodological differences between studies. The limited availability of comprehensive cost data for approved drugs also means that many studies rely on limited data sets and must make assumptions to arrive at a reasonable estimate.

There are still multiple practical ways that can be used to reduce study costs, including expert trial design planning by statisticians, implementation of biomarker-guided trials to reduce the risk of failure, AI integration and digitisation of trials, improving operational efficiency, improving accrual, and introducing standardised trial management metrics.

References

Moore T, Zhang H, Anderson G, Alexander G. Estimated Costs of Pivotal Trials for Novel Therapeutic Agents Approved by the US Food and Drug Administration, 2015-2016. JAMA Internal Medicine. 2018;178(11):1451-1457.

.1 Moore T, Zhang H, Anderson G, Alexander G. Estimated Costs of Pivotal Trials for Novel Therapeutic Agents Approved by the US Food and Drug Administration, 2015-2016. JAMA Internal Medicine. 2018;178(11):1451-1457.

2. Moore T, Heyward J, Anderson G, Alexander G. Variation in the estimated costs of pivotal clinical benefit trials supporting the US approval of new therapeutic agents, 2015–2017: a cross-sectional study. BMJ Open. 2020;10(6):e038863.

3. Martin L, Hutchens M, Hawkins C, Radnov A. How much do clinical trials cost?. Nature Reviews Drug Discovery. 2017;16(6):381-382.

4. Bentley C, Cressman S, van der Hoek K, Arts K, Dancey J, Peacock S. Conducting clinical trials—costs, impacts, and the value of clinical trials networks: A scoping review. Clinical Trials. 2019;16(2):183-193.

5. Sertkaya A, Birkenbach A, Berlind A, Eyraud J. Examination of Clinical Trial Costs and Barriers for Drug Development [Internet]. ASPE; 2014. Available from: https://aspe.hhs.gov/reports/examination-clinical-trial-costs-barriers-drug-development-0

6. Sertkaya A, Wong H, Jessup A, Beleche T. Key cost drivers of pharmaceutical clinical trials in the United States. Clinical Trials. 2016;13(2):117-126.

7. Qiao Y, Alexander G, Moore T. Globalization of clinical trials: Variation in estimated regional costs of pivotal trials, 2015–2016. Clinical Trials. 2019;16(3):329-333.

8. Monitor Deloitte. Early Value Assessment: Optimising the upside value potential of your asset [Internet]. Deloitte; 2020 p. 1-14. Available from: https://www2.deloitte.com/content/dam/Deloitte/be/Documents/life-sciences-health-care/Deloitte%20Belgium_Early%20Value%20Assessment.pdf

9. May E, Taylor K, Cruz M, Shah S, Miranda W. Nurturing growth: Measuring the return from pharmaceutical innovation 2021 [Internet]. Deloitte; 2022 p. 1-28. Available from: https://www2.deloitte.com/content/dam/Deloitte/uk/Documents/life-sciences-health-care/Measuring-the-return-of-pharmaceutical-innovation-2021-Deloitte.pdf

10. DiMasi J, Grabowski H, Hansen R. Innovation in the pharmaceutical industry: New estimates of R&D costs. Journal of Health Economics. 2016;47:20-33.

11. Farid S, Baron M, Stamatis C, Nie W, Coffman J. Benchmarking biopharmaceutical process development and manufacturing cost contributions to R&D. mAbs. 2020;12(1):e1754999.

12. Wouters O, McKee M, Luyten J. Estimated Research and Development Investment Needed to Bring a New Medicine to Market, 2009-2018. JAMA. 2020;323(9):844-853.

13. Prasad V, Mailankody S. Research and Development Spending to Bring a Single Cancer Drug to Market and Revenues After Approval. JAMA Internal Medicine. 2017;177(11):1569-1575.

14. Battelle Technology Partnership Practice. Biopharmaceutical Industry-Sponsored Clinical Trials: Impact on State Economies [Internet]. Pharmaceutical Research and Manufacturers of America; 2015. Available from: http://phrma-docs.phrma.org/sites/default/files/pdf/biopharmaceutical-industry-sponsored-clinical-trials-impact-on-state-economies.pdf

15. Gouglas D, Thanh Le T, Henderson K, Kaloudis A, Danielsen T, Hammersland N et al. Estimating the cost of vaccine development against epidemic infectious diseases: a cost minimisation study. The Lancet Global Health. 2018;6(12):e1386-e1396.
16. Hind D, Reeves B, Bathers S, Bray C, Corkhill A, Hayward C et al. Comparative costs and activity from a sample of UK clinical trials units. Trials. 2017;18(1).

17.Jayasundara K, Hollis A, Krahn M, Mamdani M, Hoch J, Grootendorst P. Estimating the clinical cost of drug development for orphan versus non-orphan drugs. Orphanet Journal of Rare Diseases. 2019;14(1).

19. Speich B, von Niederhäusern B, Schur N, Hemkens L, Fürst T, Bhatnagar N et al. Systematic review on costs and resource use of randomized clinical trials shows a lack of transparent and comprehensive data. Journal of Clinical Epidemiology. 2018;96:1-11.

20. Light D, Warburton R. Demythologizing the high costs of pharmaceutical research. BioSocieties. 2011;6(1):34-50.

21. Adams C, Brantner V. Estimating The Cost Of New Drug Development: Is It Really $802 Million?. Health Affairs. 2006;25(2):420-428.

22. Thomas D, Chancellor D, Micklus A, LaFever S, Hay M, Chaudhuri S et al. Clinical Development Success Rates and Contributing Factors 2011–2020 [Internet]. BIO|QLS Advisors|Informa UK; 2021. Available from: https://pharmaintelligence.informa.com/~/media/informa-shop-window/pharma/2021/files/reports/2021-clinical-development-success-rates-2011-2020-v17.pdf

23. Wong C, Siah K, Lo A. Estimation of clinical trial success rates and related parameters. Biostatistics. 2019;20(2):273-286.
24. Chit A, Chit A, Papadimitropoulos M, Krahn M, Parker J, Grootendorst P. The Opportunity Cost of Capital: Development of New Pharmaceuticals. INQUIRY: The Journal of Health Care Organization, Provision, and Financing. 2015;52:1-5.
25. Harrington, S.E. Cost of Capital for Pharmaceutical, Biotechnology, and Medical Device Firms. In Danzon, P.M. & Nicholson, S. (Eds.), The Oxford Handbook of the Economics of the Biopharmaceutical Industry, (pp. 75-99). New York: Oxford University Press. 2012.
26. Zhuang J, Liang Z, Lin T, De Guzman F. Theory and Practice in the Choice of Social Discount Rate for Cost-Benefit Analysis: A Survey [Internet]. Manila, Philippines: Asian Development Bank; 2007. Available from: https://www.adb.org/sites/default/files/publication/28360/wp094.pdf
27. Rennane S, Baker L, Mulcahy A. Estimating the Cost of Industry Investment in Drug Research and Development: A Review of Methods and Results. INQUIRY: The Journal of Health Care Organization, Provision, and Financing. 2021;58:1-11.
28. Ledesma P. How Much Does a Clinical Trial Cost? [Internet]. Sofpromed. 2020 [cited 26 June 2022]. Available from: https://www.sofpromed.com/how-much-does-a-clinical-trial-cost


The Role of Precision Medicine in Drug Development and Clinical Trials

With the help of precision medicine, or personalised medicine, modern medicine has moved away from a ‘one size fits all’ approach to treating disease and towards therapeutic approaches that are tailored to individuals and subgroups. These treatments are designed to be more efficacious due to targeting population subgroups based on their genetic or molecular nuances, rather than operating on the assumption that all bodies function and respond the same way and to the same degree to a given treatment. Molecular knowledge can now be utilised to tailor treatment to the patient at the correct dosage and time point, usually with the aid of pharmacogenomic approaches and molecular biomarkers.

Information about an individual’s genetic makeup, such as genetic variants that may influence treatment efficacy, toxicity, and adverse events can help to determine how patients will respond to a certain treatment. In addition to genomic, recent technological advances have led to the identification of many transcriptomic and proteomic biomarkers. This knowledge is useful in all stages of therapeutic development and can influence both the design of the therapeutic itself and of the clinical trial.


Drug Development

Inter-individual variations in drug response can result from polymorphisms in drug metabolizing enzymes. Thorough examination of gene expression and mutations in disease populations can lead to the identification of distinct disease subpopulations that share certain characteristics.  Further exploration of these genes and their interactions can uncover possible drug target genes for the treatment of a disease subpopulation.

Furthermore, an awareness of genetic variation in disease subpopulations means that the involved pathways and components can be more accurately recreated in pre-clinical studies. Bringing the gap between animal and human toxicity findings allows for more representative disease models. This allows variability in treatment response and optimal dosage to be explored more precisely.


Clinical Trial Design

Originally, clinical trials were designed to evaluate differences between novel treatments and standard treatments or controls, not among individual responses in treated groups. As a result, a therapeutic that was very effective in only a subgroup of the trial population may not have produced significant results and a therapeutic that caused adverse events in only a small subgroup could have been deemed too toxic for overall use.

The goal of clinical trials to gain regulatory approval remains unchanged. With the emergence of precision medicine come biomarker-driven trials that include patient subgroups in their design. Master protocols for trials enable the application of one treatment to multiple diseases, or multiple treatments to one disease, allowing a trial to adapt during its course. This room for adaptation can reduce financial impact due to ineffective treatments being abandoned earlier and targeting the most suitable groups. Incorporating a diagnostic assay in trial design can offer multiple advantages and prevent research from straying in the wrong direction.

Targeted therapies can be tested in the most appropriate patient groups likely to benefit by biomarker testing of patients prior to clinical trial participation. Screening patients for those more likely to respond well to treatment gives a greater estimate of treatment effect in the subgroup.  This increases the likelihood of demonstrating efficacy in a clinical trial. It also reduces the size of the sample population required to see statistically significant results, which can speed up the process.

Identifying responders before enrolment in such a manner minimises the number of exposed patients who would not benefit from treatment. Decreasing the risk of exposing non-responders to potential adverse events can improve the benefit/risk analysis.

Patient stratification is another aspect of trial design that utilises patient’s molecular biomarker profiles. Stratifying trial participants into subgroups can classify disease subtypes. Particularly in oncology, genomic approaches can guide the stratification of patients by their tumour mutations. It is notably useful in umbrella, basket, and platform trials and can reduce the financial impact by allowing for adaptive trials.

Umbrella trials test multiple targeted therapeutics in different biomarker cohorts of a single disease. Basket trials, on the other hand, test one or more targeted therapeutics in a patient cohort with matched biomarkers. Platform trials have a randomised structure and allow the evaluation of multiple targeted therapeutics in multiple biomarker-selected populations.


Application in developed therapeutics

While precision medicine approaches are most beneficial when included throughout the drug development process, their application can also improve or salvage existing treatments and prevent a clinical trial from failing. For example, a developed drug may cause severe adverse events in a small disease subpopulation.  Upon investigation it is found that the drug has a secondary target, which is only present in that subgroup.  With this knowledge, patients can be screened for presence/absence of the safety biomarker and intervention with said drug can be avoided in that subgroup while continued in the remaining population.

Alternatively, a drug may have clinically meaningful results in only a small number of patients. The responsive subgroup can be explored for potential biomarkers associated with degree of responsiveness to treatment. The clinical trial can then resume with a focus on patients likely to respond well to the therapeutic.


Response Monitoring

Throughout and after a clinical trial, biomarkers can be used as a means of observing patient response to intervention, and account for variability in response. Safety and efficacy monitoring markers will reveal individual cases where treatment is working effectively or needs to be halted due to adverse events. For example, a cancer-related gene mutation or protein detected in blood may no longer be present after successful treatment has been administered, showing that the treatment has worked.

Identified responders or non-responders can be further stratified into subgroups and studied.  Genomic information can aid in the understanding of outliers and changes to treatment response. This will contribute to disease and therapeutic understanding, so that the right patients can be given the right dose, getting the most benefit out of treatment.


Challenges of Precision Medicine

It should be noted that the development of a targeted therapy requires the right data, both for the identification of the drug target and suitable patients. Molecular data from disease populations in previous studies may not always be available during drug development. If available, it may not be the correct type of data or generated by the most appropriate assay. Developing a targeted therapy is not possible without suitable data to understand disease mechanisms and identify putative drug targets.

Biomarker-driven therapies require genetic tests and companion diagnostics to identify and distinguish suitable patients. Incorporating diagnostic methods in a clinical trial is an added cost and the process can be burdensome as it can make participant recruitment harder. Clinical intervention according to the results of stratification should also be well-defined before a trial phase commences.
 


References
Di Liello, R., Piccirillo, M., Arenare, L., Gargiulo, P., Schettino, C., Gravina, A. and Perrone, F., 2021. Master Protocols for Precision Medicine in Oncology: Overcoming Methodology of Randomized Clinical Trials. Life, 11(11), p.1253.
Dugger, S., Platt, A. and Goldstein, D., 2017. Drug development in the era of precision medicine. Nature Reviews Drug Discovery, 17(3), pp.183-196.
Mirsadeghi, S. and Larijani, B., 2017. Personalized Medicine: Pharmacogenomics and Drug Development. Acta Med Iran, 55(3), pp.150-165.
Woodcock, J., 2007. The Prospects for “Personalized Medicine” in Drug Development and Drug Therapy. Clinical Pharmacology & Therapeutics, 81(2), pp.164-169.
 

Clinical Trial Phases in Drug Development


The development of new drugs starts far before they are even seen in clinical trials. The discovery of multiple candidate drugs occur early on in the development process, often as a result of new information about how a disease functions, large-scale screening of small molecules, or the release of a new technology.

After a promising drug has been found, pre-clinical studies can be performed. A pre-clinical study for a new drug is used to determine important information about toxicity and suitable dosage amounts. These studies can be in vitro (in cell culture) and/or in vivo (in animal models) and determine whether a treatment will continue to the clinical trials stage.

Clinical trials test whether these experimental treatments are safe for use in humans, and whether they are more effective in treating or preventing a disease when compared to existing treatments. Clinical trials consist of several stages, called phases, where each phase is focused on answering a different clinical question: Progression of a treatment to the next phase requires the study to meet several parameters to ensure a treatment’s safety or efficacy.

  • Phase 0: Is the new treatment safe to use in humans in small doses?
  • Phase I: Is the new treatment safe to use in humans in therapeutic doses?
  • Phase II: Is the new treatment effective in humans?
  • Phase III: Is the new treatment more effective than existing treatments?
  • Phase IV: Does the new treatment remain safe and effective post-market?
Key phases of a pharmaceutical clinical trial

Phase 0: Small dose safety

Phase 0 studies can help to streamline the other clinical trial phases. Phase 0 consists of giving a few patients small, sub-therapeutic doses of the new treatment. This is to make sure that the new treatment behaves as expected by researchers and isn’t harmful to humans prior to using higher doses in phase I trials.

Phase I: Therapeutic dose safety

Phase I studies evaluate the safety of various doses of the new treatment in humans. This takes several months with typically around 20-80 healthy volunteers. In some cases, such as in anti-cancer drug trials, the study participants are patients with the targeted cancer type. A treatment may not pass phase I if the treatment leads to any serious adverse events.

Initial dosages in phase I studies can be informed based on data obtained during pre-clinical animal studies, and adjustments can be made to investigate the treatment’s side effect profile and develop an optimal dosing program. This could also include comparing different methods of giving a drug to patients (e.g., oral, intravenous etc.).

Phase II: Treatment efficacy

After passing phase I trials and having proven safety in humans, a new treatment advances to phase II studies designed to assess whether it may prevent or treat a disease. This phase can take between several months to 2 years, testing the new treatment in up to several hundred patients with the disease. Using a larger number of patients over a longer time period provides researchers with additional safety and effectiveness data, which is essential for the design of phase III trials.

To further test safety and efficacy, it is common to have a control group that receives either a placebo (a harmless pill or injection without the new treatment) or other current treatment (in trials where the disease is fatal unless treated e.g., cancer).

Phase III: Comparing to current treatments

Phase III studies are the last stage of a clinical trial before a new treatment can be approved for market use. The primary focus of a phase III study is to compare the safety and efficacy of a new treatment with current, existing treatments in patients with the target disease. Anywhere from several hundred to 3,000 patients may be included in a phase III study for between 1 to 4 years. Due to the scale of this phase, long-term or rare side effects are more likely to be uncovered.

Phase III studies are often randomised control trials, where patients will be randomly designated to different treatment groups. These groups may receive placebo, a current treatment (control group), the new treatment, or variations of the new treatment (e.g., different drug combinations). Randomised control trials are often double-blinded, where both the patient and the clinician administering their treatment do not know which treatment group they are assigned to.

A new treatment may continue to market and phase IV trials if the results prove it is as safe and effective as an existing treatment.

Phase IV: Post-market surveillance

If a new treatment passes phase III and is approved by the MHRA, FDA, or other national regulatory agency, it can be put to market. Phase IV is carried out in the post-market surveillance of the new treatment to keep updated on any emerging or long-term safety and efficacy concerns. This may include rare or long-term adverse side effects that were not yet discovered, or long-term analyses to see if the new treatment improves the life expectancy of a patient after recovery from disease.

Summary

Clinical trials are ultimately designed to mitigate risk. This includes the risk to the safety of trial participants by limiting the use of potentially unsafe treatments to small doses in a small number of patients before scaling up to testing therapeutic dose safety. Risk mitigation is not only for patient safety but also for preventing financial misspending as a treatment that is deemed unsafe in phase 0 would not proceed to the later, more costly clinical trial phases.

Not all clinical trials are the same, however, as each trial will have a different disease and treatment context. Trials for medical devices are somewhat different from pharmaceutical trials (for more information about the differences between medical device and pharma trials, click here). In addition, while sample sizes expand with phase progression, the required sample size for each trial and each phase is dependent on several factors including disease context (a rare disease may require lower sample sizes), patient availability (location of trial), trial budget and effect size. The sample size values mentioned earlier in this blog are purely indications of what each phase may use (for more information on how a biostatistician determines a suitable sample size, click here).

References

https://www.fda.gov/patients/drug-development-process/step-3-clinical-research

https://www.healthline.com/health/clinical-trial-phases

Medical Device Clinical Trials vs Pharmaceutical Clinical Trials – What’s the Difference?

Medical devices and drugs share the same goal – to safely improve the health of patients. Despite this, substantial differences can be observed between the two. Principally, drugs interact with biochemical pathways in human bodies while medical devices can encompass a wide range of different actions and reactions, for example, heat, radiation (Taylor and Iglesias, 2009). Additionally, medical devices encompass not only therapeutic devices but diagnostic devices, as well (Stauffer, 2020).

More specifically medical device categories can include therapeutic and surgical devices, patient monitoring, diagnostic and medical imaging devices, among others; making it a very heterogeneous area (Stauffer, 2020). As such, medical device research spills over into many different fields of healthcare services and manufacturing. This research is mostly undertaken by SME’s ( small to medium enterprises) instead of larger well-established companies as is more predominantly the case with pharmaceutical research. SME’s and start-ups undertake the majority of the early stage device development, particularly where any new class of medical device is concerned, whereas the larger firms get involved in later stages of the testing process (Taylor and Iglesias, 2009).

Classification criteria for medical devices

There are strict regulations that researchers and developers need to follow, which includes general device classification criteria. This classification criterion consists of three classes of medical devices, the higher class medical device the stricter regulatory controls are for the medical device. 

  • Class I, typically do not require premarket notifications
  • Class II,  require premarket notifications
  • Class III, require premarket approval

Food and Drug Administration (FDA)

Drug licensing and market access approval by the Food and Drug Administration (FDA) and international equivalents require manufacturers to undertake phase II and III randomised controlled trials in order to provide the regulator with evidence of their drug’s efficacy and safety (Taylor and Iglesias, 2009).

Key stages of medical device clinical trials

In general medical device clinical trials are smaller than drug trials and usually start with feasibility study. This provides a limited clinical evaluation of the device. Next a pivotal trial is conducted to demonstrate the device in question is safe and effective (Stauffer, 2020).

Overall the medical device trials can be considered to have three stages:

  • Feasibility study,
  • Pivotal study to determine if the device is safe and effective,
  • Post-market study to analyse the long-term effectiveness of the device.

Clinical evaluation for medical devices

Clinical evaluation is an ongoing process conducted throughout the life cycle of a medical device. It is first performed during the development of a medical device in order to identify data that need to be generated for regulatory purposes and will inform if a new device clinical investigation is necessary. It is then repeated periodically as new safety, clinical performance and/or effectiveness information about the medical device is obtained during its use.(International Medical Device Regulators Forum, 2019)

During the evaluative process, a distinction must be made between device types – diagnostic or therapeutic. The criteria for diagnostic technology evaluations are usually divided into four groups:

  • technical capacity
  • diagnostic accuracy
  • diagnostic and therapeutic impact
  • patient outcome

The importance of evaluation

Evaluations provide important information about a device and can indicate the possible risks and complications. The main measures of diagnostic performance are sensitivity and specificity. Based on the results of the clinical investigation the intervention may be approved for the market. When placing a medical device on the market, the manufacturer must have demonstrated through the use of appropriate conformity assessment procedures that the medical device complies with the Essential Principles of Safety and Performance of Medical Devices(International Medical Device Regulators Forum, 2019).The information on effectiveness can be observed by conducting experimental or observational studies.

Post-market surveillance

Manufacturers are expected to implement and maintain surveillance programs that routinely monitor the safety, clinical performance and/or effectiveness of the medical device as part of their Quality Management System (International Medical Device Regulators Forum, 2019). The scope and nature of such post market surveillance should be appropriate to the medical device and its intended use. Using data generated from such programs (e.g. safety reports, including adverse event reports; results from published literature, any further clinical investigations), a manufacturer should periodically review performance, safety and the benefit-risk assessment for the medical device through a clinical evaluation, and update the clinical evidence accordingly.

The use of databases in medical device clinical trials

The variations in the available evidence-base for devices means that, unlike with drugs, medical devices will typically require the consideration and analysis of data from observational studies in ascertaining their clinical and cost-effectiveness. Using modern observational databases has advantages because these databases represent continuous monitoring of the device in real-life practice, including the outcome (Maresova et al., 2020).

Bayesian methods as an alternative framework for evaluation

Bayesian methods for the analysis of trial data have been proposed as an alternative framework for evaluation within the FDA’s Center for Devices and Radiological Health. These methods provide flexibility and may make them particularly well suited to address many of the issues associated with the assessment of clinical and economic evidence on medical devices, for example, learning effects and lack of head-to-head comparisons between different devices.

Use of placebo in medical vs pharmaceutical trials

An additional key difference between drug and medical device trials are that use of placebo in medical device trials are rare. If placebo is used in a trial for surgical / implanted devices  it would usually be a sham surgery or implantation of a sham device (Taylor and Iglesias, 2009). Sham procedures are high risk and may be considered unethical. Without this kind of control, however, there is in many cases no sure way of knowing whether the device is providing real clinical benefit or if the benefit experienced is due to the placebo effect. 

Conclusion

            In conclusion, there are many similarities between medical device and pharmaceutical clinical trials, but there are also some really important differences that one should not miss:

  1.  In general medical device clinical trials are smaller than drug trials.
  2.  The research is mostly undertaken by SME’s ( small to medium enterprises) instead of big well-known companies
  3. Drugs interact with biochemical pathways in human bodies whereas medical devices use a wide range of different actions and reactions, for example, heat, radiation.
  4. Medical devices can be used for not only diagnostic purposes but therapeutical purposes as well.
  5.  The use of placebo in medical device trials are rare in comparison to pharmaceutical clinical trials.

References:

Bokai WANG, C., 2017. Comparisons of Superiority, Non-inferiority, and Equivalence Trials. [online] PubMed Central (PMC). Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5925592/> [Accessed 28 February 2022].

Chen, M., Ibrahim, J., Lam, P., Yu, A. and Zhang, Y., 2011. Bayesian Design of Noninferiority Trials for Medical Devices Using Historical Data. Biometrics, 67(3), pp.1163-1170.

E, L., 2008. Superiority, equivalence, and non-inferiority trials. [online] PubMed. Available at: <https://pubmed.ncbi.nlm.nih.gov/18537788/> [Accessed 28 February 2022].

Gubbiotti, S., 2008. Bayesian Methods for Sample Size Determination and their use in Clinical Trials. [online] Core.ac.uk. Available at: <https://core.ac.uk/download/pdf/74322247.pdf> [Accessed 28 February 2022].

U.S. Food and Drug Administration. 2010. Guidance for the Use of Bayesian Statistics in Medical Device Clinical. [online] Available at: <https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials> [Accessed 28 February 2022].

van Ravenzwaaij, D., Monden, R., Tendeiro, J. and Ioannidis, J., 2019. Bayes factors for superiority, non-inferiority, and equivalence designs. BMC Medical Research Methodology, 19(1).

Master Protocols for Clinical Trials

Part 1: Basket & Umbrella Trial Designs

Introduction

As the clinical research landscape becomes ever more complex and interdisciplinary alongside an evolving genomic and biomolecular understanding of disease, the statistical design component that underpins this research must adapt to accommodate this. Accuracy of evidence and speed with which novel therapeutics are brought to market remain hurdles to be surmounted.

While efficacy studies or non-inferiority clinical trials in the drug development space traditionally only included broad disease states usually with patients randomised to a dual arm of new treatment compared to an existing standard treatment. Due to patient biomarker heterogeneity, effective treatments could be left unsupported by evidence. Similarly treatments found effective in a clinical trial don’t always translate to show real world effectiveness in a broader range of patients.

Our current ability to assess individual genomic, proteomic and transcriptomic data and other patient bio-markers for disease, as well as immunologic and receptor site activity, has shown that different patients respond differently to the same treatment and, the same disease may benefit from different treatments in different patients – thus the beginnings of precision medicine.  In addition to this is the scenario where a single therapeutic may be effective against a number of different diseases or subclasses of a disease based on the agent’s mechanism of action on molecular processes common to the disease states under evaluation.

Master protocols, or complex innovative designs, are designed to pool resources to avoid redundancy and test multiple hypotheses under one clinical trial, rather than multiple clinical trials being carried out separately over a longer period of time.

Due to this fairly novel evolution in the clinical research paradigm and also due to inherent flexibility within each study design, conflicting information related to the definition and characterisation of master protocols such as basket and umbrella clinical trials as well as cases in the published literature where the terms “basket” and “umbrella” trials have been used interchangeably or are ill-defined exists. For this reason a brief definition and overview of basket and umbrella clinical trials is included in the paragraphs that follow. Based on systematic reviews of existing research it seeks the clarity of consensus, before detailing some key statistical and operational elements of each design.

Master protocols for bio-marker based clinical trials.
Diagram of a basket trial design.

Basket trial:

A basket clinical trial design consists of a targeted therapy, such as a drug or treatment device, that is being tested on multiple disease states characterised by a common molecular process that is impacted by the treatment’s mechanism of action. These disease states could also share a common genetic or proteomic alteration that researchers are looking to target.

Basket trials can be either exploratory or confirmatory and range from full randomised, controlled double-blinded designs to single arm designs, or anything in between. Single arm designs are an option when feasibility is limited and are more focused on the pre-clinical stage of determining efficacy or whether a particular treatment has clear-cut commercial potential evidenced by a sizable enough retreat in disease symptomology. Depending on the nuances of the patient populations being evaluated final study data may be analyses by pooling disease states or by each disease state separately. Basket trials allow drug development companies to target the lowest hanging fruit in terms of treatment efficacy, focusing resources on therapeutics with the highest potential of success in terms of real patient outcomes.

Master protocol umbrella trial
Diagram of an umbrella trial design.

Umbrella trial:

An umbrella clinical trial design consists of multiple targeted treatments of a single disease where patients can be sub-categorised into biomarker subgroups defined by molecular characteristics that may lend themselves to one treatment over another.

Umbrella trials can be randomised, controlled double-blind studies that in which each intervention and control pair is analysed independently of other treatments in the trial, or where feasibility issues dictate, they can be conducted without a control group with results analysed together in-order to compare the different treatments directly.

Umbrella trials may be useful when a treatment has shown efficacy in some patients and not others, they increase the potential for confirmatory trial success by honing in on patient sub-populations that are most likely to benefit due to biomarker characteristics, rather than grouping all patients together as a whole.

Basket & Umbrella trials compared:

Both basket and umbrella trials are typically biomarker guided. The difference being that basket trials aim to evaluate tissue-agnostic treatments to multiple diseases based on common molecular characteristics, whereas umbrella trials aim to evaluate nuanced treatment approaches to the same disease based on differing molecular characteristics between patients.

Biomarker guided trials have an additional feasibility constraint to non-biomarker guided trials in that the size of the eligible patient pool is reduced in proportion to the prevalence of the biomarker/s of interest within that patient pool. This is why master protocol methodology becomes instrumental in enabling these appropriately complex research questions to be pursued.

Statistical Concepts and considerations of basket and umbrella Trials

Effect size

Basket and umbrella trials generally require a larger effect size than traditional clinical trials, in order to achieve statistical significance. This is in a large part due to the smaller sample sizes and higher variance that comes with that. While patient heterogeneity in terms of genomic or molecular diversity, and thus expected treatment outcome, has been reduced by the precision targeting of the trial design, there is a certain degree of between-patient heterogeneity that can only be expected when relying on treatment arms of very small sample sizes.

If resources, including time, are tight then basket trials enable drug developers to focus on less risky treatments that are more likely to end in profitability. It should be noted that this does not always mean that the treatments that are rejected by basket trials are truly clinically ineffective. A single arm exploratory basket trial could end up rejecting a potential new treatment that, if subject to a standard trial with more drawn out patient acquisition and a larger sample size, would have been deemed effective at a narrower effect size.

Screening efficiency

If researchers carry out separate clinical studies for each biomarker of interest, then a separate screening sample needs to be recruited for each study. The rarer the biomarker, the larger the recruited screening sample would need to find enough people with the biomarker to participate in the study. This number needs to be multiplied by the number of biomarkers. A benefit of master protocols is that a single sample of people can be screened for multiple biomarkers at once, greatly reducing the required screening sample size.

 For example, researchers interested in 4 different biomarkers could collectively reduce the required screening sample by three quarters compared to conducting separate clinical studies for each biomarker. This maximisation of resources can be particularly helpful when dealing with rare biomarkers or diseases.

Patient allocation considerations

If relevant biomarkers are not mutually exclusive a patient could fit into multiple biomarker groups for which treatment is being assessed in the study. In this scenario a decision has to be made as to which category the patient will be assigned and the decision process may occur at random where appropriate. If belonging to two overlapping biomarker groups is problematic in terms of introducing bias in small sample sizes, or if several patients have the same overlap, then a decision may be made to collapse the two biomarkers into a single group or eliminate one of the groups. If a rare genetic mutation is a priority focus in the study then feasibility would dictate that the patient be assigned to this biomarker group.

Sample Size calculations

Generally speaking, sample size calculation for basket trials should be based on the overall cohort, whereas sample size calculations for umbrella trials are typically undertaken individually for each treatment.

Basket and umbrella trials can be useful in situations where a smaller sample size is more feasible due to specifics of the patient population under investigation. Statistically designing for this smaller sample size typically comes at the cost of necessitating a greater effect size (difference between treatment and control) and this translates to lower overall study power and greater chance of type 1 error (false negative result) when compared to a standard clinical trial design. Despite these limitations master protocols such as basket or umbrella trials allow to evaluation of certain treatments to the highest possible level of evidence that otherwise might be too heterogeneous or rare to evaluate using a traditional phase II or III trial.

Randomisation and control

Randomised controlled designs are recommended for confirmatory analysis of an established treatment or target of interest. The control group typically treats patients with the established standard of care for their particular disease or, in the absence of one, placebo.

In master basket trials the established standard of care is likely to differ by disease or disease sub-type. For this reason it may be necessary for randomised controlled basket trials pair a control group with each disease sub-group rather than just incorporating a single overall control group and potentially pooling results from all diseases under one statistical analysis of treatment success. Instead it is worth considering if each disease type and corresponding control pair could be analysed separately to enhance statistical robustness in a truly randomised controlled methodology.

Single arm (non-randomised designs) are sometimes necessary for exploratory analysis of potential treatments or targets. These designs often require a greater margin of success (treatment efficacy) to be statistically significant as a trade-off for a smaller sample size required.

Blinding

To increase the quality of evidence, all clinical studies should be double blinded where possible.

To truly evaluate the effectiveness of a treatment without undue bias from a statistical perspective double-blinding is recommended.

Aside from increased risk of type 2 error that may be inherent in master protocol designs, there is a greater potential for statistical bias to be introduced. Bias can introduce itself in a myriad of ways and results in a reduction in the quality of evidence that a study can produce. Two key sources of bias are lack of randomisation (mentioned above) and lack of blinding.

Single armed trials do not include a control arm and therefore patients cannot be randomised to a treatment arm where double-blinding of patients, practitioners, researchers and data managers etc will prevent various types of bias creeping in to influence the study outcomes. With so many factors at play it is important not to overlook the importance of study blinding and implement it whenever feasible to do so.

If the priority is getting a new treatment or product to market fast to benefit patients and potentially save lives, accommodating this bias can be a necessary trade-off. It is after-all typically quite a challenge to have clinical data and patient populations that are at homogeneous and matched to any great degree, and this reality is especially noticeable with rare diseases or rare biomarkers.

Biomarker Assay methodology

The reliability of biologic variables included in a clinical trial should be assessed, for example the established sensitivity and specificity of particular assays needs to be taken into account. When considering patient allocation by biomarker group, the degree of potential inaccuracy of this allocation can have a significant impact on trial results, particularly when there is a small sample size. If the false positive rate of a biomarker assay is too high this will result in the wrong patients qualifying for treatment arms, in some cases this may reduce the statistical power of the study.

A further consideration of assay methodology pertains to the potential for non-uniform bio-specimen quality at different collection sites which may bias study results. A monitoring framework should be considered in order to mitigate this.

Patient tissue samples required for assays, can inhibit feasibility and increase time and cost in the short term and make study reproducibility more complicated. While this is important to note these techniques are in many cases necessary in effectively assessing treatments based on our contemporary understanding a many disease states such as cancer within the modern oncology paradigm. Without incorporating this level of complexity and personalisation into clinical research it will not be possible to develop evidence based treatments that translate into real-world effectiveness and thus widespread positive outcomes for patients.

Data management and statistical analysis

The ability to statistically analyse multiple research hypotheses at once within a single dataset increases efficiency at the biostatisticians end and allows frameworks for greater reproducibility of the methodology and final results, compared to the execution and analysis of multiple separate clinical trials testing the same hypotheses. Master protocols also enable increased data sharing and collaboration between sites and stakeholders.

Deloitte research estimated that master protocols can save clinical trials 12-15% in cost and 13-18% in study duration. These savings of course apply to situations where master protocols were a good fit for the clinical research context, rather than to the blanket application of these study designs across any or all clinical studies. Applying a master protocol study design to the wrong clinical study could actually end up increasing required resources and costs without benefit, therefore it is important to assess whether a master protocol study design is indeed the optimal approach for the goals of a particular clinical study or studies.

umbrella trials for precision medicine
Master protocols for precision medicine.

References:

Bitterman DS, Cagney DN, Singer LL, Nguyen PL, Catalano PJ, Mak RH. Master Protocol Trial Design for Efficient and Rational Evaluation of Novel Therapeutic Oncology Devices. J Natl Cancer Inst. 2020 Mar 1;112(3):229-237. doi: 10.1093/jnci/djz167. PMID: 31504680; PMCID: PMC7073911.

Lesser N, Na B, Master protocols: Shifting the drug development paradigm, Deloitte Center for Health solutions

Lai TL, Sklar M, Thomas, N, Novel clinical trial solutions and statistical methods in the era of precision medicine, Technical Report No. 2020-06, June 2020

Renfro LA, Sargent DJ. Statistical controversies in clinical research: basket trials, umbrella trials, and other master protocols: a review and examples. Ann Oncol. 2017 Jan 1;28(1):34-43. doi: 10.1093/annonc/mdw413. PMID: 28177494; PMCID: PMC5834138.

Park, J.J.H., Siden, E., Zoratti, M.J. et al. Systematic review of basket trials, umbrella trials, and platform trials: a landscape analysis of master protocols. Trials 20, 572 (2019). https://doi.org/10.1186/s13063-019-3664-1