Mini Report: Why do clinical trials fail? 

Overview

Clinical studies are time-consuming, expensive, and frequently pose challenges for participants and sponsors alike. This article explores some of the numerous factors contributing the failure of a clinical study and offers suggestions on how to increase the likelihood of designing and carrying out effective clinical trials.

Pharmaceutical and medical device clinical trials present several chances for failure. Failures can occur due to lack of treatment effectiveness, problems with safety, or a failure to demonstrate either of these through use of the appropriate study design, as well as though budgetary constraint [1]. Other factors include not adhering to MHRA or FDA guidelines, failure to maintain acceptable study protocol, or issues with patient recruiting, enrolment, and retention. [2] In order to decide whether or not a clinical trial should continue, it is crucial to produce accurate and sufficient results and insights from data.

Briefly, some common reasons for failure at any phase include:

  1. The treatment is not effective: The treatment may not work as well as expected, or may not work for a significant portion of the study population.
  2. The treatment has unacceptable side effects: The treatment may have serious or unacceptable side effects that outweigh its potential benefits.
  3. The treatment is not better than existing treatments: The treatment may not be significantly better than existing treatments, or may not offer a clear advantage in terms of efficacy or convenience. If the study design is a non-inferiority or equivalence design this would not be a concern, thus it is important to choose the optimal study design for a specific context.
  4. The study design is flawed: The study may be poorly designed or executed, leading to inaccurate or misleading results.
  5. The study population is not representative: The study population may not accurately reflect the population for which the treatment is intended, leading to results that do not generalize to the wider population or to the study population of the subsequent phase of the clinical trial.

Overall, the high failure rate of clinical trials is a reflection of the complex and uncertain nature of medical research and the difficulty of developing new treatments that are both effective and safe.

One of the most expensive stages for a therapeutic clinical trial to fail in at the Phase III or phase IV stage. This is because so much investment of time, money and resources has already taken place across the previous clinical trial phases, as well as at the R&D and pre-clinical stages Unfortunately one of the most common phases for a clinical trial to fail is at phase III.

A 2016 review found that  46% of orphan-designated drugs were approved by regulatory agencies compared to 34% of non-orphan drugs (OR=2.3; 95% CI: 1.4-3.7); 27% of oncological agents were approved compared to 39% of agents targeted at other diseases (non-oncological) (OR=0.5; 95% CI: 0.3-0.9) and; 28% of agents sponsored by small and medium-size companies were approved vs 42% for those sponsored by larger firms (OR=0.4; 95% CI: 0.3-0.7). [5]

Earlier reviews of clinical trial results have found that on average there is a trend towards increased response rates in phase II trials than in the subsequent phase III trials of the same therapeutics. (zia et al, Clinical oncology 2005>. Further to this, 2019 study found a ratio of the hazard ratios for overall survival (OS) and progression free survival (PFS) of phase II compared to phase III trials of .074 indicating a significant reduction in observed hazard ratios in both the positive or negative direction (away from 1) for the phase III trials.[13]

Phase III clinical trials are the final stage of testing before a potential new treatment is submitted for approval to regulatory authorities. These trials are conducted on large groups of people, usually several hundred to several thousand, to confirm the effectiveness of the treatment, monitor side effects, and compare it to existing treatments. Despite careful planning and design, phase III clinical trials can fail for a variety of reasons.

Factors associated with patient eligibility: exclusion and inclusion criteria

The inclusion and exclusion criteria of a clinical study should ideally result in a population that proportionally matches the general patient population who experiences the disease, or would benefit from the therapeutic under investigation, to the extent that that is practical from a recruitment and safety stand-point.[4] Inclusion criteria may vary across studies or across sites of a single study and this has the potential impact the clarity of results. Having inclusion criteria that is too specific or narrow can also reduce the pool of eligible participants which typically translates to longer recruitment time horizons. On study found that 16% of protocol amendments are due to changes in inclusion or exclusion criteria,[5] this can lead to differences in the patient populations before and after the amendments.

Having a long list of exclusion criteria for a clinical trial can becomes problematic when it limits the resulting sample of patients too severely. It can artificially reduce the variance to the extent that the results of the trial are not generalisable. This creates problems at later trial phases and may have a negative impact on the chances of gaining regulatory approval. It could prevent the sponsor from acquiring timely knowledge regarding patient populations that are likely to be more reflective of real world use cases for the therapeutic under investigation.[1] In some cases this act of limitation in phase II of clinical development in particular, has been justified as a means of reducing variability;[2] and as a counter-point there is little published evidence to state universally increasing the diversity of the patient population with other criteria will inevitably increase the variability of the primary endpoint. This having narrower criteria could conceivably be justified in some contexts if carefully considered.[3]

Patient factors related to recruitment and retention

A failure to enrol a sufficient number of patients is a long standing problem with a UK study of 114 trials 10 indicated that only 31% met enrolment goals.[1] Bottlenecks or other problems with patient recruitment can hamper the success of clinical trials. If too many companies are using the same preferred trial site this can dilute to potential subject pool, and where the target disease is rare this can pose particular problems with recruiting.

Logistical constraints may also occur where patients who are ill cannot easily travel to designated hospitals. Some companies are trying to address this problem by potentially bringing the trial to people’s homes, however this could present further issues.[2] Some studies offer certain remunerations to patients to cover expenses in the hope that recruitment could be improved, however evidence suggests that paying patients to participate does not tend to generate better recruitment as it does scant to overcome the logistical challenges.[3] Although financial incentives did not result in better recruitment, it was reported that financial incentives did increase participant’s response to questionnaires for the trial.[4]

Biological effects of the drug do not translate across species or across patient populations

One review of clinical trial failure found that out of 640 phase III trials of novel therapeutics, 57% failed due to inability to demonstrate treatment efficacy[7]

Failure at phase I may result due to biological unsuitability of the therapeutic in humans, evidenced by the fact that successful animal studies do not translate into comparable effects in humans largely due do inter-species biomolecular differences. For example there may be unforeseen adverse events which make the therapeutic unsafe to consume even at an active dose. Failure at a later phase due to biological unsuitability may occur when the therapeutic does demonstrate a clinical effect against the disease but this effect does not translate into increased overall survival of the disease.

Toxicity or adverse events in the diseased population could be one explanation for this scenario. As phase I trials are typically conducted on healthy volunteers and phase II trials on less ill patients than Phase II trials, these patients may be more able to tolerate the therapeutic at a particular dose than those patients weakened by disease. Alternatively, this difference in study population could mean that the determined dose may be biologically active yet sub-optimal to combat the disease.

Changes in patient population between trial phases

If a successful phase II trial is conducted and supported by strong evidence and then the patient population is changed in some way for the phase III trial this may be another avenue to failure. This is a common occurrence as typically less ill patients will be chosen for a phase II study in determining therapeutic dose than in a phase III study, both for safety reasons and due to the uncertainty of whether the treatment will be shown to be sufficiently effective.

Accelerated Approvals

Certain therapeutics may receive accelerated approval to enable early patient access before phase III and IV trials have been successfully completed. It should be noted that these agents still need to complete the full clinical trials process in order to remain on the market. Therapeutics receiving accelerated approvals by the FDA at phase II, have unfortunately been found to have increased failure rates at subsequent phases. A 2018 study by Beaver et al, showed that 72% of the clinical studies receiving accelerated approval were single arm studies and almost 90% of studies receiving accelerated approval had response rate as the primary endpoint. [14] This points to an issue with study design whereby patient outcomes are being too easily over-estimated. The fact that 66% of accelerated approvals were granted to oncology therapeutics may go some way to explain these choices. However, given these figures and the obvious methodological problems manifest with accelerated approvals it isn’t difficult to explain the reduced approval rate observed for oncological agents compared to agents for other diseases (mentioned in a previous section of this article). In fact, single arm studies are not an advisable study design unless due to recruitment or other constraints there is absolutely no other alternative. Where possible a two armed randomised design should be prioritised.

Hacking bias

Phase II trials in which the analysis has relaxed the alpha cut-of from a p value of 0.05 to 0.01, thus deeming a trial successful (statistically significant) based on essentially insufficient evidence, can also lead to failure in the subsequent phase. Hacking bias can also occur if sub-group analyses were conducted posthumously to salvage the results of an unpromising trial result. This practice can be legitimate but if similar sub-groups are not considered at later phase trials then any positive findings may not be replicated. This is a good example of where it might be safer to cut losses at an earlier phase rather than forcing a result only to invest more time and budget chasing what will ultimately be considered an ineffective treatment. [16]

Focusing on the wrong endpoints is another mishap which could come under the umbrella of hacking bias or equally be reflective of poor study design. A common example of this is where response rate (RR) is chosen as a primary endpoint for a phase II trial in situations where response rate does not translate well to overall survival (OS) or even progression free survival (PFS), which may be the more informative outcome. This choice could lead to a therapeutic being deemed effective at phase II on the basis of RR then falling short in subsequent phases.

Optimism bias and regression to the mean

Single arm studies and patient differences aside, there are scenarios where a randomised phase II trial with adequate sample size and no real changes to the patient population in phase III is still not able to reproduce positive results. This can be explained on a statistical basis by optimism bias and regression to the mean. [16]

Clinical studies only progress to the next phase of a clinical trial if the previous phase study was found to be successful. This translates to statistically significant for superiority designs or non-significant in the case of non-inferiority and equivalence designs. For example, Phase III studies only result from successful phase II studies. Phase II studies that were not statistically significant, in other words where the results were neutral or negative, are ignored in the decision to progress to the next phase. This means that false positives are potentially forming the evidence base for progression to the net phase leading to a biased expectation of success at the next phase. The result of this is overly-optimistic expectations related to variability or treatment delta and effect size that may lead to underestimation of the sample size required for the subsequent trial and generally inadequate study design to fairly evaluate the therapeutic at the next phase. It may also lead to an ineffective treatment being pursued when it shouldn’t be.

There may be many reasons for a study to fail before being repeated successfully, such as due to an adjustment in patient sample or other methodological changes. Regardless of whether these reasons seem justified it is important not to abandon the unsuccessful results in calculating expectations for the study design of the subsequent phase. If an average was taken of the failures and successes at a particular phase, a more conservative and potentially less optimistic expectation of future therapeutic performance and outcomes at the subsequent phase would result – the by-product of this being a better designed next-phase trial.

Exaggeration Bias

A 2021 study based on the results of nearly 24,000 clinical trials observed a tendency towards the observed treatment effects of the clinical trial being larger than the true treatment effects and puts forward the idea of an exaggeration ratio. As a result of regression to the mean the ratio of the observed treatment effect to the true treatment effect tends to demonstrate a higher observed treatment effect than the true effect on average. This exaggerated expectation means that when clinical studies go on to the next phase it can be harder to replicate the results. At the 0.05 alpha level the paper estimated demonstrated a 25% chance that the observed effect is >3x larger than the true effect of the treatment; 50% chance that the observed effect is >1.5x larger than the true effect; and a 25% chance that the observed effect more or less approximated the true value. While based on a large but limited subsection of all clinical trials, the authors of this paper suggest taking this exaggeration ratio into account and adjusting for it in any subsequent clinical study design such as in the next phase of a study. [15]

Study design

A poor study design can lead to trial failures, for instance selecting the wrong patients or the wrong endpoint, not to mention bad data, can lead to problems in the trial.[1] However data sources can help sponsors be sure that the right patients are then recruited as well as choosing the proper and correct sites and countries to enhance the likely hood of success.

Another common cause of failure in clinical research is based on not being able to meet criteria that have been predetermined by the MHRA or FDA. As well as this, it is important to recognise that a sponsor is necessary to move a drug or device forward in the clinical trial process. If studies are rushed into phase III after a successful phase II it could lack time for reflection on how to address safety in phase III. [2]

Data-related biases

Problems in data collection, missing data, attrition bias may that are not sufficiently accounted for may also lead to unexpected failure, as may rater-bias and unintentional un-blinding. Factors such as non-compliance to the treatment protocol, whether for site-based logistic reasons or for reasons resulting from the individual disease state, in certain patient populations can often be a factor influence results.

Financial concerns

One review found that 22% of phase III studies failed due to a lack of funding[3]. This financial burden also leads to ethical issues regarding the patients that are involved in the trial, patients are under the impression that their involvement would lead to the advancement of the trial and its successful completion.[4] Therefore underfunded trials are likely to lack the enrolment needed to demonstrate efficacy.

Financial risks occur at all stages of product development, however the costs associated with having to re-do studies or to delay studies will escalate the cost further. Taking steps to identify and address risks early on in the development process is key. Companies that do not carefully monitor for risks sometimes don’t identify problems until much further down the line when it is difficult to address them cost-effectively.[5] Sometimes this comes from the hesitation of companies to terminate a project prematurely. In a study of 842 molecules and 637 development program failures, it was evident that the companies that took time to identify problems early on and stop development on an imperilled trial, had a much higher likelihood of reaching the market with their drug.[6]

Other factors that can result in trial failure include; misspent funding, lack of a correct design study, insufficient funding designated to the trial from the outset, which implies costing may not have been accurately calculated.[7] Patient dropout rates also effect the financial stability of clinical trials and difficulties with treatment adherence such as side effects, or a lack of follow ups will also contribute to the financial impact.

Delays and unforeseen costs & challenges

With patient recruiting, there are additional expenses that might be challenging to predict and become very changeable. It is clear that marketing tactics like advertising can have a significant impact on a trial’s capacity to make a profit. [5] Additionally, healthcare professionals can have a big impact on patient recruitment. For example, recruitment and retention may suffer if staff members are absent or appear to be absent, or if there is a regular turnover of staff members and no rapport can form between them and the patients. Building this trust and communication may result in increased participation.

The experiment is impacted by all of these patient recruiting issues, some of which result in significant delays. Only 6% of clinical trials are finished before the deadline, and another 80% are at least a month behind schedule. [6] These delays increase the possibility for loss and have an impact on study costs as well as subsequent sales. Since costs are considerable, there is a risk of financial loss; consequently, increasing the rate of recruitment and retention will yield enormous benefits. [7]

Ethical issues increase the risk of trial failure, severely damaging the reputation of all parties involved, i.e. the pharmaceutical or medical-device company, the CRO and the associated physicians.[1] Many industry cases illustrate that a pursuit of short-term gains can rapidly turn into long-term losses.[2] The general problems with the ethics of clinical trials come from the fact that participants bear the risk and burden. Participation in a clinical trial has an increased level of risk; this is because of the exposure to effects of new treatment. These risks however are not “offset by a prospective clinical benefit”[3], this is because the goal of the trial is not to treat trial participants but to produce generalised medical knowledge.

Take-aways from a statistical perspective

  • Avoid single arm studies where possible or at least consider the possible trade-offs.
  • Be careful about switching to more promising endpoints, alpha levels or sub-groups as a short-term tactic to get a study over the line.
  • Be cognisant that changing the study population from one phase to the next may alter the study outcomes, and not always in the desired direction.
  • Consider “Exaggeration Bias” and adjust for it when designing the next phase of your clinical trial.

References:

[1] Saberwal, Gayatri. “Biobusiness in Brief: What Ails Clinical Trials?” Current Science, vol. 115, no. 9, Current Science Association, 2018, pp. 1648–52, https://www.jstor.org/stable/26978474.

[2] Pharmafile “clinical trials and their patient”  https://www.pharmafile.com/news/511225/clinical-trials-and-their-patients-rising-costs-and-how-stem-loss (2016)

[3] Chris Plaford “why do most clinical trials fail” https://www.clinicalleader.com/doc/why-do-most-clinical-trials-fail-0001#_ftn1 (2015)

 [4]  Hwang T.J., Carpenter D., Lauffenburger J.C., Wang B., Franklin J.M., Kesselheim A.S. Failure of investigational drugs in late-stage clinical development and publication of trial results. JAMA Intern. Med. 2016;176:1826–1833.

 [5] Tukey, John W. “Use of Many Covariates in Clinical Trials.” International Statistical Review / Revue Internationale de Statistique, vol. 59, no. 2, [Wiley, International Statistical Institute (ISI)], 1991, pp. 123–37, https://www.jstor.org/stable/1403439?origin=crossref.

[6] Worrall, John. “<em>What</Em> Evidence in Evidence‐Based Medicine?” Philosophy of Science, vol. 69, no. S3, [The University of Chicago Press, Philosophy of Science Association], 2002, pp. S316–30, https://doi.org/10.1086/341855.

[7] Altman, Douglas G. “Size Of Clinical Trials.” British Medical Journal (Clinical Research Edition), vol. 286, no. 6381, BMJ, 1983, pp. 1842–43, https://www.jstor.org/stable/29511193.

[8] Fogel DB. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: A review. Contemp Clin Trials Commun. 2018;11:156-164. Published 2018 Aug 7. doi:10.1016/j.conctc.2018.08.001

[9] Jansen, Lynn A. “The Problem with Optimism in Clinical Trials.” IRB: Ethics & Human Research, vol. 28, no. 4, Hastings Center, 2006, pp. 13–19, https://www.jstor.org/stable/30033204.

[10] Chris Plaford “why do most clinical trials fail” https://www.clinicalleader.com/doc/why-do-most-clinical-trials-fail-0001#_ftn1 (2015)

[11] Kobak Kenneth “why do clinical trials fail? Journal of Clinical Psychopharmacology: February 2007 – Volume 27 – Issue 1 – p 1-5 doi: 10.1097/JCP.0b013e31802eb4b7

[12] Liang et al. European Journal of Cancer 2019; 121:19-28

[13] Tap et al, JAMA 2020; 323(13):1266-1276

[14] Beaver et al. JAMA Oncology 2018; 4:849-856

[15] van Zwet et al, Significance, December 2021; 16

[16] Michiels & Wason, European Journal of Cancer 2019; 123:116

Emerging use-cases for AI in clinical trials

Overview

Clinical trials are becoming more expensive where, according to a Deloitte’s report1, the average cost to get a drug to market in the USA was $1.188 billion in 2010, and $1.981 billion in 2019. This increase in cost reflects the difficulties that are associated with current linear clinical trial designs. Clinical trials can take a long time due to difficulty in finding suitable/eligible patients for each study and the growing amount of data available used to plan or inform a trial. Current clinical trials are still evolving to make use of the rapidly developing technologies, scientific methods, and data availability of recent years.

One possible way to improve and transform the clinical trial process as we know it would be through the implementation of Artificial Intelligence (AI). AI incorporates all intelligence demonstrated by machines and includes important aspects such as Machine Learning and Natural Language Processing. It is already widely used in modern technology, such as in smartphones and online website searches, but has also been used more recently to innovate the drug discovery process. AI implementation could be of benefit across diverse tasks is the planning, execution and analysis stages of clinical trials to improve cost effectiveness, study time, treatment efficacy, and quality of data2.

Many emerging developments in artificial intelligence (AI) have the potential to benefit the clinical trials landscape.

AI in Adaptive clinical trial design

Many clinical trials are designed linearly, however adaptive designs are being used to allow predetermined changes during a study in response to ongoing trial data. Adaptive designs provide the flexibility to optimise resource allocation, end an unproductive trial early, and better characterise a treatment’s efficacy and safety through multiple endpoints. AI can help to inform and optimise adaptive designs through the analysis of healthcare data to select optimal endpoints, determine and monitor parameters for early stopping, and identify appropriate protocols for the trial3. These study design changes can increase the efficiency of a study, resulting in a trial which is more cost and time effective while maintaining high quality data collection and analysis.

Meta-analysis

AI can also enhance the exploratory analysis of data from previous trials. AI-enabled technology can collect, organise, and analyse increasing amounts of data, which could be applied to data from previous trials. This is normally performed manually by a biostatistician as part of a meta-analysis, but AI could help to gather and perform initial analyses before a more in-depth statistical approach is taken. This could highlight potentially important patterns in collected evidence which would then be used in informing trial design.

Synthetic control arms

Current clinical trials typically compare an experimental treatment to placebo and established treatments, assigning enrolled patients to either a treatment or control group. Synthetic control arms are an AI-driven solution for having a control group in a single-arm trial, which usually have only a treatment group. Synthetic control arms use data from previous studies to simulate the control treatment in patients. This would allow for all patients in a trial to receive active treatment to provide more evidence for the treatment efficacy and safety at each clinical trial stage. This may also have an impact on patient enrolment as patients generally show less interest in enrolling on placebo-controlled trials4,5.

As synthetic control arms are relatively novel, comparisons should be made with traditional control groups. In a typical blinded trial, patients are unaware if they are receiving the experimental active treatment, or a placebo. This is to test that the new treatment is causative of any clinically meaningful response. Synthetic control arms could create more single-arm clinical trials, and the fact that patients are aware of receiving active treatment would be a factor given clinically meaningful results are found.

Site selection

Identification of suitable sites and investigators to perform a trial is an important factor in study efficiency and feasibility. A study site should be amply equipped to carry out a study, must be of a suitable size to process the needs of study participants, and be located in an accessible area to potential participants and investigators. Identification of target locations and investigators can be optimised through AI implementation, which also enables real-time monitoring of site performance once the trial has started. Study sites can be evaluated and compared through the development of a points-based algorithm which could factor in location, site size, and equipment.

Patient enrolment/recruitment

Most patients enrol on a clinical trial if they have not responded to existing treatments in a clinically meaningful way. However, there are often strict eligibility criteria required for patients to enrol for a trial, including diagnostic tests, biomarker profiles, and demographics. Currently, patients find out about clinical trials either through manually searching online databases or, on occasion, through a clinician’s recommendation. This puts a lot of responsibility on patients to search for potential trials, just to be faced with trying to understand eligibility criteria full of medical jargon. This can lead to low patient recruitment.

AI and natural language processing offer a potential solution for this issue. Natural language processing could be used to match patients with trials based on eligibility criteria and patient electronic health records. Potentially suitable trials could then be suggested to patients or their clinicians, making it easier for patients to find suitable trials and for trials to recruit patients. While this is an improvement to the current recruitment method, natural language processing may initially have some difficulty with clinical notes due to heavy use of acronyms, medical jargon, and deciphering of hand-written clinical notes. These problems aren’t specific to AI though, as currently patients looking for suitable trials have the same difficulties.

AI-driven patient recruitment could also be used to reduce population heterogeneity and use prognostic and predictive enrichment to increase study power. While reduced heterogeneity can be beneficial (e.g. in testing the safety of drugs for patients unable to enrol on early clinical trials), caution should be taken with limiting patient diversity. Selectively enrolling patients who are more likely to respond well to treatment initially could lead to advancing a treatment that may not be as wel tolerated in the post-market patient population.

AI balancing of diversity in clinical trials could be used to balance this. It is important to test safety and efficacy in with differing demographics for a treatment to be properly characterised. This may include different ethnic groups, body types (height, weight, BMI), ages, and sexes. For example, a drug intended for female patients should specify dosage programs and any specific adverse effects in female patients before approval.

Patient diagnostics

Trial data often uses diagnostic tests to measure a patient’s response to treatment. AI can be used to improve diagnostic accuracy and objectivity during trials, reduce the potential for bias, and help with blinding a trial. AI programs have already been shown to provide more accurate diagnoses compared to clinicians6, which could be extended to use in clinical trials. AI can also be used to integrate multiple biomarkers or large data sets (e.g. bioinformatics data) to better monitor and understand a patient’s response, and to make any needed changes in dosing.

Another application of AI in clinical trials would be in patient management. Wearable devices or apps could be used to provide real time safety and effectiveness data to both clinicians and patients, leading to better data quality and higher patient retention.

Patient monitoring, retainment, & medication adherence

AI could be used to monitor patients through automatic data capture and digital clinical assessments. Automated monitoring with AI can allow for personalised adherence alerts, and wearable devices could provide real time safety and efficacy data shared with both patient and clinician to increase retainment and adherence rates. Video consultations with clinicians could improve retainment by reducing travelling required from patients, but there would still be a risk of drop due to travel as some diagnostic tests would need to be done at an appropriate site, which may warrant additional costs if tests for research purposes are not covered by a patient’s health insurance or national health service.

Currently, medication adherence is mostly dependent on each patient’s diary/record keeping or memory, which is then discussed with clinicians during routine appointments. This can make it difficult to accurately track adherence. Digitising this through use of a website/app would allow for more accurate adherence data to be obtained, in addition to providing patients with notifications, educational content, and adherence records. Other medical devices such as timed medication bottles could also be used to ensure medication is used in appropriate intervals, and smart bottles could be used to synchronise this with a smartphone app if applicable.

Data Cleaning

Data cleaning for clinical trials is typically performed by trained biostatisticians and is essential to ensure that collected data is consistently formatted and free from inputting errors. However, the data cleaning process can be time-consuming, especially with large datasets collected during clinical trials. AI could be implemented through machine learning methods to identify and correct errors found in clinical trial datasets7, leading to better quality data which is optimised for analysis. An AI approach may also reduce the amount of time spent on data cleaning.

AI implementation and clinical trial digitisation

AI implementation is relevant in several aspects of clinical trials, be it in study design, patient diagnostics, or trial management. Several tech giants (including Apple & Google) have invested in developing solutions to process electronic health records, monitor patients remotely, and integrate healthcare data into devices. By improving the cost and time effectiveness of clinical trials, both patients and pharma-tech companies benefit with more affordably priced treatment costs for patients and greater return of investment for companies.

However, for clinical trials to implement AI successfully, many aspects of clinical trials would first require digitisation. Many trials still use paper documents instead of digital alternatives, which results in lost documents and slowed trial progression. Integrating electronic health records, digital copies of clinicians’ notes, and digital patient monitoring alone would help in designing and managing a trial. There is a concern that a switch to digital may be difficult for patients unfamiliar with technology, or for those who might prefer to keep paper diaries. However, digital solutions would allow for the development and implementation of AI-based solutions which would modernise and streamline the clinical trial process.

References

  1. Taylor K, Properzi F, Cruz M, Ronte H, Haughey J. Intelligent clinical trials [Internet]. www2.deloitte.com. 2020 [cited 16 March 2022]. Available from: https://www2.deloitte.com/content/dam/insights/us/articles/22934_intelligent-clinical-trials/DI_Intelligent-clinical-trials.pdf

  2. Glass L, Shorter G, Patil R. AI IN CLINICAL DEVELOPMENT [Internet]. IQVIA. 2019 [cited 16 March 2022]. Available from: https://www.iqvia.com/-/media/iqvia/pdfs/library/white-papers/ai-in-clinical-development.pdf
  • Bhatt A. Artificial intelligence in managing clinical trial design and conduct: Man and machine still on the learning curve?. Perspectives in Clinical Research. 2021;12(1):1-3. Available from: https://doi.org/10.4103/picr.PICR_312_20

4 common study designs for clinical trials

Clinical trial design is an important aspect of interventional trials that serves to optimise, ergonomise and economise the clinical trial conduct. The goals of a clinical trial, whether medtech or pharma, can encompass assessment of safety, dosage optimisation, evaluation of efficacy or accuracy and comparison to existing treatments or diagnostics. This of course varies with the phase of the trial. For phase III or IV trials the goal is most often to determine superiority, non-inferiority, or equivalence of the novel therapeutic or device to one in standard use. A well-conducted study that achieves regulatory approval for the asset in an efficient way depends upon the design that informs it. An optimal design, from a statistical and data collection perspective, ensures accurate evaluation efficacy and safety, as well as getting the product to market sooner. Knowing which study designs best suit your research will improve the chances of success, enable the best method for sample size estimation and re-estimation, save time and reduce unnecessary costs (Evans, 2010). While many clinical study designs exist this article focuses on perhaps the most rudimentary and frequently used designs

  • Parallel group design
  • Crossover design
  • Factorial design
  • Randomised withdrawal design

1. Parallel group study design

A commonly used study design is a parallel arm design. When using this as a study design, subjects are randomised and allocated to one or more study arms. In a parallel group study design, each study arm is allocated a different intervention. After study subjects have been randomised and allocated to a study arms they can not be allocated to another arm throughout the study.

Advantages of parallel group trial study design

A key advantage of parallel group trial design is that it can be applied to many different diseases as well as allows for conducting multiple experiments simultaneously between many groups. A further advantage is that these different groups need not be sourced from the same site.

Note: Once patients have been randomised and assigned to a specific arm, these arms are mutually exclusive. This means that unplanned co- interventions or cross-overs between different treatments cannot be introduced.

Steps involved in a parallel arm trial design:

1. Eligibility of study subject assessed

2. Recruitment into study after consent

 3. Randomisation

4. Allocation to either treatment or control arm
 

2. Cross-over study design

There are some ethical limitations to the use of placebo controls that can be partially overcome by using a cross over design. This means that every patient taking part in the clinical trial will receive both treatment and placebo being given in a randomised order (Evans, 2010). Cross-over study design can also be used in the absence of placebo where the intention is to compare the new treatment to the standard one.

Advantages of cross-over design

One of the advantages of cross over design is the fact that each patient acts as their own control results in order to balance the covariates in treatment and control arm. Another major advantage of cross over design is the fact that it requires a smaller sample size (Nair, 2019).

Note: When cross over design is applicable and chosen for the study, some of the patients will start the trial with using intervention A and then switch to intervention B which is known as a AB sequence, whereas other patients will start with using intervention B and later switch to intervention A which is known as BA sequence.

! There needs to be an adequate washout period before the crossover in order to eliminate the effects from initially assigned and administrated intervention. After all data has been collected the results are then compared within the same subject assessing the effect of intervention A vs. effect of intervention B (Nair, 2019).

Variations of cross-over design

(i) Switch back design (ABA vs BAB arms) –

1. Drug A -> Drug B-> Drug A

2.Drug B -> Drug A -> Drug B

The switch back and multiple switchback designs are of emerging relevance with the advent of biosimilars where switchability and interchangeability of a biosimilar to a bio-originator molecule can only be confirmed by such trial designs.

(ii) N of 1 design – N of 1 trials or “single-subject” or “structured within-patient randomized controlled multi-crossover trial design”

This type of cross over design is used for evaluating all interventions in a single patient. A typical N of 1 design clinical trial consists of repeating experimental/ control treatment periods number of times. The interventions being tested are assigned randomly within each period pair. This design has gained a lot of popularity, because in most cases the aim of using this type of design is to determine which treatment works best for the individual patient.

3. Factorial design

Factorial design is most suited when the study is looking at two or more interventions in various combinations within one study setting. This design helps in the study of interactive effects that have resulted from a combination of different interventions (Nair, 2019).

Advantages of Factorial design

A key advantage of factorial design is that it can help answer multiple research questions in a clinical trial instead of conducting multiple trials.  This helps to optimise resources, thereby reducing costs and speeding up research pipelines.

2 × 2 factorial design with placebo

In a 2 × 2 factorial design with placebo, patients are randomized into four groups:

i) treatment A plus placebo
 ii) treatment B plus placebo
 iii) both treatments A and B
 iv) neither of them, placebo only.

Limitations of the factorial design

The main limitations of using factorial design for clinical trials is the fact that:

○  Increased complexity of the trial overall

○  Makes it more difficult to meet inclusion criteria

○  Inability to combine multiple incompatible interventions

○  The protocols are complex

○  High complexity of statistical analysis

4. Randomised withdrawal design (EERW)

The aim of randomised withdrawal design is to evaluate the optimal duration of the treatment for patients that are responsive to the intervention.

 After the initial enrichment period (open label period) which main purpose is to assign the subjects to intervention, the subjects that are not responding are removed (dropped) from the study and the subjects that did respond are randomised into receiving the intervention or placebo during the second phase of the clinical trial (Nair, 2019).

Note: This means that only subjects that have responded are carried forward to the second stage of the study and randomised.

Statistical analysis of randomised withdrawal design

When using randomised withdrawal design the analysis of the study is conducted using only data from the withdrawal phase. Outcome is usually set to relapse of symptoms. The aim of the enrichment phase is to increase the statistical power for the estimated sample size.

Advantages of EERW

A main advantage of a randomised withdrawal design is that it can reduce the time patients receive placebo. Only patients that are responsive to the intervention are randomised to placebo, hence an increased ethical advantage. A further advantage of this study design is that it can help to determine if the treatment should be stopped or continued (Nair,2019).

Conclusion

One of the key stages of planning a clinical trial involves deciding on the appropriate study design to ensure the success of the research and help to choose the right method for sample size estimation and re-estimation, save time and reduce unnecessary costs.

The most commonly used study designs are :

  • Parallel group study design
  • Cross over study design
  • Factorial study design
  • Randomised withdrawal study design (EERW )

A well-conducted study with optimal design, that encorporates a robust hypothesis evolved from clinical practice, goes a long way in facilitating the regulatory approval process – evaluating efficacy and safety, and getting the product to market. Therefore when undertaking a clinical trial close attention should be paid to ensure that a study design forms a solid foundation upon which to conduct the trial phases.

References
 Evans, S., 2010. Fundamentals of clinical trial design. [online] PubMed Central (PMC). Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3083073/>.
 Expert, T., 2022. Clinical Trial Designs & Clinical Trial Phases | Credevo Articles. [online] Credevo Articles. Available at: <https://credevo.com/articles/2021/02/05/the-phase-of- study-clinical-trial-design/>.
 Nair, B., 2019. Clinical Trial Designs. [online] PubMed Central (PMC). Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6434767/>.
 The BMJ | The BMJ: leading general medical journal. Research. Education. Comment. n.d. 13. Study design and choosing a statistical test | The BMJ. [online] Available at: <https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/ 13-study-design-and-choosing-statisti>.

Bayesian approach for sample size estimation and re-adjustment in clinical trials

Bayesian approach for sample size estimation and re-adjustment in clinical trials

            Accurate sample size calculation plays an important role in clinical research. Sample size in this context simply refers to the number of human patients, wheather healthy or diseased, taking part in the study. Clinical studies conducted using an insufficient sample size can lack the statistical power to adequately evaluate the treatment of interest, whereas a superfluous sample size can unnecessarily waste limited resources.

          Various methods can be applied for determining the optimal sample size for a specific clinical study. Methods also exist for any re-adjustments throughout the study,  if required. These methods vary widely from straightforward tests and formulas to complex, time-consuming ones, depending on the type of study and available information from which to make the estimate. Most commonly used sample size calculation procedures are developed from a frequentist perspective

Importance of knowing your study parameters

          Accurate sample size calculation requires, information on several key study and research parameters. These parameters usually include an effect size and variability estimate, derived from available sources; a clinically meaningful difference. In practice these parameters are  generally unknown and must be estimated from the existing literature or from pilot studies.

The Bayesian Framework in sample size estimations and re-adjustments

The Bayesian Framework has gradually become one of the most frequently mentioned methods when it comes to randomised clinical trial sample size estimations and re-adjustments.

In practice, sample size calculation is usually treated explicitly as a decision problem and employs a loss or utility function.

The Bayesian approach involves three key stages:

  • 1. Prior estimate

A researcher has a prior estimate about the treatment effect (and other study parameters) that has been derived from meta-analysis of existing research, pilot studies, or  based on expert opinion in absence of these.

  • 2. Likelihood

Data is simulated to derive a likelihood estimate of prior parameters.

  • 3. Posterior estimate

Based on the insights obtained, prior estimates from the first stage are updated to give a more precise final estimate.

A challenge of using this approach is knowing when to stop this cycle when enough evidence has been gathered and avoid creating bias (Dreibe,2021). Peaking at the data in order to make a stopping decision is called “optional stopping”. In general an optional stopping rule is cautioned against as it can increase type one error rates (de Heide & Grunewald, 2021).

How to decide when to stop the simulation cycle?

There are two approaches one could take.

  • 1. Posterior probability

            Calculating the posterior probability that the mean difference between the treatment and control arm is equal or greater than the estimated effect of the intervention. Based on the level of probably calculated (low or high) the cycle could be stopped and without any further need to gather more data.

  • 2. PPOS ( predictive probability of success)

        Calculating the predictive probability of achieving a successful result at the end of the study is a commonly used approach. It is really helpful when it comes to determining the success or failure of a study. Similarly, as with posterior probability based on the level of probability a decision could be made to stop or continue the study.

How to plan a Bayesian sample size calculation for a clinical trial

The key elements to consider when planning a Bayesian clinical trial are the same as for frequentists clinical trial.

Key planning stages:

  • Determine the objective of the clinical study
  • Determine and set endpoints
  • Decide on the appropriate study design
  • Run a meta analysis or review of existing evidence related to your research objective
  • Statistical test and statistical analysis plan (SAP)

Even though the key planning stages are the same for both approaches it does not mean that they can be mixed through out the study. If you have chosen to use one approach you can’t change to another method once the calculations have been generated and research started.

Bayesian approach vs Frequentist approach for sample size calculations

BayesianFrequentist
Prior and posterior( uses probability of hypothesis and data)No prior or posterior( never gives probability of hypothesis)
Sample size depends on the prior and likelihoodSample size depend on the likelihood
Requeres finding/deciding on prior in order to estimate sample sizeDoes not require prior to estimate sample size
Computationally intensive due to integration over many parametersLess computationally intense

          Frequentist measures such as p-values and confidence intervals continue to predominate the methodology across life sciences research, however, the use of the Bayesian approach in sample size estimations and re-estimation for RTCs has been increasing over time.

Bayesian approach for sample size calculations in medical device clinical trial

           In the recent years Bayesian approach has gained more popularity as the method used in clinical trials including medical device studies. One of the reasons being that if good prior information about the use of the specific therapeutic or device is available, the Bayesian approach may allow to include this information into the statistical analysis part of the clinical trial. Sometimes, the available prior information for a device of interest may be used as a justification for smaller sample size and shorten the length of the pivotal trial (Chen et al., 2011).

Computational algorithms and growing popularity of Bayesian approach

          Bayesian statistical analysis can be computationally intense. Despite that there have been multiple breakthroughs with computational algorithms and increased computing speed that have made it much easier to calculate and build more realistic Bayesian models, further contributing to the popularity of Bayesian approach. (FDA, 2010).

Markov Chain Monte Carlo (MCMC) method

          One of the basic computational tools being used is Markov Chain Monte Carlo ( MCMC) method. This method computes large number of simulations from the distributions of random quantities.

Why MCMC?

          MCMC helps to deal with computational difficulties one often can face when using Bayesian approach for needed sample  size estimations. The MCMC is an advanced random variable generation technique which allows one to simulate different samples from more sophisticated probability distributions.

Conclusion

          Sample size calculation plays an important role in clinical research. If underestimated, statistical power for the detection of a clinically meaningful difference will likely be insufficient; if overestimated, resources are wasted unnecessarilly.

          The Bayesian Framework has become quite popular approach for sample size estimation. There are advantages of using the Bayesian method, depite this there has been some criticism of this approach as a sample size estimation and re-adjustment method due to the prior being subjective and possibility of different researchers selecting different priors leading to different posteriors and final conclusions.

In reality, both the Bayesian and frequentist approaches to sample size calculation involve deriving the relevant input parameters from the literature or clinical expertise and could potentially differ due to variations in individual expert opinion as to which studies to include or exclude in this process.

          Bayesian approach is more computationally intensive compared to the traditional frequentist approaches. Therefore, when it comes to selecting a method for sample size estimation, it should be chosen carefully to best fit the particular study design and base-on advice provided by statistical professionals with expertise in clinical trials.

References:

Bokai WANG, C., 2017. Comparisons of Superiority, Non-inferiority, and Equivalence Trials. [online] PubMed Central (PMC). Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5925592/> [Accessed 28 February 2022].

Chen, M., Ibrahim, J., Lam, P., Yu, A. and Zhang, Y., 2011. Bayesian Design of Noninferiority Trials for Medical Devices Using Historical Data. Biometrics, 67(3), pp.1163-1170.

E, L., 2008. Superiority, equivalence, and non-inferiority trials. [online] PubMed. Available at: <https://pubmed.ncbi.nlm.nih.gov/18537788/> [Accessed 28 February 2022].

Gubbiotti, S., 2008. Bayesian Methods for Sample Size Determination and their use in Clinical Trials. [online] Core.ac.uk. Available at: <https://core.ac.uk/download/pdf/74322247.pdf> [Accessed 28 February 2022].

U.S. Food and Drug Administration. 2010. Guidance for the Use of Bayesian Statistics in Medical Device Clinical. [online] Available at: <https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials> [Accessed 28 February 2022].

van Ravenzwaaij, D., Monden, R., Tendeiro, J. and Ioannidis, J., 2019. Bayes factors for superiority, non-inferiority, and equivalence designs. BMC Medical Research Methodology, 19(1).

de Heide. R, Grunewald, P.D, 2021, Why optional stopping can be a problem for Bayesians; Psychonomic Bulletin & Review, 21(2), 201-208.

Clinical Trial Phases in Drug Development


The development of new drugs starts far before they are even seen in clinical trials. The discovery of multiple candidate drugs occur early on in the development process, often as a result of new information about how a disease functions, large-scale screening of small molecules, or the release of a new technology.

After a promising drug has been found, pre-clinical studies can be performed. A pre-clinical study for a new drug is used to determine important information about toxicity and suitable dosage amounts. These studies can be in vitro (in cell culture) and/or in vivo (in animal models) and determine whether a treatment will continue to the clinical trials stage.

Clinical trials test whether these experimental treatments are safe for use in humans, and whether they are more effective in treating or preventing a disease when compared to existing treatments. Clinical trials consist of several stages, called phases, where each phase is focused on answering a different clinical question: Progression of a treatment to the next phase requires the study to meet several parameters to ensure a treatment’s safety or efficacy.

  • Phase 0: Is the new treatment safe to use in humans in small doses?
  • Phase I: Is the new treatment safe to use in humans in therapeutic doses?
  • Phase II: Is the new treatment effective in humans?
  • Phase III: Is the new treatment more effective than existing treatments?
  • Phase IV: Does the new treatment remain safe and effective post-market?
Key phases of a pharmaceutical clinical trial

Phase 0: Small dose safety

Phase 0 studies can help to streamline the other clinical trial phases. Phase 0 consists of giving a few patients small, sub-therapeutic doses of the new treatment. This is to make sure that the new treatment behaves as expected by researchers and isn’t harmful to humans prior to using higher doses in phase I trials.

Phase I: Therapeutic dose safety

Phase I studies evaluate the safety of various doses of the new treatment in humans. This takes several months with typically around 20-80 healthy volunteers. In some cases, such as in anti-cancer drug trials, the study participants are patients with the targeted cancer type. A treatment may not pass phase I if the treatment leads to any serious adverse events.

Initial dosages in phase I studies can be informed based on data obtained during pre-clinical animal studies, and adjustments can be made to investigate the treatment’s side effect profile and develop an optimal dosing program. This could also include comparing different methods of giving a drug to patients (e.g., oral, intravenous etc.).

Phase II: Treatment efficacy

After passing phase I trials and having proven safety in humans, a new treatment advances to phase II studies designed to assess whether it may prevent or treat a disease. This phase can take between several months to 2 years, testing the new treatment in up to several hundred patients with the disease. Using a larger number of patients over a longer time period provides researchers with additional safety and effectiveness data, which is essential for the design of phase III trials.

To further test safety and efficacy, it is common to have a control group that receives either a placebo (a harmless pill or injection without the new treatment) or other current treatment (in trials where the disease is fatal unless treated e.g., cancer).

Phase III: Comparing to current treatments

Phase III studies are the last stage of a clinical trial before a new treatment can be approved for market use. The primary focus of a phase III study is to compare the safety and efficacy of a new treatment with current, existing treatments in patients with the target disease. Anywhere from several hundred to 3,000 patients may be included in a phase III study for between 1 to 4 years. Due to the scale of this phase, long-term or rare side effects are more likely to be uncovered.

Phase III studies are often randomised control trials, where patients will be randomly designated to different treatment groups. These groups may receive placebo, a current treatment (control group), the new treatment, or variations of the new treatment (e.g., different drug combinations). Randomised control trials are often double-blinded, where both the patient and the clinician administering their treatment do not know which treatment group they are assigned to.

A new treatment may continue to market and phase IV trials if the results prove it is as safe and effective as an existing treatment.

Phase IV: Post-market surveillance

If a new treatment passes phase III and is approved by the MHRA, FDA, or other national regulatory agency, it can be put to market. Phase IV is carried out in the post-market surveillance of the new treatment to keep updated on any emerging or long-term safety and efficacy concerns. This may include rare or long-term adverse side effects that were not yet discovered, or long-term analyses to see if the new treatment improves the life expectancy of a patient after recovery from disease.

Summary

Clinical trials are ultimately designed to mitigate risk. This includes the risk to the safety of trial participants by limiting the use of potentially unsafe treatments to small doses in a small number of patients before scaling up to testing therapeutic dose safety. Risk mitigation is not only for patient safety but also for preventing financial misspending as a treatment that is deemed unsafe in phase 0 would not proceed to the later, more costly clinical trial phases.

Not all clinical trials are the same, however, as each trial will have a different disease and treatment context. Trials for medical devices are somewhat different from pharmaceutical trials (for more information about the differences between medical device and pharma trials, click here). In addition, while sample sizes expand with phase progression, the required sample size for each trial and each phase is dependent on several factors including disease context (a rare disease may require lower sample sizes), patient availability (location of trial), trial budget and effect size. The sample size values mentioned earlier in this blog are purely indications of what each phase may use (for more information on how a biostatistician determines a suitable sample size, click here).

References

https://www.fda.gov/patients/drug-development-process/step-3-clinical-research

https://www.healthline.com/health/clinical-trial-phases

Medical Device Clinical Trials vs Pharmaceutical Clinical Trials – What’s the Difference?

Medical devices and drugs share the same goal – to safely improve the health of patients. Despite this, substantial differences can be observed between the two. Principally, drugs interact with biochemical pathways in human bodies while medical devices can encompass a wide range of different actions and reactions, for example, heat, radiation (Taylor and Iglesias, 2009). Additionally, medical devices encompass not only therapeutic devices but diagnostic devices, as well (Stauffer, 2020).

More specifically medical device categories can include therapeutic and surgical devices, patient monitoring, diagnostic and medical imaging devices, among others; making it a very heterogeneous area (Stauffer, 2020). As such, medical device research spills over into many different fields of healthcare services and manufacturing. This research is mostly undertaken by SME’s ( small to medium enterprises) instead of larger well-established companies as is more predominantly the case with pharmaceutical research. SME’s and start-ups undertake the majority of the early stage device development, particularly where any new class of medical device is concerned, whereas the larger firms get involved in later stages of the testing process (Taylor and Iglesias, 2009).

Classification criteria for medical devices

There are strict regulations that researchers and developers need to follow, which includes general device classification criteria. This classification criterion consists of three classes of medical devices, the higher class medical device the stricter regulatory controls are for the medical device. 

  • Class I, typically do not require premarket notifications
  • Class II,  require premarket notifications
  • Class III, require premarket approval

Food and Drug Administration (FDA)

Drug licensing and market access approval by the Food and Drug Administration (FDA) and international equivalents require manufacturers to undertake phase II and III randomised controlled trials in order to provide the regulator with evidence of their drug’s efficacy and safety (Taylor and Iglesias, 2009).

Key stages of medical device clinical trials

In general medical device clinical trials are smaller than drug trials and usually start with feasibility study. This provides a limited clinical evaluation of the device. Next a pivotal trial is conducted to demonstrate the device in question is safe and effective (Stauffer, 2020).

Overall the medical device trials can be considered to have three stages:

  • Feasibility study,
  • Pivotal study to determine if the device is safe and effective,
  • Post-market study to analyse the long-term effectiveness of the device.

Clinical evaluation for medical devices

Clinical evaluation is an ongoing process conducted throughout the life cycle of a medical device. It is first performed during the development of a medical device in order to identify data that need to be generated for regulatory purposes and will inform if a new device clinical investigation is necessary. It is then repeated periodically as new safety, clinical performance and/or effectiveness information about the medical device is obtained during its use.(International Medical Device Regulators Forum, 2019)

During the evaluative process, a distinction must be made between device types – diagnostic or therapeutic. The criteria for diagnostic technology evaluations are usually divided into four groups:

  • technical capacity
  • diagnostic accuracy
  • diagnostic and therapeutic impact
  • patient outcome

The importance of evaluation

Evaluations provide important information about a device and can indicate the possible risks and complications. The main measures of diagnostic performance are sensitivity and specificity. Based on the results of the clinical investigation the intervention may be approved for the market. When placing a medical device on the market, the manufacturer must have demonstrated through the use of appropriate conformity assessment procedures that the medical device complies with the Essential Principles of Safety and Performance of Medical Devices(International Medical Device Regulators Forum, 2019).The information on effectiveness can be observed by conducting experimental or observational studies.

Post-market surveillance

Manufacturers are expected to implement and maintain surveillance programs that routinely monitor the safety, clinical performance and/or effectiveness of the medical device as part of their Quality Management System (International Medical Device Regulators Forum, 2019). The scope and nature of such post market surveillance should be appropriate to the medical device and its intended use. Using data generated from such programs (e.g. safety reports, including adverse event reports; results from published literature, any further clinical investigations), a manufacturer should periodically review performance, safety and the benefit-risk assessment for the medical device through a clinical evaluation, and update the clinical evidence accordingly.

The use of databases in medical device clinical trials

The variations in the available evidence-base for devices means that, unlike with drugs, medical devices will typically require the consideration and analysis of data from observational studies in ascertaining their clinical and cost-effectiveness. Using modern observational databases has advantages because these databases represent continuous monitoring of the device in real-life practice, including the outcome (Maresova et al., 2020).

Bayesian methods as an alternative framework for evaluation

Bayesian methods for the analysis of trial data have been proposed as an alternative framework for evaluation within the FDA’s Center for Devices and Radiological Health. These methods provide flexibility and may make them particularly well suited to address many of the issues associated with the assessment of clinical and economic evidence on medical devices, for example, learning effects and lack of head-to-head comparisons between different devices.

Use of placebo in medical vs pharmaceutical trials

An additional key difference between drug and medical device trials are that use of placebo in medical device trials are rare. If placebo is used in a trial for surgical / implanted devices  it would usually be a sham surgery or implantation of a sham device (Taylor and Iglesias, 2009). Sham procedures are high risk and may be considered unethical. Without this kind of control, however, there is in many cases no sure way of knowing whether the device is providing real clinical benefit or if the benefit experienced is due to the placebo effect. 

Conclusion

            In conclusion, there are many similarities between medical device and pharmaceutical clinical trials, but there are also some really important differences that one should not miss:

  1.  In general medical device clinical trials are smaller than drug trials.
  2.  The research is mostly undertaken by SME’s ( small to medium enterprises) instead of big well-known companies
  3. Drugs interact with biochemical pathways in human bodies whereas medical devices use a wide range of different actions and reactions, for example, heat, radiation.
  4. Medical devices can be used for not only diagnostic purposes but therapeutical purposes as well.
  5.  The use of placebo in medical device trials are rare in comparison to pharmaceutical clinical trials.

References:

Bokai WANG, C., 2017. Comparisons of Superiority, Non-inferiority, and Equivalence Trials. [online] PubMed Central (PMC). Available at: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5925592/> [Accessed 28 February 2022].

Chen, M., Ibrahim, J., Lam, P., Yu, A. and Zhang, Y., 2011. Bayesian Design of Noninferiority Trials for Medical Devices Using Historical Data. Biometrics, 67(3), pp.1163-1170.

E, L., 2008. Superiority, equivalence, and non-inferiority trials. [online] PubMed. Available at: <https://pubmed.ncbi.nlm.nih.gov/18537788/> [Accessed 28 February 2022].

Gubbiotti, S., 2008. Bayesian Methods for Sample Size Determination and their use in Clinical Trials. [online] Core.ac.uk. Available at: <https://core.ac.uk/download/pdf/74322247.pdf> [Accessed 28 February 2022].

U.S. Food and Drug Administration. 2010. Guidance for the Use of Bayesian Statistics in Medical Device Clinical. [online] Available at: <https://www.fda.gov/regulatory-information/search-fda-guidance-documents/guidance-use-bayesian-statistics-medical-device-clinical-trials> [Accessed 28 February 2022].

van Ravenzwaaij, D., Monden, R., Tendeiro, J. and Ioannidis, J., 2019. Bayes factors for superiority, non-inferiority, and equivalence designs. BMC Medical Research Methodology, 19(1).